Closed Bug 1232543 Opened 9 years ago Closed 9 years ago

In jittests tests/bin/js is not executable

Categories

(Testing :: General, defect)

Product:
Testing
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox46 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla46
Tracking Flags:
Tracking Status
firefox46 --- fixed

People

(Reporter: armenzg, Unassigned)

Details

Attachments

(1 file)

MozReview Request: Bug 1232543: treat target.jsshell.zip as a zip file; r?jmaher
40 bytes, text/x-review-board-request
jmaher
: review+
Details 
I fixed jittests in one of my latest pushes to actually run (instead of being green for not running tests).

It seems that js shell is not executable.

https://public-artifacts.taskcluster.net/Z1yqQuhzSBGHp1XbRyx57A/0/public/logs/live_backing.log
23:22:43     INFO - #### Running jittest suites
23:22:43     INFO - mkdir: /home/worker/workspace/build/blobber_upload_dir
23:22:43     INFO - ENV: MOZ_UPLOAD_DIR is now /home/worker/workspace/build/blobber_upload_dir
23:22:43     INFO - ENV: MINIDUMP_SAVE_PATH is now /home/worker/workspace/build/blobber_upload_dir
23:22:43     INFO - Running command: ['/home/worker/workspace/build/venv/bin/python', '-u', '/home/worker/workspace/build/tests/jit-test/jit-test/jit_test.py', '--total-chunks', '2', '--this-chunk', '1', 'tests/bin/js', '--no-slow', '--no-progress', '--format=automation', '--jitflags=all'] in /home/worker/workspace/build
23:22:43     INFO - Copy/paste: /home/worker/workspace/build/venv/bin/python -u /home/worker/workspace/build/tests/jit-test/jit-test/jit_test.py --total-chunks 2 --this-chunk 1 tests/bin/js --no-slow --no-progress --format=automation --jitflags=all
23:22:43     INFO - Using env: {'DISPLAY': ':0',
23:22:43     INFO -  'GECKO_HEAD_REPOSITORY': 'https://hg.mozilla.org/try/',
23:22:43     INFO -  'GECKO_HEAD_REV': '4abad467f7c0a58625265e54a058d9e39876bf5b',
23:22:43     INFO -  'HOME': '/home/worker',
23:22:43     INFO -  'HOSTNAME': 'taskcluster-worker',
23:22:43     INFO -  'LANG': 'en_US.UTF-8',
23:22:43     INFO -  'LC_ALL': 'en_US.UTF-8',
23:22:43     INFO -  'LOGNAME': 'worker',
23:22:43     INFO -  'MINIDUMP_SAVE_PATH': '/home/worker/workspace/build/blobber_upload_dir',
23:22:43     INFO -  'MOZHARNESS_CONFIG': 'mozharness/configs/unittests/linux_unittest.py mozharness/configs/remove_executables.py\n',
23:22:43     INFO -  'MOZHARNESS_SCRIPT': 'mozharness/scripts/desktop_unittest.py',
23:22:43     INFO -  'MOZHARNESS_URL': 'https://queue.taskcluster.net/v1/task/E3e32HArQ9GAsAVWCwrspQ/artifacts/public/build/mozharness.zip',
23:22:43     INFO -  'MOZILLA_BUILD_URL': 'https://queue.taskcluster.net/v1/task/E3e32HArQ9GAsAVWCwrspQ/artifacts/public/build/target.tar.bz2',
23:22:43     INFO -  'MOZ_UPLOAD_DIR': '/home/worker/workspace/build/blobber_upload_dir',
23:22:43     INFO -  'NEED_PULSEAUDIO': 'true',
23:22:43     INFO -  'NEED_WINDOW_MANAGER': 'true',
23:22:43     INFO -  'OLDPWD': '/home/worker/workspace',
23:22:43     INFO -  'PATH': '/home/worker/workspace/build/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin',
23:22:43     INFO -  'PWD': '/home/worker/workspace',
23:22:43     INFO -  'RUN_ID': '0',
23:22:43     INFO -  'SHELL': '/bin/bash',
23:22:43     INFO -  'SHLVL': '1',
23:22:43     INFO -  'SUDO_COMMAND': '/bin/bash /home/worker/bin/test.sh --no-read-buildbot-config --installer-url=https://queue.taskcluster.net/v1/task/E3e32HArQ9GAsAVWCwrspQ/artifacts/public/build/target.tar.bz2 --test-packages-url=https://queue.taskcluster.net/v1/task/E3e32HArQ9GAsAVWCwrspQ/artifacts/public/build/test_packages.json --download-symbols=ondemand --jittest-suite=jittest-chunked --total-chunk=2 --this-chunk=1',
23:22:43     INFO -  'SUDO_GID': '0',
23:22:43     INFO -  'SUDO_UID': '0',
23:22:43     INFO -  'SUDO_USER': 'root',
23:22:43     INFO -  'TASK_ID': 'Z1yqQuhzSBGHp1XbRyx57A',
23:22:43     INFO -  'TERM': 'xterm',
23:22:43     INFO -  'USER': 'worker',
23:22:43     INFO -  'USERNAME': 'worker',
23:22:43     INFO -  '_': '/usr/bin/python2.7'}
23:22:43     INFO - Calling ['/home/worker/workspace/build/venv/bin/python', '-u', '/home/worker/workspace/build/tests/jit-test/jit-test/jit_test.py', '--total-chunks', '2', '--this-chunk', '1', 'tests/bin/js', '--no-slow', '--no-progress', '--format=automation', '--jitflags=all'] with output_timeout 1000
23:22:43     INFO -  Usage: jit_test.py [options] JS_SHELL [TESTS]
23:22:43     INFO -  jit_test.py: error: shell is not executable: /home/worker/workspace/build/tests/bin/js
23:22:43    ERROR - Return code: 2
23:22:43    ERROR - No tests run or test summary not found
Armen, the Testing/TaskCluster component is deprecated.  For issues around decision task configuration, TaskCluster/Integration is best; otherwise, as for this case, another testing category is probably best.

Where does this executable come from?  Is it from the build task, or is it created during the execution of the tests?
Component: TaskCluster → General
It comes from the test.zip generated on the build.
I did:
wget https://public-artifacts.taskcluster.net/E3e32HArQ9GAsAVWCwrspQ/0/public/build/target.jsshell.zip
unzip target.jsshell.zip
./js

and I got a shell.  So somehow this is either not downloading target.jsshell.zip (which I see no mention in the log), or we have some permissions issue.
It is mentioned in the log:

23:21:22     INFO - Downloading packages: [u'target.common.tests.zip', u'target.jsshell.zip'] for test suite category: jittest
...
23:21:26     INFO - https://queue.taskcluster.net/v1/task/E3e32HArQ9GAsAVWCwrspQ/artifacts/public/build/target.jsshell.zip matches https://queue.taskcluster.net
23:21:26     INFO - trying https://queue.taskcluster.net/v1/task/E3e32HArQ9GAsAVWCwrspQ/artifacts/public/build/target.jsshell.zip
23:21:26     INFO - Downloading https://queue.taskcluster.net/v1/task/E3e32HArQ9GAsAVWCwrspQ/artifacts/public/build/target.jsshell.zip to /home/worker/workspace/build/target.jsshell.zip
23:21:26     INFO - retry: Calling _download_file with args: (), kwargs: {'url': u'https://queue.taskcluster.net/v1/task/E3e32HArQ9GAsAVWCwrspQ/artifacts/public/build/target.jsshell.zip', 'file_name': u'/home/worker/workspace/build/target.jsshell.zip'}, attempt #1
23:21:27     INFO - Downloaded 8006100 bytes.
23:21:27     INFO - Running command: ['unzip', '-q', '-o', u'/home/worker/workspace/build/target.jsshell.zip', 'bin/*', 'certs/*', 'modules/*', 'mozbase/*', 'config/*', 'jit-test/*'] in /home/worker/workspace/build/tests
23:21:27     INFO - Copy/paste: unzip -q -o /home/worker/workspace/build/target.jsshell.zip bin/* certs/* modules/* mozbase/* config/* jit-test/*
23:21:27     INFO - Calling ['unzip', '-q', '-o', u'/home/worker/workspace/build/target.jsshell.zip', 'bin/*', 'certs/*', 'modules/*', 'mozbase/*', 'config/*', 'jit-test/*'] with output_timeout 1760
23:21:27     INFO -  caution: filename not matched:  bin/*
23:21:27     INFO -  caution: filename not matched:  certs/*
23:21:27     INFO -  caution: filename not matched:  modules/*
23:21:27     INFO -  caution: filename not matched:  mozbase/*
23:21:27     INFO -  caution: filename not matched:  config/*
23:21:27     INFO -  caution: filename not matched:  jit-test/*
23:21:27     INFO - Return code: 11

so it looks like there's some disagreement about the proper shape of that zip file -- it should have bin/js, but instead per comment 3 it has ./js.  In fact, it looks like it only contains the bin/ directory:

dustin@hopper ~/tmp/jsshell $ unzip ../target.jsshell.zip 
Archive:  ../target.jsshell.zip
  inflating: js                      
  inflating: libnspr4.so             
  inflating: libplds4.so             
  inflating: libplc4.so  

Although this is the same for https://queue.taskcluster.net/v1/task/qQf8LwgbR_6QhKk_ZK3axQ/artifacts/public/build/jsshell-linux-i686.zip in http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-linux-pgo/1450161018/mozilla-inbound_ubuntu32_vm_test_pgo-jittest-1-bm04-tests1-linux32-build335.txt.gz

Curiously, though, the unzip command is different for that job:

01:16:02     INFO - https://queue.taskcluster.net/v1/task/qQf8LwgbR_6QhKk_ZK3axQ/artifacts/public/build/jsshell-linux-i686.zip matches https://queue.taskcluster.net
01:16:02     INFO - URL Candidate: http://queue.taskcluster.net.proxxy1.srv.releng.usw2.mozilla.com/v1/task/qQf8LwgbR_6QhKk_ZK3axQ/artifacts/public/build/jsshell-linux-i686.zip
01:16:02     INFO - trying http://queue.taskcluster.net.proxxy1.srv.releng.usw2.mozilla.com/v1/task/qQf8LwgbR_6QhKk_ZK3axQ/artifacts/public/build/jsshell-linux-i686.zip
01:16:02     INFO - Downloading http://queue.taskcluster.net.proxxy1.srv.releng.usw2.mozilla.com/v1/task/qQf8LwgbR_6QhKk_ZK3axQ/artifacts/public/build/jsshell-linux-i686.zip to /builds/slave/test/build/jsshell-linux-i686.zip
01:16:02     INFO - retry: Calling _download_file with args: (), kwargs: {'url': u'http://queue.taskcluster.net.proxxy1.srv.releng.usw2.mozilla.com/v1/task/qQf8LwgbR_6QhKk_ZK3axQ/artifacts/public/build/jsshell-linux-i686.zip', 'file_name': u'/builds/slave/test/build/jsshell-linux-i686.zip'}, attempt #1
01:16:03     INFO - Downloaded 7880746 bytes.
01:16:03     INFO - Running command: ['unzip', '-q', '-o', u'/builds/slave/test/build/jsshell-linux-i686.zip'] in /builds/slave/test/build/tests/bin
01:16:03     INFO - Copy/paste: unzip -q -o /builds/slave/test/build/jsshell-linux-i686.zip
01:16:03     INFO - Calling ['unzip', '-q', '-o', u'/builds/slave/test/build/jsshell-linux-i686.zip'] with output_timeout 1760
01:16:04     INFO - Return code: 0

...and here's the culprit:

435             for file_name in target_packages:
436                 target_dir = test_install_dir
437                 unzip_dirs = target_unzip_dirs
438                 if "jsshell-" in file_name:
439                     unzip_dirs = None
440                     target_dir = dirs['abs_test_bin_dir']
441                 url = self.query_build_dir_url(file_name)
442                 self._download_unzip(url, target_dir,
443                                      target_unzip_dirs=unzip_dirs)

We sure do love to match on filenames :)
Bug 1232543: treat target.jsshell.zip as a zip file; r?jmaher
Attachment #8698452 - Flags: review?(jmaher)
I suspect somewhere along the way we started using the jsshell.zip package instead of the test package for jit tests, but I don't know why it'd work in some places but not others.

Also: we should definitely stop matching filenames. That's terrible. Where do we get this filename from, test_packages.json? (I see it listed in there if I pick an arbitrary build: http://ftp.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-win32/1449745722/test_packages.json ). If so, maybe we should just enhance the test_packages.json format to allow specifying a subdir under which a package should be extracted?
Attachment #8698452 - Flags: review?(jmaher) → review+
Comment on attachment 8698452 [details]
MozReview Request: Bug 1232543: treat target.jsshell.zip as a zip file; r?jmaher

https://reviewboard.mozilla.org/r/28001/#review25069

thanks Dustin!
Ted that's a good suggestion - if you'd like, please throw up a patch to accomplish it!
https://hg.mozilla.org/mozilla-central/rev/fd8301031738
Status: NEW → RESOLVED
Closed: 9 years ago
status-firefox46: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla46
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: