Closed Bug 1385629 Opened 7 years ago Closed 7 years ago

Intermittent Can't create directory /home/worker/tooltool-cache!

Categories

(Release Engineering :: General, defect, P5)

defect

Tracking

(firefox56 fixed, firefox57 fixed)

RESOLVED FIXED
Tracking Status
firefox56 --- fixed
firefox57 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: gbrown)

References

Details

(Keywords: intermittent-failure)

Attachments

(1 file)

Filed by: archaeopteryx [at] coole-files.de

https://treeherder.mozilla.org/logviewer.html#?job_id=119106534&repo=mozilla-inbound

https://queue.taskcluster.net/v1/task/DOB3jYPeRZeDV5sH2TMxtQ/runs/0/artifacts/public/logs/live_backing.log

14:56:33     INFO - mkdir: /Users/cltbld/tasks/task_1501278951/build/application
14:56:33     INFO - Getting output from command: ['/Users/cltbld/tasks/task_1501278951/build/venv/bin/mozinstall', '/Users/cltbld/tasks/task_1501278951/build/target.dmg', '--destination', '/Users/cltbld/tasks/task_1501278951/build/application']
14:56:33     INFO - Copy/paste: /Users/cltbld/tasks/task_1501278951/build/venv/bin/mozinstall /Users/cltbld/tasks/task_1501278951/build/target.dmg --destination /Users/cltbld/tasks/task_1501278951/build/application
14:56:55     INFO - Reading from file tmpfile_stdout
14:56:55     INFO - Output received:
14:56:55     INFO -  /Users/cltbld/tasks/task_1501278951/build/application/NightlyDebug.app/Contents/MacOS/firefox
14:56:55     INFO - Running post-action listener: _resource_record_post_action
14:56:55     INFO - [mozharness: 2017-07-28 21:56:55.347317Z] Finished install step (success)
14:56:55     INFO - [mozharness: 2017-07-28 21:56:55.347459Z] Running run-tests step.
14:56:55     INFO - Running pre-action listener: _pre_run_tests
14:56:55     INFO - Running pre-action listener: _resource_record_pre_action
14:56:55     INFO - Running pre-action listener: _set_gcov_prefix
14:56:55     INFO - Running main action method: run_tests
/Users/cltbld/tasks/task_1501278951/build/venv/lib/python2.7/site-packages/mozrunner/utils.py:18: UserWarning: Module mozprocess was already imported from /Users/cltbld/tasks/task_1501278951/mozharness/mozprocess/__init__.py, but /Users/cltbld/tasks/task_1501278951/build/venv/lib/python2.7/site-packages is being added to sys.path
  import pkg_resources
14:56:55     INFO - Minidump filename unknown. Determining based upon platform and architecture.
14:56:55     INFO - Minidump tooltool manifest unknown. Determining based upon platform and architecture.
14:56:55     INFO - grabbing minidump binary from tooltool
14:56:55     INFO - mkdir: /home/worker/tooltool-cache
14:56:55    ERROR - Can't create directory /home/worker/tooltool-cache!
14:56:55     INFO - proxxy config: {'regions': ['.use1.', '.usw2.'], 'instances': ['proxxy1.srv.releng.use1.mozilla.com', 'proxxy1.srv.releng.usw2.mozilla.com'], 'urls': [('http://ftp.mozilla.org', 'ftp.mozilla.org'), ('https://ftp.mozilla.org', 'ftp.mozilla.org'), ('https://ftp-ssl.mozilla.org', 'ftp.mozilla.org'), ('http://pypi.pvt.build.mozilla.org', 'pypi.pvt.build.mozilla.org'), ('http://pypi.pub.build.mozilla.org', 'pypi.pub.build.mozilla.org')]}
14:56:55     INFO - retry: Calling run_command with args: (['/tools/tooltool.py', '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--authentication-file', '/builds/relengapi.tok', 'fetch', '-m', '/Users/cltbld/tasks/task_1501278951/build/tests/config/tooltool-manifests/macosx64/releng.manifest', '-o', '-c', '/home/worker/tooltool-cache'],), kwargs: {'output_timeout': 600, 'error_list': [{'substr': 'command not found', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x10422b2c0>, 'level': 'warning'}, {'substr': 'Traceback (most recent call last)', 'level': 'error'}, {'substr': 'SyntaxError: ', 'level': 'error'}, {'substr': 'TypeError: ', 'level': 'error'}, {'substr': 'NameError: ', 'level': 'error'}, {'substr': 'ZeroDivisionError: ', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x10457d6e8>, 'level': 'critical'}, {'regex': <_sre.SRE_Pattern object at 0x1043032a0>, 'level': 'critical'}, {'substr': 'ERROR - ', 'level': 'error'}], 'cwd': '/Users/cltbld/tasks/task_1501278951/build', 'privileged': False}, attempt #1
14:56:55     INFO - Running command: ['/tools/tooltool.py', '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--authentication-file', '/builds/relengapi.tok', 'fetch', '-m', '/Users/cltbld/tasks/task_1501278951/build/tests/config/tooltool-manifests/macosx64/releng.manifest', '-o', '-c', '/home/worker/tooltool-cache'] in /Users/cltbld/tasks/task_1501278951/build
14:56:55     INFO - Copy/paste: /tools/tooltool.py --url https://api.pub.build.mozilla.org/tooltool/ --authentication-file /builds/relengapi.tok fetch -m /Users/cltbld/tasks/task_1501278951/build/tests/config/tooltool-manifests/macosx64/releng.manifest -o -c /home/worker/tooltool-cache
14:56:55     INFO - Calling ['/tools/tooltool.py', '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--authentication-file', '/builds/relengapi.tok', 'fetch', '-m', '/Users/cltbld/tasks/task_1501278951/build/tests/config/tooltool-manifests/macosx64/releng.manifest', '-o', '-c', '/home/worker/tooltool-cache'] with output_timeout 600
14:56:55     INFO -  INFO - File macosx64-minidump_stackwalk not present in local cache folder /home/worker/tooltool-cache
14:56:55     INFO -  INFO - Attempting to fetch from 'https://api.pub.build.mozilla.org/tooltool/'...
14:56:58     INFO -  INFO - File macosx64-minidump_stackwalk fetched from https://api.pub.build.mozilla.org/tooltool/ as /Users/cltbld/tasks/task_1501278951/build/tmpwXBL3r
14:56:58     INFO -  INFO - File integrity verified, renaming tmpwXBL3r to macosx64-minidump_stackwalk
14:56:58     INFO -  INFO - Updating local cache /home/worker/tooltool-cache...
14:56:58     INFO -  INFO - Creating cache in /home/worker/tooltool-cache...
14:56:58     INFO -  WARNING - Impossible to add file macosx64-minidump_stackwalk to cache folder /home/worker/tooltool-cache
14:56:58    ERROR -  Traceback (most recent call last):
14:56:58     INFO -    File "/tools/tooltool.py", line 669, in fetch_files
14:56:58     INFO -      os.makedirs(cache_folder, 0700)
14:56:58     INFO -    File "/Users/cltbld/tasks/task_1501278951/build/venv/lib/python2.7/os.py", line 150, in makedirs
14:56:58     INFO -      makedirs(head, mode)
14:56:58     INFO -    File "/Users/cltbld/tasks/task_1501278951/build/venv/lib/python2.7/os.py", line 157, in makedirs
14:56:58     INFO -      mkdir(name, mode)
14:56:58     INFO -  OSError: [Errno 45] Operation not supported: '/home/worker'
All of these failures are osx and either marionette or firefox-ui-functional tests. It looks like those test jobs always have the error "Can't create directory /home/worker/tooltool-cache!"...even in green jobs.

For example,

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=980e4db21b3ed6779c36d1a21be21598565d1f16&filter-searchStr=osx+marionette

https://public-artifacts.taskcluster.net/TMYwCbmGTiO1n6CcZIUieA/0/public/logs/live_backing.log

14:40:03     INFO - grabbing minidump binary from tooltool
14:40:03     INFO - mkdir: /home/worker/tooltool-cache
14:40:04    ERROR - Can't create directory /home/worker/tooltool-cache!


Other types of osx tests, like mochitests, use a different directory, successfully:

https://public-artifacts.taskcluster.net/Nde_4L6PQYuc2qo57Cn-GA/0/public/logs/live_backing.log

14:44:00     INFO - grabbing minidump binary from tooltool
...
14:44:00     INFO -  INFO - File macosx64-minidump_stackwalk retrieved from local cache /builds/tooltool_cache


So I think /builds/tooltool_cache should be used in all osx tests.


:glandium - I see you made some changes from /builds/tooltool_cache to /home/worker/tooltool-cache back in bug 1356520. Do you agree that probably should not apply to osx tests?
Flags: needinfo?(mh+mozilla)
Those errors are red herrings. Those logs have *actual* failures, but the tooltool-cache errors still appear in the summary, making things confusing for sheriffs. It would make sense to use /build/tooltool_cache on those osx tests only, just because of that.
Flags: needinfo?(mh+mozilla)
Assignee: nobody → gbrown
On osx, the tooltool cache is at /builds/tooltool_cache instead of /home/worker/tooltool-cache, but Marionette and Firefox-ui tests do not currently make a distinction between platforms.

This patch adds new mozharness configuration files to override the default cache location on osx, for Marionette and Firefox-ui tests.


Seems to work well:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=192c565dd3501154aaeed30a36ec8201d275385e

https://public-artifacts.taskcluster.net/HOTSo_7TRbSLqfzhedi2hA/0/public/logs/live_backing.log

00:16:12     INFO - grabbing minidump binary from tooltool
00:16:12     INFO - proxxy config: {'regions': ['.use1.', '.usw2.'], 'instances': ['proxxy1.srv.releng.use1.mozilla.com', 'proxxy1.srv.releng.usw2.mozilla.com'], 'urls': [('http://ftp.mozilla.org', 'ftp.mozilla.org'), ('https://ftp.mozilla.org', 'ftp.mozilla.org'), ('https://ftp-ssl.mozilla.org', 'ftp.mozilla.org'), ('http://pypi.pvt.build.mozilla.org', 'pypi.pvt.build.mozilla.org'), ('http://pypi.pub.build.mozilla.org', 'pypi.pub.build.mozilla.org')]}
00:16:12     INFO - retry: Calling run_command with args: (['/tools/tooltool.py', '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--authentication-file', '/builds/relengapi.tok', 'fetch', '-m', '/Users/cltbld/tasks/task_1501830897/build/tests/config/tooltool-manifests/macosx64/releng.manifest', '-o', '-c', '/builds/tooltool_cache'],), kwargs: {'output_timeout': 600, 'error_list': [{'substr': 'command not found', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x101625a70>, 'level': 'warning'}, {'substr': 'Traceback (most recent call last)', 'level': 'error'}, {'substr': 'SyntaxError: ', 'level': 'error'}, {'substr': 'TypeError: ', 'level': 'error'}, {'substr': 'NameError: ', 'level': 'error'}, {'substr': 'ZeroDivisionError: ', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x101797030>, 'level': 'critical'}, {'regex': <_sre.SRE_Pattern object at 0x1015bb9f0>, 'level': 'critical'}, {'substr': 'ERROR - ', 'level': 'error'}], 'cwd': '/Users/cltbld/tasks/task_1501830897/build', 'privileged': False}, attempt #1
00:16:12     INFO - Running command: ['/tools/tooltool.py', '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--authentication-file', '/builds/relengapi.tok', 'fetch', '-m', '/Users/cltbld/tasks/task_1501830897/build/tests/config/tooltool-manifests/macosx64/releng.manifest', '-o', '-c', '/builds/tooltool_cache'] in /Users/cltbld/tasks/task_1501830897/build
00:16:12     INFO - Copy/paste: /tools/tooltool.py --url https://api.pub.build.mozilla.org/tooltool/ --authentication-file /builds/relengapi.tok fetch -m /Users/cltbld/tasks/task_1501830897/build/tests/config/tooltool-manifests/macosx64/releng.manifest -o -c /builds/tooltool_cache
00:16:12     INFO - Calling ['/tools/tooltool.py', '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--authentication-file', '/builds/relengapi.tok', 'fetch', '-m', '/Users/cltbld/tasks/task_1501830897/build/tests/config/tooltool-manifests/macosx64/releng.manifest', '-o', '-c', '/builds/tooltool_cache'] with output_timeout 600
00:16:12     INFO -  INFO - File macosx64-minidump_stackwalk retrieved from local cache /builds/tooltool_cache
00:16:12     INFO - Return code: 0
00:16:12     INFO - Chmoding /Users/cltbld/tasks/task_1501830897/build/macosx64-minidump_stackwalk to 0755
Attachment #8893884 - Flags: review?(hskupin)
This is actually a dupe of bug 1376191, whereby it looks like we are closer to a final result here? Wander, do you think we should continue here and obsolete the work you did so far?
Flags: needinfo?(wcosta)
(In reply to Henrik Skupin (:whimboo) from comment #5)
> This is actually a dupe of bug 1376191, whereby it looks like we are closer
> to a final result here? Wander, do you think we should continue here and
> obsolete the work you did so far?

I didn't have time to work on bug 1376191 so far. I should be able to come back to it next week, but feel free to close it as a dup and proceeding with :gbrown patch :-) (which is pretty close to what I did, iirc)
Flags: needinfo?(wcosta)
Comment on attachment 8893884 [details] [diff] [review]
use /builds/tooltool_cache on osx

Review of attachment 8893884 [details] [diff] [review]:
-----------------------------------------------------------------

I think a refactoring of the config files should be done at some point. But for now this looks fine. With the review comments addressed r=me.

Uh, splinter-reviewing... is somewhat strange those days. :)

::: testing/mozharness/configs/firefox_ui_tests/taskcluster_mac.py
@@ +1,1 @@
> +# Configuration over-rides for osx

Currently we clone the full file for a new platform. Could you please add that this is based on taskcluster.py?

::: testing/mozharness/configs/marionette/mac_taskcluster_config.py
@@ +1,1 @@
> +# Configuration over-rides for osx

Same here and that this is based on prod_config.py?
Attachment #8893884 - Flags: review?(hskupin) → review+
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/8b22f1b5e9f3
Use correct tooltool cache directory for Marionette and Firefox-UI tests on osx; r=whimboo
https://hg.mozilla.org/mozilla-central/rev/8b22f1b5e9f3
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.