Closed
Bug 1385629
Opened 7 years ago
Closed 7 years ago
Intermittent Can't create directory /home/worker/tooltool-cache!
Categories
(Release Engineering :: General, defect, P5)
Release Engineering
General
Tracking
(firefox56 fixed, firefox57 fixed)
RESOLVED
FIXED
People
(Reporter: intermittent-bug-filer, Assigned: gbrown)
References
Details
(Keywords: intermittent-failure)
Attachments
(1 file)
2.57 KB,
patch
|
whimboo
:
review+
|
Details | Diff | Splinter Review |
Filed by: archaeopteryx [at] coole-files.de https://treeherder.mozilla.org/logviewer.html#?job_id=119106534&repo=mozilla-inbound https://queue.taskcluster.net/v1/task/DOB3jYPeRZeDV5sH2TMxtQ/runs/0/artifacts/public/logs/live_backing.log 14:56:33 INFO - mkdir: /Users/cltbld/tasks/task_1501278951/build/application 14:56:33 INFO - Getting output from command: ['/Users/cltbld/tasks/task_1501278951/build/venv/bin/mozinstall', '/Users/cltbld/tasks/task_1501278951/build/target.dmg', '--destination', '/Users/cltbld/tasks/task_1501278951/build/application'] 14:56:33 INFO - Copy/paste: /Users/cltbld/tasks/task_1501278951/build/venv/bin/mozinstall /Users/cltbld/tasks/task_1501278951/build/target.dmg --destination /Users/cltbld/tasks/task_1501278951/build/application 14:56:55 INFO - Reading from file tmpfile_stdout 14:56:55 INFO - Output received: 14:56:55 INFO - /Users/cltbld/tasks/task_1501278951/build/application/NightlyDebug.app/Contents/MacOS/firefox 14:56:55 INFO - Running post-action listener: _resource_record_post_action 14:56:55 INFO - [mozharness: 2017-07-28 21:56:55.347317Z] Finished install step (success) 14:56:55 INFO - [mozharness: 2017-07-28 21:56:55.347459Z] Running run-tests step. 14:56:55 INFO - Running pre-action listener: _pre_run_tests 14:56:55 INFO - Running pre-action listener: _resource_record_pre_action 14:56:55 INFO - Running pre-action listener: _set_gcov_prefix 14:56:55 INFO - Running main action method: run_tests /Users/cltbld/tasks/task_1501278951/build/venv/lib/python2.7/site-packages/mozrunner/utils.py:18: UserWarning: Module mozprocess was already imported from /Users/cltbld/tasks/task_1501278951/mozharness/mozprocess/__init__.py, but /Users/cltbld/tasks/task_1501278951/build/venv/lib/python2.7/site-packages is being added to sys.path import pkg_resources 14:56:55 INFO - Minidump filename unknown. Determining based upon platform and architecture. 14:56:55 INFO - Minidump tooltool manifest unknown. Determining based upon platform and architecture. 14:56:55 INFO - grabbing minidump binary from tooltool 14:56:55 INFO - mkdir: /home/worker/tooltool-cache 14:56:55 ERROR - Can't create directory /home/worker/tooltool-cache! 14:56:55 INFO - proxxy config: {'regions': ['.use1.', '.usw2.'], 'instances': ['proxxy1.srv.releng.use1.mozilla.com', 'proxxy1.srv.releng.usw2.mozilla.com'], 'urls': [('http://ftp.mozilla.org', 'ftp.mozilla.org'), ('https://ftp.mozilla.org', 'ftp.mozilla.org'), ('https://ftp-ssl.mozilla.org', 'ftp.mozilla.org'), ('http://pypi.pvt.build.mozilla.org', 'pypi.pvt.build.mozilla.org'), ('http://pypi.pub.build.mozilla.org', 'pypi.pub.build.mozilla.org')]} 14:56:55 INFO - retry: Calling run_command with args: (['/tools/tooltool.py', '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--authentication-file', '/builds/relengapi.tok', 'fetch', '-m', '/Users/cltbld/tasks/task_1501278951/build/tests/config/tooltool-manifests/macosx64/releng.manifest', '-o', '-c', '/home/worker/tooltool-cache'],), kwargs: {'output_timeout': 600, 'error_list': [{'substr': 'command not found', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x10422b2c0>, 'level': 'warning'}, {'substr': 'Traceback (most recent call last)', 'level': 'error'}, {'substr': 'SyntaxError: ', 'level': 'error'}, {'substr': 'TypeError: ', 'level': 'error'}, {'substr': 'NameError: ', 'level': 'error'}, {'substr': 'ZeroDivisionError: ', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x10457d6e8>, 'level': 'critical'}, {'regex': <_sre.SRE_Pattern object at 0x1043032a0>, 'level': 'critical'}, {'substr': 'ERROR - ', 'level': 'error'}], 'cwd': '/Users/cltbld/tasks/task_1501278951/build', 'privileged': False}, attempt #1 14:56:55 INFO - Running command: ['/tools/tooltool.py', '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--authentication-file', '/builds/relengapi.tok', 'fetch', '-m', '/Users/cltbld/tasks/task_1501278951/build/tests/config/tooltool-manifests/macosx64/releng.manifest', '-o', '-c', '/home/worker/tooltool-cache'] in /Users/cltbld/tasks/task_1501278951/build 14:56:55 INFO - Copy/paste: /tools/tooltool.py --url https://api.pub.build.mozilla.org/tooltool/ --authentication-file /builds/relengapi.tok fetch -m /Users/cltbld/tasks/task_1501278951/build/tests/config/tooltool-manifests/macosx64/releng.manifest -o -c /home/worker/tooltool-cache 14:56:55 INFO - Calling ['/tools/tooltool.py', '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--authentication-file', '/builds/relengapi.tok', 'fetch', '-m', '/Users/cltbld/tasks/task_1501278951/build/tests/config/tooltool-manifests/macosx64/releng.manifest', '-o', '-c', '/home/worker/tooltool-cache'] with output_timeout 600 14:56:55 INFO - INFO - File macosx64-minidump_stackwalk not present in local cache folder /home/worker/tooltool-cache 14:56:55 INFO - INFO - Attempting to fetch from 'https://api.pub.build.mozilla.org/tooltool/'... 14:56:58 INFO - INFO - File macosx64-minidump_stackwalk fetched from https://api.pub.build.mozilla.org/tooltool/ as /Users/cltbld/tasks/task_1501278951/build/tmpwXBL3r 14:56:58 INFO - INFO - File integrity verified, renaming tmpwXBL3r to macosx64-minidump_stackwalk 14:56:58 INFO - INFO - Updating local cache /home/worker/tooltool-cache... 14:56:58 INFO - INFO - Creating cache in /home/worker/tooltool-cache... 14:56:58 INFO - WARNING - Impossible to add file macosx64-minidump_stackwalk to cache folder /home/worker/tooltool-cache 14:56:58 ERROR - Traceback (most recent call last): 14:56:58 INFO - File "/tools/tooltool.py", line 669, in fetch_files 14:56:58 INFO - os.makedirs(cache_folder, 0700) 14:56:58 INFO - File "/Users/cltbld/tasks/task_1501278951/build/venv/lib/python2.7/os.py", line 150, in makedirs 14:56:58 INFO - makedirs(head, mode) 14:56:58 INFO - File "/Users/cltbld/tasks/task_1501278951/build/venv/lib/python2.7/os.py", line 157, in makedirs 14:56:58 INFO - mkdir(name, mode) 14:56:58 INFO - OSError: [Errno 45] Operation not supported: '/home/worker'
Comment hidden (Intermittent Failures Robot) |
Assignee | ||
Comment 2•7 years ago
|
||
All of these failures are osx and either marionette or firefox-ui-functional tests. It looks like those test jobs always have the error "Can't create directory /home/worker/tooltool-cache!"...even in green jobs. For example, https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=980e4db21b3ed6779c36d1a21be21598565d1f16&filter-searchStr=osx+marionette https://public-artifacts.taskcluster.net/TMYwCbmGTiO1n6CcZIUieA/0/public/logs/live_backing.log 14:40:03 INFO - grabbing minidump binary from tooltool 14:40:03 INFO - mkdir: /home/worker/tooltool-cache 14:40:04 ERROR - Can't create directory /home/worker/tooltool-cache! Other types of osx tests, like mochitests, use a different directory, successfully: https://public-artifacts.taskcluster.net/Nde_4L6PQYuc2qo57Cn-GA/0/public/logs/live_backing.log 14:44:00 INFO - grabbing minidump binary from tooltool ... 14:44:00 INFO - INFO - File macosx64-minidump_stackwalk retrieved from local cache /builds/tooltool_cache So I think /builds/tooltool_cache should be used in all osx tests. :glandium - I see you made some changes from /builds/tooltool_cache to /home/worker/tooltool-cache back in bug 1356520. Do you agree that probably should not apply to osx tests?
Flags: needinfo?(mh+mozilla)
Comment 3•7 years ago
|
||
Those errors are red herrings. Those logs have *actual* failures, but the tooltool-cache errors still appear in the summary, making things confusing for sheriffs. It would make sense to use /build/tooltool_cache on those osx tests only, just because of that.
Flags: needinfo?(mh+mozilla)
Assignee | ||
Updated•7 years ago
|
Assignee: nobody → gbrown
Assignee | ||
Comment 4•7 years ago
|
||
On osx, the tooltool cache is at /builds/tooltool_cache instead of /home/worker/tooltool-cache, but Marionette and Firefox-ui tests do not currently make a distinction between platforms. This patch adds new mozharness configuration files to override the default cache location on osx, for Marionette and Firefox-ui tests. Seems to work well: https://treeherder.mozilla.org/#/jobs?repo=try&revision=192c565dd3501154aaeed30a36ec8201d275385e https://public-artifacts.taskcluster.net/HOTSo_7TRbSLqfzhedi2hA/0/public/logs/live_backing.log 00:16:12 INFO - grabbing minidump binary from tooltool 00:16:12 INFO - proxxy config: {'regions': ['.use1.', '.usw2.'], 'instances': ['proxxy1.srv.releng.use1.mozilla.com', 'proxxy1.srv.releng.usw2.mozilla.com'], 'urls': [('http://ftp.mozilla.org', 'ftp.mozilla.org'), ('https://ftp.mozilla.org', 'ftp.mozilla.org'), ('https://ftp-ssl.mozilla.org', 'ftp.mozilla.org'), ('http://pypi.pvt.build.mozilla.org', 'pypi.pvt.build.mozilla.org'), ('http://pypi.pub.build.mozilla.org', 'pypi.pub.build.mozilla.org')]} 00:16:12 INFO - retry: Calling run_command with args: (['/tools/tooltool.py', '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--authentication-file', '/builds/relengapi.tok', 'fetch', '-m', '/Users/cltbld/tasks/task_1501830897/build/tests/config/tooltool-manifests/macosx64/releng.manifest', '-o', '-c', '/builds/tooltool_cache'],), kwargs: {'output_timeout': 600, 'error_list': [{'substr': 'command not found', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x101625a70>, 'level': 'warning'}, {'substr': 'Traceback (most recent call last)', 'level': 'error'}, {'substr': 'SyntaxError: ', 'level': 'error'}, {'substr': 'TypeError: ', 'level': 'error'}, {'substr': 'NameError: ', 'level': 'error'}, {'substr': 'ZeroDivisionError: ', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x101797030>, 'level': 'critical'}, {'regex': <_sre.SRE_Pattern object at 0x1015bb9f0>, 'level': 'critical'}, {'substr': 'ERROR - ', 'level': 'error'}], 'cwd': '/Users/cltbld/tasks/task_1501830897/build', 'privileged': False}, attempt #1 00:16:12 INFO - Running command: ['/tools/tooltool.py', '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--authentication-file', '/builds/relengapi.tok', 'fetch', '-m', '/Users/cltbld/tasks/task_1501830897/build/tests/config/tooltool-manifests/macosx64/releng.manifest', '-o', '-c', '/builds/tooltool_cache'] in /Users/cltbld/tasks/task_1501830897/build 00:16:12 INFO - Copy/paste: /tools/tooltool.py --url https://api.pub.build.mozilla.org/tooltool/ --authentication-file /builds/relengapi.tok fetch -m /Users/cltbld/tasks/task_1501830897/build/tests/config/tooltool-manifests/macosx64/releng.manifest -o -c /builds/tooltool_cache 00:16:12 INFO - Calling ['/tools/tooltool.py', '--url', 'https://api.pub.build.mozilla.org/tooltool/', '--authentication-file', '/builds/relengapi.tok', 'fetch', '-m', '/Users/cltbld/tasks/task_1501830897/build/tests/config/tooltool-manifests/macosx64/releng.manifest', '-o', '-c', '/builds/tooltool_cache'] with output_timeout 600 00:16:12 INFO - INFO - File macosx64-minidump_stackwalk retrieved from local cache /builds/tooltool_cache 00:16:12 INFO - Return code: 0 00:16:12 INFO - Chmoding /Users/cltbld/tasks/task_1501830897/build/macosx64-minidump_stackwalk to 0755
Attachment #8893884 -
Flags: review?(hskupin)
Comment 5•7 years ago
|
||
This is actually a dupe of bug 1376191, whereby it looks like we are closer to a final result here? Wander, do you think we should continue here and obsolete the work you did so far?
Flags: needinfo?(wcosta)
Comment 6•7 years ago
|
||
(In reply to Henrik Skupin (:whimboo) from comment #5) > This is actually a dupe of bug 1376191, whereby it looks like we are closer > to a final result here? Wander, do you think we should continue here and > obsolete the work you did so far? I didn't have time to work on bug 1376191 so far. I should be able to come back to it next week, but feel free to close it as a dup and proceeding with :gbrown patch :-) (which is pretty close to what I did, iirc)
Flags: needinfo?(wcosta)
Comment hidden (Intermittent Failures Robot) |
Comment 9•7 years ago
|
||
Comment on attachment 8893884 [details] [diff] [review] use /builds/tooltool_cache on osx Review of attachment 8893884 [details] [diff] [review]: ----------------------------------------------------------------- I think a refactoring of the config files should be done at some point. But for now this looks fine. With the review comments addressed r=me. Uh, splinter-reviewing... is somewhat strange those days. :) ::: testing/mozharness/configs/firefox_ui_tests/taskcluster_mac.py @@ +1,1 @@ > +# Configuration over-rides for osx Currently we clone the full file for a new platform. Could you please add that this is based on taskcluster.py? ::: testing/mozharness/configs/marionette/mac_taskcluster_config.py @@ +1,1 @@ > +# Configuration over-rides for osx Same here and that this is based on prod_config.py?
Attachment #8893884 -
Flags: review?(hskupin) → review+
Comment 10•7 years ago
|
||
Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/8b22f1b5e9f3 Use correct tooltool cache directory for Marionette and Firefox-UI tests on osx; r=whimboo
Comment 11•7 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/8b22f1b5e9f3
Comment 12•7 years ago
|
||
bugherder uplift |
https://hg.mozilla.org/releases/mozilla-beta/rev/276cdb43ab4a
status-firefox56:
--- → fixed
Comment hidden (Intermittent Failures Robot) |
Updated•6 years ago
|
Component: General Automation → General
You need to log in
before you can comment on or make changes to this bug.
Description
•