Closed
Bug 1159941
Opened 9 years ago
Closed 9 years ago
tooltool.py fails because it uses unsupported Bearer auth
Categories
(Release Engineering :: General, defect)
Release Engineering
General
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: jgriffin, Assigned: armenzg)
Details
Attachments
(2 files)
Running xpcshell locally now fails with a tooltool download, because it uses the Bearer auth mechanism, which is apparently now unsupported by tooltool.
Reporter | ||
Comment 1•9 years ago
|
||
Symptom: 15:33:58 INFO - We want to download this file https://raw.githubusercontent.com/mozilla/build-tooltool/master/tooltool.py 15:34:00 INFO - Downloaded 36212 bytes. 15:34:00 INFO - Chmoding /home/jgriffin/mozharness/build/tooltool.py to 0755 15:34:00 INFO - proxxy config: {'regions': ['.use1.', '.usw2.'], 'instances': ['proxxy1.srv.releng.use1.mozilla.com', 'proxxy1.srv.releng.usw2.mozilla.com'], 'urls': [('http://ftp.mozilla.org', 'ftp.mozilla.org'), ('https://ftp.mozilla.org', 'ftp.mozilla.org'), ('https://ftp-ssl.mozilla.org', 'ftp.mozilla.org'), ('http://pvtbuilds.pvt.build.mozilla.org', 'pvtbuilds.mozilla.org'), ('http://tooltool.pvt.build.mozilla.org', 'tooltool.pvt.build.mozilla.org'), ('http://pypi.pvt.build.mozilla.org', 'pypi.pvt.build.mozilla.org'), ('http://pypi.pub.build.mozilla.org', 'pypi.pub.build.mozilla.org')]} 15:34:00 INFO - retry: Calling run_command with args: (['/home/jgriffin/mozharness/build/tooltool.py', '--authentication-file', '/home/jgriffin/.mozilla/credentials.cfg', '--url', 'https://secure.pub.build.mozilla.org/tooltool/pvt/build', 'fetch', '-m', '/home/jgriffin/mozharness/build/tests/config/tooltool-manifests/linux64/releng.manifest', '-o', '-c', '/home/jgriffin/.mozilla/releng/builds/tooltool_cache'],), kwargs: {'error_list': [{'substr': 'command not found', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x7fefb998e7e0>, 'level': 'warning'}, {'substr': 'Traceback (most recent call last)', 'level': 'error'}, {'substr': 'SyntaxError: ', 'level': 'error'}, {'substr': 'TypeError: ', 'level': 'error'}, {'substr': 'NameError: ', 'level': 'error'}, {'substr': 'ZeroDivisionError: ', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x7fefb99a6da0>, 'level': 'critical'}, {'regex': <_sre.SRE_Pattern object at 0x7fefb99a79f0>, 'level': 'critical'}, {'substr': 'ERROR - ', 'level': 'error'}], 'cwd': '/home/jgriffin/mozharness/build', 'privileged': False}, attempt #1 15:34:00 INFO - Running command: ['/home/jgriffin/mozharness/build/tooltool.py', '--authentication-file', '/home/jgriffin/.mozilla/credentials.cfg', '--url', 'https://secure.pub.build.mozilla.org/tooltool/pvt/build', 'fetch', '-m', '/home/jgriffin/mozharness/build/tests/config/tooltool-manifests/linux64/releng.manifest', '-o', '-c', '/home/jgriffin/.mozilla/releng/builds/tooltool_cache'] in /home/jgriffin/mozharness/build 15:34:00 INFO - Copy/paste: /home/jgriffin/mozharness/build/tooltool.py --authentication-file /home/jgriffin/.mozilla/credentials.cfg --url https://secure.pub.build.mozilla.org/tooltool/pvt/build fetch -m /home/jgriffin/mozharness/build/tests/config/tooltool-manifests/linux64/releng.manifest -o -c /home/jgriffin/.mozilla/releng/builds/tooltool_cache 15:34:00 INFO - INFO - File linux64-minidump_stackwalk not present in local cache folder /home/jgriffin/.mozilla/releng/builds/tooltool_cache 15:34:00 INFO - INFO - Attempting to fetch from 'https://secure.pub.build.mozilla.org/tooltool/pvt/build/'... 15:34:02 INFO - INFO - ...failed to fetch 'linux64-minidump_stackwalk' from https://secure.pub.build.mozilla.org/tooltool/pvt/build/ 15:34:02 ERROR - ERROR - The following files failed: 'linux64-minidump_stackwalk' 15:34:02 ERROR - Return code: 1
Reporter | ||
Comment 2•9 years ago
|
||
Attachment #8599591 -
Flags: review?(dustin)
Reporter | ||
Comment 3•9 years ago
|
||
with patch: INFO - File linux64-minidump_stackwalk not present in local cache folder /home/jgriffin/.mozilla/releng/builds/tooltool_cache DEBUG - fetching linux64-minidump_stackwalk INFO - Attempting to fetch 'https://secure.pub.build.mozilla.org/tooltool/pvt/build/sha512/2b7c8922f27244b5d244a2d4a6f7dcbe2b29227952fbde9396575805f83f34f62afd3e659614eb38210a817788dbef8042b049938e56779bcba0f72798272144'... DEBUG - using bearer token in /home/jgriffin/.mozilla/credentials.cfg DEBUG - opened https://secure.pub.build.mozilla.org/tooltool/pvt/build/sha512/2b7c8922f27244b5d244a2d4a6f7dcbe2b29227952fbde9396575805f83f34f62afd3e659614eb38210a817788dbef8042b049938e56779bcba0f72798272144 for reading INFO - File linux64-minidump_stackwalk fetched from https://secure.pub.build.mozilla.org/tooltool/pvt/build/ as /home/jgriffin/mozharness/tmpumbcSg DEBUG - hashed 'tmpumbcSg' with sha512 to be 2b7c8922f27244b5d244a2d4a6f7dcbe2b29227952fbde9396575805f83f34f62afd3e659614eb38210a817788dbef8042b049938e56779bcba0f72798272144 INFO - File integrity verified, renaming tmpumbcSg to linux64-minidump_stackwalk INFO - Updating local cache /home/jgriffin/.mozilla/releng/builds/tooltool_cache... INFO - Creating cache in /home/jgriffin/.mozilla/releng/builds/tooltool_cache... INFO - Local cache /home/jgriffin/.mozilla/releng/builds/tooltool_cache updated with linux64-minidump_stackwalk
Comment 4•9 years ago
|
||
Comment on attachment 8599591 [details] [review] Link to Github pull-request: https://github.com/mozilla/build-tooltool/pull/18 Bearer is the correct authentication type. You're using the wrong URL, though (one that was set up for developers to download files from, but not for use in production). The correct new URL is https://api.pub.build.mozilla.org/tooltol/. That will be in all of the mozharness configs tomorrow morning, per bug 1155238. The files available from the URLs are the same. After the landing, access to tooltool via secure.pub.build.mozilla.org will be deprecated.
Attachment #8599591 -
Flags: review?(dustin) → review-
Reporter | ||
Comment 5•9 years ago
|
||
Is https://github.com/mozilla/build-tooltool/blob/master/tooltool.py going to be updated accordingly as well?
Comment 6•9 years ago
|
||
Not sure what you mean -- it already is updated.
Reporter | ||
Comment 7•9 years ago
|
||
Nm, I think the local developer case should be fixed by the mozharness update also.
Assignee | ||
Comment 8•9 years ago
|
||
I've spend a lot of time fixing this use case for developers. Can someone please take ownership of fixing this? The developer mode fetches tooltool.py from github raw [1] so we can use their credentials to grab the files that are needed to run the automation. [1] https://raw.githubusercontent.com/mozilla/build-tooltool/master/tooltool.py
Comment 9•9 years ago
|
||
Armen: for the next few hours you should use an older revision of tooltool: https://raw.githubusercontent.com/mozilla/build-tooltool/c03e10815ec561d5ac9c367f9234700aee978f8b/tooltool.py After that, users will need to use a different form of authentication (a token) and a different URL, and all should be well again. This transition has taken a lot longer than the "few days" I was hoping it would take. Sorry about that :(
Assignee | ||
Comment 10•9 years ago
|
||
Assignee: jgriffin → armenzg
Attachment #8599867 -
Flags: review?(jgriffin)
Comment 11•9 years ago
|
||
Comment on attachment 8599867 [details] [diff] [review] [mozharness] Point to specific revision of tooltool.py do not land this, please - it will burn automation.
Attachment #8599867 -
Flags: review?(jgriffin) → review-
Assignee | ||
Comment 12•9 years ago
|
||
I think we're done in here. I've been able to use tooltool.py on mozharness by putting a token in here: https://wiki.mozilla.org/ReleaseEngineering/Mozharness/How_to_run_tests_as_a_developer#Step_2_-_Create_a_tooltool_token_.28fetches_artifacts_for_you.29
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Comment 13•9 years ago
|
||
I'm a little bummed it has to go in /builds/relengapi.tok, and not somewhere under $HOME, but if you're cool with that, I won't object. I don't understand MozHarness well enough to fix it, anyway.
Updated•6 years ago
|
Component: General Automation → General
You need to log in
before you can comment on or make changes to this bug.
Description
•