1159941 - tooltool.py fails because it uses unsupported Bearer auth

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

Running xpcshell locally now fails with a tooltool download, because it uses the Bearer auth mechanism, which is apparently now unsupported by tooltool.

Comment 1

[Jonathan Griffin (:jgriffin)]

Symptom:

15:33:58     INFO - We want to download this file https://raw.githubusercontent.com/mozilla/build-tooltool/master/tooltool.py
15:34:00     INFO - Downloaded 36212 bytes.
15:34:00     INFO - Chmoding /home/jgriffin/mozharness/build/tooltool.py to 0755
15:34:00     INFO - proxxy config: {'regions': ['.use1.', '.usw2.'], 'instances': ['proxxy1.srv.releng.use1.mozilla.com', 'proxxy1.srv.releng.usw2.mozilla.com'], 'urls': [('http://ftp.mozilla.org', 'ftp.mozilla.org'), ('https://ftp.mozilla.org', 'ftp.mozilla.org'), ('https://ftp-ssl.mozilla.org', 'ftp.mozilla.org'), ('http://pvtbuilds.pvt.build.mozilla.org', 'pvtbuilds.mozilla.org'), ('http://tooltool.pvt.build.mozilla.org', 'tooltool.pvt.build.mozilla.org'), ('http://pypi.pvt.build.mozilla.org', 'pypi.pvt.build.mozilla.org'), ('http://pypi.pub.build.mozilla.org', 'pypi.pub.build.mozilla.org')]}
15:34:00     INFO - retry: Calling run_command with args: (['/home/jgriffin/mozharness/build/tooltool.py', '--authentication-file', '/home/jgriffin/.mozilla/credentials.cfg', '--url', 'https://secure.pub.build.mozilla.org/tooltool/pvt/build', 'fetch', '-m', '/home/jgriffin/mozharness/build/tests/config/tooltool-manifests/linux64/releng.manifest', '-o', '-c', '/home/jgriffin/.mozilla/releng/builds/tooltool_cache'],), kwargs: {'error_list': [{'substr': 'command not found', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x7fefb998e7e0>, 'level': 'warning'}, {'substr': 'Traceback (most recent call last)', 'level': 'error'}, {'substr': 'SyntaxError: ', 'level': 'error'}, {'substr': 'TypeError: ', 'level': 'error'}, {'substr': 'NameError: ', 'level': 'error'}, {'substr': 'ZeroDivisionError: ', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x7fefb99a6da0>, 'level': 'critical'}, {'regex': <_sre.SRE_Pattern object at 0x7fefb99a79f0>, 'level': 'critical'}, {'substr': 'ERROR - ', 'level': 'error'}], 'cwd': '/home/jgriffin/mozharness/build', 'privileged': False}, attempt #1
15:34:00     INFO - Running command: ['/home/jgriffin/mozharness/build/tooltool.py', '--authentication-file', '/home/jgriffin/.mozilla/credentials.cfg', '--url', 'https://secure.pub.build.mozilla.org/tooltool/pvt/build', 'fetch', '-m', '/home/jgriffin/mozharness/build/tests/config/tooltool-manifests/linux64/releng.manifest', '-o', '-c', '/home/jgriffin/.mozilla/releng/builds/tooltool_cache'] in /home/jgriffin/mozharness/build
15:34:00     INFO - Copy/paste: /home/jgriffin/mozharness/build/tooltool.py --authentication-file /home/jgriffin/.mozilla/credentials.cfg --url https://secure.pub.build.mozilla.org/tooltool/pvt/build fetch -m /home/jgriffin/mozharness/build/tests/config/tooltool-manifests/linux64/releng.manifest -o -c /home/jgriffin/.mozilla/releng/builds/tooltool_cache
15:34:00     INFO -  INFO - File linux64-minidump_stackwalk not present in local cache folder /home/jgriffin/.mozilla/releng/builds/tooltool_cache
15:34:00     INFO -  INFO - Attempting to fetch from 'https://secure.pub.build.mozilla.org/tooltool/pvt/build/'...
15:34:02     INFO -  INFO - ...failed to fetch 'linux64-minidump_stackwalk' from https://secure.pub.build.mozilla.org/tooltool/pvt/build/
15:34:02    ERROR -  ERROR - The following files failed: 'linux64-minidump_stackwalk'
15:34:02    ERROR - Return code: 1

Comment 2

[Jonathan Griffin (:jgriffin)]

Attachment: Link to Github pull-request: https://github.com/mozilla/build-tooltool/pull/18

Comment 3

[Jonathan Griffin (:jgriffin)]

with patch:

INFO - File linux64-minidump_stackwalk not present in local cache folder /home/jgriffin/.mozilla/releng/builds/tooltool_cache
DEBUG - fetching linux64-minidump_stackwalk
INFO - Attempting to fetch 'https://secure.pub.build.mozilla.org/tooltool/pvt/build/sha512/2b7c8922f27244b5d244a2d4a6f7dcbe2b29227952fbde9396575805f83f34f62afd3e659614eb38210a817788dbef8042b049938e56779bcba0f72798272144'...
DEBUG - using bearer token in /home/jgriffin/.mozilla/credentials.cfg
DEBUG - opened https://secure.pub.build.mozilla.org/tooltool/pvt/build/sha512/2b7c8922f27244b5d244a2d4a6f7dcbe2b29227952fbde9396575805f83f34f62afd3e659614eb38210a817788dbef8042b049938e56779bcba0f72798272144 for reading
INFO - File linux64-minidump_stackwalk fetched from https://secure.pub.build.mozilla.org/tooltool/pvt/build/ as /home/jgriffin/mozharness/tmpumbcSg
DEBUG - hashed 'tmpumbcSg' with sha512 to be 2b7c8922f27244b5d244a2d4a6f7dcbe2b29227952fbde9396575805f83f34f62afd3e659614eb38210a817788dbef8042b049938e56779bcba0f72798272144
INFO - File integrity verified, renaming tmpumbcSg to linux64-minidump_stackwalk
INFO - Updating local cache /home/jgriffin/.mozilla/releng/builds/tooltool_cache...
INFO - Creating cache in /home/jgriffin/.mozilla/releng/builds/tooltool_cache...
INFO - Local cache /home/jgriffin/.mozilla/releng/builds/tooltool_cache updated with linux64-minidump_stackwalk

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8599591 [details] [review]
Link to Github pull-request: https://github.com/mozilla/build-tooltool/pull/18

Bearer is the correct authentication type.  You're using the wrong URL, though (one that was set up for developers to download files from, but not for use in production).  The correct new URL is https://api.pub.build.mozilla.org/tooltol/.  That will be in all of the mozharness configs tomorrow morning, per bug 1155238.

The files available from the URLs are the same.

After the landing, access to tooltool via secure.pub.build.mozilla.org will be deprecated.

Comment 5

[Jonathan Griffin (:jgriffin)]

Is https://github.com/mozilla/build-tooltool/blob/master/tooltool.py going to be updated accordingly as well?

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Not sure what you mean -- it already is updated.

Comment 7

[Jonathan Griffin (:jgriffin)]

Nm, I think the local developer case should be fixed by the mozharness update also.

Comment 8

[Armen [:armenzg]]

I've spend a lot of time fixing this use case for developers.
Can someone please take ownership of fixing this?

The developer mode fetches tooltool.py from github raw [1] so we can use their credentials to grab the files that are needed to run the automation.

[1] https://raw.githubusercontent.com/mozilla/build-tooltool/master/tooltool.py

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Armen: for the next few hours you should use an older revision of tooltool:
  https://raw.githubusercontent.com/mozilla/build-tooltool/c03e10815ec561d5ac9c367f9234700aee978f8b/tooltool.py

After that, users will need to use a different form of authentication (a token) and a different URL, and all should be well again.

This transition has taken a lot longer than the "few days" I was hoping it would take.  Sorry about that :(

Comment 10

[Armen [:armenzg]]

Attachment: [mozharness] Point to specific revision of tooltool.py

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8599867 [details] [diff] [review]
[mozharness] Point to specific revision of tooltool.py

do not land this, please - it will burn automation.

Comment 12

[Armen [:armenzg]]

I think we're done in here.
I've been able to use tooltool.py on mozharness by putting a token in here:
https://wiki.mozilla.org/ReleaseEngineering/Mozharness/How_to_run_tests_as_a_developer#Step_2_-_Create_a_tooltool_token_.28fetches_artifacts_for_you.29

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

I'm a little bummed it has to go in /builds/relengapi.tok, and not somewhere under $HOME, but if you're cool with that, I won't object.  I don't understand MozHarness well enough to fix it, anyway.