Android wrench test task unable to download alternate sdk from tooltool
Categories
(Testing :: General, defect, P3)
Tracking
(Not tracked)
People
(Reporter: gbrown, Unassigned)
References
Details
In bug 1556058 I am trying to update the android emulator used for Android 7.0 tests in CI. When I do that, the new android wrench tests fail: they are forbidden from downloading the new sdk. All other tests are okay.
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=249467128&repo=try&lineNumber=1247
[task 2019-05-31T20:48:26.860Z] 20:48:26 INFO - Writing to file /builds/worker/workspace/build/tooltool.tt
[task 2019-05-31T20:48:26.860Z] 20:48:26 INFO - Contents:
[task 2019-05-31T20:48:26.860Z] 20:48:26 INFO -
[task 2019-05-31T20:48:26.861Z] 20:48:26 INFO - [
[task 2019-05-31T20:48:26.861Z] 20:48:26 INFO - {
[task 2019-05-31T20:48:26.861Z] 20:48:26 INFO - "algorithm": "sha512",
[task 2019-05-31T20:48:26.861Z] 20:48:26 INFO - "visibility": "internal",
[task 2019-05-31T20:48:26.861Z] 20:48:26 INFO - "filename": "android-sdk_r29.0.11.0-linux-x86emu.tar.gz",
[task 2019-05-31T20:48:26.861Z] 20:48:26 INFO - "unpack": true,
[task 2019-05-31T20:48:26.861Z] 20:48:26 INFO - "digest": "954d6c7ecf3e10468ae0ca8d97f930eb1e1665ddf5d9317dd4bb8fbc13271cf12c4e343170aee782c33f1b6e15e5915f62c9e4a2a66eb32cc0b919cd6fb9659b",
[task 2019-05-31T20:48:26.861Z] 20:48:26 INFO - "size": 330652164
[task 2019-05-31T20:48:26.861Z] 20:48:26 INFO - }
[task 2019-05-31T20:48:26.861Z] 20:48:26 INFO - ]
[task 2019-05-31T20:48:26.861Z] 20:48:26 INFO - retry: Calling run_command with args: (['/builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python', '-u', '/builds/worker/checkouts/gecko/mach', 'artifact', 'toolchain', '-v', '--tooltool-url', 'https://tooltool.mozilla-releng.net/', '--tooltool-manifest', '/builds/worker/workspace/build/tooltool.tt', '--artifact-manifest', '/builds/worker/checkouts/gecko/toolchains.json', '--cache-dir', '/builds/worker/tooltool-cache'],), kwargs: {'output_timeout': 600, 'error_list': [{'substr': 'command not found', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x7fa455332550>, 'level': 'warning'}, {'substr': 'Traceback (most recent call last)', 'level': 'error'}, {'substr': 'SyntaxError: ', 'level': 'error'}, {'substr': 'TypeError: ', 'level': 'error'}, {'substr': 'NameError: ', 'level': 'error'}, {'substr': 'ZeroDivisionError: ', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x7fa4558d9da0>, 'level': 'critical'}, {'regex': <_sre.SRE_Pattern object at 0x7fa4558cdb28>, 'level': 'critical'}, {'substr': 'ERROR - ', 'level': 'error'}], 'cwd': '/builds/worker/workspace/build', 'privileged': False}, attempt #1
[task 2019-05-31T20:48:26.862Z] 20:48:26 INFO - Running command: ['/builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python', '-u', '/builds/worker/checkouts/gecko/mach', 'artifact', 'toolchain', '-v', '--tooltool-url', 'https://tooltool.mozilla-releng.net/', '--tooltool-manifest', '/builds/worker/workspace/build/tooltool.tt', '--artifact-manifest', '/builds/worker/checkouts/gecko/toolchains.json', '--cache-dir', '/builds/worker/tooltool-cache'] in /builds/worker/workspace/build
[task 2019-05-31T20:48:26.862Z] 20:48:26 INFO - Copy/paste: /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python -u /builds/worker/checkouts/gecko/mach artifact toolchain -v --tooltool-url https://tooltool.mozilla-releng.net/ --tooltool-manifest /builds/worker/workspace/build/tooltool.tt --artifact-manifest /builds/worker/checkouts/gecko/toolchains.json --cache-dir /builds/worker/tooltool-cache
[task 2019-05-31T20:48:26.862Z] 20:48:26 INFO - Calling ['/builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python', '-u', '/builds/worker/checkouts/gecko/mach', 'artifact', 'toolchain', '-v', '--tooltool-url', 'https://tooltool.mozilla-releng.net/', '--tooltool-manifest', '/builds/worker/workspace/build/tooltool.tt', '--artifact-manifest', '/builds/worker/checkouts/gecko/toolchains.json', '--cache-dir', '/builds/worker/tooltool-cache'] with output_timeout 600
[task 2019-05-31T20:48:27.223Z] 20:48:27 INFO - 0:00.29 materialized mozbuild.action.tooltool.FileRecord(filename='android-sdk_r29.0.11.0-linux-x86emu.tar.gz', size=330652164, digest='954d6c7ecf3e10468ae0ca8d97f930eb1e1665ddf5d9317dd4bb8fbc13271cf12c4e343170aee782c33f1b6e15e5915f62c9e4a2a66eb32cc0b919cd6fb9659b', algorithm='sha512', visibility=u'internal')
[task 2019-05-31T20:48:27.223Z] 20:48:27 INFO - 0:00.29 loaded manifest from file '/builds/worker/workspace/build/tooltool.tt'
[task 2019-05-31T20:48:27.223Z] 20:48:27 INFO - 0:00.29 Downloading android-sdk_r29.0.11.0-linux-x86emu.tar.gz
[task 2019-05-31T20:48:27.224Z] 20:48:27 INFO - 0:00.29 attempt 1/5
[task 2019-05-31T20:48:27.224Z] 20:48:27 INFO - 0:00.29 Downloading to temporary location /builds/worker/tooltool-cache/954d6c7ecf3e10468ae0ca8d97f930eb1e1665ddf5d9317dd4bb8fbc13271cf12c4e343170aee782c33f1b6e15e5915f62c9e4a2a66eb32cc0b919cd6fb9659b
[task 2019-05-31T20:48:27.586Z] 20:48:27 INFO - 0:00.65 Downloading... 0.0 %
[task 2019-05-31T20:48:27.586Z] 20:48:27 INFO - 0:00.65 Downloading... 100.0 %
[task 2019-05-31T20:48:27.586Z] 20:48:27 INFO - 0:00.65 403 Client Error: FORBIDDEN for url: https://tooltool.mozilla-releng.net/sha512/954d6c7ecf3e10468ae0ca8d97f930eb1e1665ddf5d9317dd4bb8fbc13271cf12c4e343170aee782c33f1b6e15e5915f62c9e4a2a66eb32cc0b919cd6fb9659b
[task 2019-05-31T20:48:27.586Z] 20:48:27 INFO - 0:00.65 Failed to download android-sdk_r29.0.11.0-linux-x86emu.tar.gz
[task 2019-05-31T20:48:27.630Z] 20:48:27 ERROR - Return code: 1
I see that the wrench tests have taskcluster configuration for internal tooltool downloads:
and I can't think of what else might be going wrong.
Comment 2•7 years ago
|
||
That's really weird. I can't think of anything that would cause this problem, assuming the new emulator was uploaded to tooltool with the same permissions as the old one.
| Reporter | ||
Comment 3•7 years ago
|
||
[task 2019-05-31T20:48:27.586Z] 20:48:27 INFO - 0:00.65 Downloading... 100.0 %
[task 2019-05-31T20:48:27.586Z] 20:48:27 INFO - 0:00.65 403 Client Error: FORBIDDEN for url: https://tooltool.mozilla-releng.net/sha512/954d6c7ecf3e10468ae0ca8d97f930eb1e1665ddf5d9317dd4bb8fbc13271cf12c4e343170aee782c33f1b6e15e5915f62c9e4a2a66eb32cc0b919cd6fb9659b
Does this mean that the file was downloaded but the cache couldn't be updated? Because of file permissions?
:dustin - Do you know what is going wrong here?
Comment 4•7 years ago
|
||
That's a 403 so unlikely to be related to the filesystem. Rok, any ideas?
| Reporter | ||
Updated•7 years ago
|
Comment 5•7 years ago
•
|
||
Hi, sorry for long response time. This is strange since you didn't change the visibility. I'm not familiar what provides the authentication there, since i would expect that relengapi-proxy or taskcluster-proxy would be used. Does somebody know how token is provided there?
Comment 6•7 years ago
|
||
at the closer look it looks like no authentication is being used which explains why it is failing. i'm assuming this was working because it was cached somehow before.
:gbrown i need to talk to the rest of the releng team how should this be fixed since it looks like this is not the only case.
| Reporter | ||
Comment 7•7 years ago
|
||
Thanks :garbas.
I note the failed wrench task had scopes including
docker-worker:relengapi-proxy:tooltool.download.public
docker-worker:relengapi-proxy:tooltool.download.internal
and also
"features": {
"relengAPIProxy": true,
"taskclusterProxy": true
},
A difference between the failed task and the other Android 7.0 tasks which succeed is that the wrench task has a source checkout, so it downloads using 'mach artifact' whereas the successful tasks run tooltool.py. Maybe that's significant?
I tried to check into the tooltool authentication file here:
From the debug logging, it looks like both the 'mach artifact' and tooltool.py cases try to use /builds/relengapi.tok, but do not find that file.
Although all the wrench tasks failed in this new try push, they all failed differently from before: Now the sdk is being downloaded successfully! Since this failure appears to no longer be reproducible, I think it would be okay to resolve this bug.
Updated•7 years ago
|
Description
•