1569817 - Android wrench bustages - Tooltool /builds/worker/workspace/build/tooltool.tt fetch failed! | 403 Client Error: FORBIDDEN for url: https://tooltool.mozilla-releng.net/sha512/e62acc ...

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Cosmin Sabou [:CosminS]]

Autoland is closed so far for this issue on Android wrench builds: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&resultStatus=success%2Cbusted%2Cexception&searchStr=wrench&fromchange=d6ce560581af17b5aadc7957d5c0472faa27ccb6&tochange=46208314f5af174efdfa81fb8e57238911246fe5&selectedJob=258879967

First push where it started: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&selectedJob=258879967&resultStatus=pending%2Crunning%2Csuccess%2Cbusted%2Cexception&searchStr=wrench&revision=e7cba26781e9e2bf66f84d9c9f4aa507be6d25bf

Failure logs: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=258885025&repo=autoland&lineNumber=1175
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=258881835&repo=autoland&lineNumber=700

[task 2019-07-29T22:46:28.844Z] 22:46:28 INFO - Copy/paste: /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python -u /builds/worker/checkouts/gecko/mach artifact toolchain -v --tooltool-url https://tooltool.mozilla-releng.net/ --tooltool-manifest /builds/worker/workspace/build/tooltool.tt --artifact-manifest /builds/worker/checkouts/gecko/toolchains.json --cache-dir /builds/worker/tooltool-cache
[task 2019-07-29T22:46:28.844Z] 22:46:28 INFO - Calling ['/builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python', '-u', '/builds/worker/checkouts/gecko/mach', 'artifact', 'toolchain', '-v', '--tooltool-url', 'https://tooltool.mozilla-releng.net/', '--tooltool-manifest', '/builds/worker/workspace/build/tooltool.tt', '--artifact-manifest', '/builds/worker/checkouts/gecko/toolchains.json', '--cache-dir', '/builds/worker/tooltool-cache'] with output_timeout 600
[task 2019-07-29T22:46:29.169Z] 22:46:29 INFO - 0:00.26 materialized mozbuild.action.tooltool.FileRecord(filename='android-sdk_r28.0.25.0-linux-x86emu.tar.gz', size=241459387, digest='e62acc91f41ccef65a4937a2672fcb56362e9946b806bacc25854035b57d5bd2d525a9c7d660a643ab6381ae2e3b660be7fea70e302ed314c4b07880b2328e18', algorithm='sha512', visibility=u'internal')
[task 2019-07-29T22:46:29.169Z] 22:46:29 INFO - 0:00.26 loaded manifest from file '/builds/worker/workspace/build/tooltool.tt'
[task 2019-07-29T22:46:29.169Z] 22:46:29 INFO - 0:00.26 Downloading android-sdk_r28.0.25.0-linux-x86emu.tar.gz
[task 2019-07-29T22:46:29.169Z] 22:46:29 INFO - 0:00.26 attempt 1/5
[task 2019-07-29T22:46:29.169Z] 22:46:29 INFO - 0:00.26 Downloading to temporary location /builds/worker/tooltool-cache/e62acc91f41ccef65a4937a2672fcb56362e9946b806bacc25854035b57d5bd2d525a9c7d660a643ab6381ae2e3b660be7fea70e302ed314c4b07880b2328e18
[task 2019-07-29T22:46:29.225Z] 22:46:29 INFO - 0:00.31 Downloading... 0.0 %
[task 2019-07-29T22:46:29.225Z] 22:46:29 INFO - 0:00.31 Downloading... 100.0 %
[task 2019-07-29T22:46:29.225Z] 22:46:29 INFO - 0:00.32 403 Client Error: FORBIDDEN for url: https://tooltool.mozilla-releng.net/sha512/e62acc91f41ccef65a4937a2672fcb56362e9946b806bacc25854035b57d5bd2d525a9c7d660a643ab6381ae2e3b660be7fea70e302ed314c4b07880b2328e18
[task 2019-07-29T22:46:29.225Z] 22:46:29 INFO - 0:00.32 Failed to download android-sdk_r28.0.25.0-linux-x86emu.tar.gz
[task 2019-07-29T22:46:29.261Z] 22:46:29 ERROR - Return code: 1
[task 2019-07-29T22:46:29.261Z] 22:46:29 FATAL - Tooltool /builds/worker/workspace/build/tooltool.tt fetch failed!
[task 2019-07-29T22:46:29.261Z] 22:46:29 FATAL - Running post_fatal callback...
[task 2019-07-29T22:46:29.261Z] 22:46:29 FATAL - Exiting -1
[task 2019-07-29T22:46:29.282Z] cleanup
[task 2019-07-29T22:46:29.283Z] + cleanup

Comment 1

[Cosmin Sabou [:CosminS]]

This is something that happened also 21 hrs ago: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=bdb7f8baac7e2fccf5c4ce76ea6a76ccb0bd6126&searchStr=wrench&selectedJob=258726835
Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=258726835&repo=autoland&lineNumber=1178

Comment 2

[Cosmin Sabou [:CosminS]]

This rerun is green: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&resultStatus=pending%2Crunning%2Csuccess%2Cbusted%2Cexception&searchStr=wrench&fromchange=d6ce560581af17b5aadc7957d5c0472faa27ccb6&tochange=1800c95a395b06d3f188a25e64e2dfbafd8e1e00&selectedJob=258889937
If the other reruns are green too will reopen the tree and downgrade the importance.

Update: only one rerun was green, all the others failed: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&resultStatus=pending%2Crunning%2Csuccess%2Cbusted%2Cexception&searchStr=wrench&tochange=46208314f5af174efdfa81fb8e57238911246fe5&fromchange=e7cba26781e9e2bf66f84d9c9f4aa507be6d25bf&selectedJob=258883163

Comment 3

[Cosmin Sabou [:CosminS]]

All the Android 7.0 x86 failures have 403 Client Error: FORBIDDEN for url: https://tooltool.mozilla-releng.net/sha512/e62acc91f41ccef65a4937a2672fcb56362e9946b806bacc25854035b57d5bd2d525a9c7d660a643ab6381ae2e3b660be7fea70e302ed314c4b07880b2328e18
Failed to download android-sdk_r28.0.25.0-linux-x86emu.tar.gz
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=258879966&repo=autoland&lineNumber=1109

The Android 8.0 Pixel2 ones have: python2.7: can't open file '/builds/task_1564443136//builds/task_1564443136/fetches/mozharness/scripts/android_wrench.py': [Errno 2] No such file or directory
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=258888171&repo=autoland&lineNumber=593

Comment 4

[Geoff Brown [:gbrown]]

:glandium - Any chance this is related to bug 1568460? (I don't know much about the wrench tasks, but I see "fetches" in comment 3 and this seemed to start after-ish bug 1568460 landed.)

Comment 5

[Cosmin Sabou [:CosminS]]

This does seem related because when it landed yesterday https://bugzilla.mozilla.org/show_bug.cgi?id=1568460#c2 there were also bustages on https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=wrench&tochange=bdb7f8baac7e2fccf5c4ce76ea6a76ccb0bd6126&fromchange=a2c0bed45b89a5d02424841507cb0fd3b57c8aaf&selectedJob=258726614

yesterday https://tinyurl.com/yxgryqor and today https://tinyurl.com/yy34xy79

Comment 6

[Tom Prince [:tomprince]]

Attachment: Bug 1569817: Use taskcluster proxy to access tooltool in android wrench jobs; r?glandium

Comment 7

[Mike Hommey [:glandium]]

Attachment: Bug 1569817 - Set tooltool servers for wrench tasks.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2a0f5a0e0e38
Set tooltool servers for wrench tasks.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8147d3db9d52
Backed out changeset 2a0f5a0e0e38 for landing without a reviewer in the commit message.
https://hg.mozilla.org/integration/autoland/rev/00d1d637d22a
Set tooltool servers for wrench tasks. r=tomprince

Comment 10

[Cosmin Sabou [:CosminS]]

Reopened autoland. https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&resultStatus=success%2Cpending%2Crunning%2Ctestfailed%2Cbusted%2Cexception&revision=00d1d637d22a8d9fe6f0f391a7cdee874f511e8d&searchStr=wrench

Comment 11

[Intermittent Failures Robot]

31 failures in 869 pushes (0.036 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 31

Platform breakdown:

• android-em-7-0-x86: 13
• android-hw-p2-8-0-arm7-api-16: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1569817&startday=2019-07-29&endday=2019-07-29&tree=all

Comment 12

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/00d1d637d22a

Comment 13

[Intermittent Failures Robot]

31 failures in 4439 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 31

Platform breakdown:

• android-em-7-0-x86: 13
• android-hw-p2-8-0-arm7-api-16: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1569817&startday=2019-07-29&endday=2019-08-04&tree=all