1533330 - deprecate tokens service and use taskcluster credentials

Status: RESOLVED FIXED

(Release Engineering :: General, enhancement)

Comment 1

[Rok Garbas [:garbas]]

Attachment: Bug 1533330 - Update tooltool.py with optional support of taskcluster auth tokens

With this change tooltool.py also supports taskcluster credentials to be passed
(in json format) to --authentication-file option. RelengAPI tokens are still
working with this patch, just additional authentication is added.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d99a232c3dd4
Update tooltool.py with optional support of taskcluster auth tokens r=rail

Comment 3

[Cosmin Sabou [:CosminS]]

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=busted&fromchange=d99a232c3dd4275f725252ac6629e305550402d3&tochange=0c8058d19e04d79cc7b78a8a8f616f10b66c33b3&selectedJob=242257408

Failure log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=242257361&repo=autoland&lineNumber=876
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=242257344&repo=autoland&lineNumber=1072

Backout link: https://hg.mozilla.org/integration/autoland/rev/0c8058d19e04d79cc7b78a8a8f616f10b66c33b3

[task 2019-04-24T10:09:55.663Z] 10:09:55 INFO - 8:14.19 Downloading... 100.0 %
[task 2019-04-24T10:09:55.665Z] 10:09:55 INFO - 8:14.19 Downloaded artifact to /builds/worker/.mozbuild/toolchains/16ce7bdceeeac39a-binutils.tar.xz
[task 2019-04-24T10:09:55.670Z] 10:09:55 INFO - 8:14.19 Failed to download binutils.tar.xz
[task 2019-04-24T10:09:55.687Z] 10:09:55 ERROR - Return code: 1
[task 2019-04-24T10:09:55.687Z] 10:09:55 ERROR - 1 not in success codes: [0]
[task 2019-04-24T10:09:55.687Z] 10:09:55 WARNING - setting return code to 2
[task 2019-04-24T10:09:55.687Z] 10:09:55 FATAL - Halting on failure while running ['/builds/worker/workspace/build/src/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python', '-u', '/builds/worker/workspace/build/src/mach', 'artifact', 'toolchain', '-v', '--retry', '4', '--artifact-manifest', '/builds/worker/workspace/build/src/toolchains.json', 'public/build/binutils.tar.xz@K5NcyYbZRF25fJF8XzJLoA', 'public/build/clang.tar.xz@eStm6oExQQeX8r3oKfPb-g', 'public/build/cbindgen.tar.xz@bPEue9VVQ_eRjPuRyJhmOA', 'public/build/sccache2.tar.xz@UsjdGlSnTpuzCwJUZuyeuA', 'public/build/rustc.tar.xz@B_aqp8pbTHaRKrXELzeK9w', 'public/build/rust-size.tar.xz@IlyF0MjpRDOhehriA_ZOnQ', 'public/build/nasm.tar.bz2@N94LYZSCRO-GGhFunBYWAw', 'public/build/node.tar.xz@EXixwTpEQ9CLAA2Auou8wg']
[task 2019-04-24T10:09:55.687Z] 10:09:55 FATAL - Running post_fatal callback...
[task 2019-04-24T10:09:55.687Z] 10:09:55 FATAL - Exiting 2
[task 2019-04-24T10:09:55.687Z] 10:09:55 INFO - [mozharness: 2019-04-24 10:09:55.687625Z] Finished build step (failed)
[task 2019-04-24T10:09:55.687Z] 10:09:55 INFO - Running post-run listener: _parse_build_tests_ccov
[task 2019-04-24T10:09:55.687Z] 10:09:55 INFO - Running post-run listener: _shutdown_sccache
[task 2019-04-24T10:09:55.687Z] 10:09:55 INFO - Running command: ['/builds/worker/workspace/build/src/sccache2/sccache', '--stop-server'] in /builds/worker/workspace/build/src
[task 2019-04-24T10:09:55.687Z] 10:09:55 INFO - Copy/paste: /builds/worker/workspace/build/src/sccache2/sccache --stop-server
[task 2019-04-24T10:09:55.690Z] 10:09:55 ERROR - caught OS error 2: No such file or directory while running ['/builds/worker/workspace/build/src/sccache2/sccache', '--stop-server']
[task 2019-04-24T10:09:55.690Z] 10:09:55 INFO - Running post-run listener: _summarize
[task 2019-04-24T10:09:55.690Z] 10:09:55 ERROR - # TBPL FAILURE #
[task 2019-04-24T10:09:55.690Z] 10:09:55 INFO - [mozharness: 2019-04-24 10:09:55.690701Z] FxDesktopBuild summary:
[task 2019-04-24T10:09:55.690Z] 10:09:55 ERROR - # TBPL FAILURE #

Comment 4

[Rok Garbas [:garbas]]

:CosminS thank you for reverting. I'm already looking into what went wrong. This was working on monday on try. Maybe a rebase I had to do messed up something.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/626ad97710e3
Update tooltool.py with optional support of taskcluster auth tokens r=rail

Comment 6

[Rok Garbas [:garbas]]

:dhouse I was told to ping you regarding VCSSync[1], since we hope to deprecate relengapi tokens (in favor of taskcluster authentication) as soon as possible. Are you the right person to talk to about adding taskcluster authentication support to VCSSync scripts?

I already did this for tooltool.py and if you point me to the source code of VCSSync I can come up with a PR to add tasckluster authentication.

[1] https://wiki.mozilla.org/ReleaseEngineering/How_To/VCSSync#What.27s_running_where

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/626ad97710e3

Comment 8

[Rok Garbas [:garbas]]

reopening since we need to land more things, but i do it in multiple steps. I see now that I need to create a separate issue for each landing (will do this in the future)

Comment 9

[:dhouse]

(In reply to Rok Garbas [:garbas] from comment #6)

:dhouse I was told to ping you regarding VCSSync[1], since we hope to deprecate relengapi tokens (in favor of taskcluster authentication) as soon as possible. Are you the right person to talk to about adding taskcluster authentication support to VCSSync scripts?

I already did this for tooltool.py and if you point me to the source code of VCSSync I can come up with a PR to add tasckluster authentication.

[1] https://wiki.mozilla.org/ReleaseEngineering/How_To/VCSSync#What.27s_running_where

Thanks Rok! Here is what we're running right now: https://hg.mozilla.org/build/mozharness/file/tip/scripts/vcs-sync/vcs_sync.py#l985

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Rok -- is there work to do here before the November 9 TCW?

Comment 11

[Rok Garbas [:garbas]]

This is now done and tooltool is running without "old" tokens for around 3 weeks.