1701137 - Perma Sentry [Tier2] error: API request failed

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect)

Description

[Narcis Beleuzu [:NarcisB]]

Push: https://treeherder.mozilla.org/jobs?repo=mozilla-central&selectedTaskRun=Icl1iAlzT8OuXIZ7aZtuLQ.0&revision=cad5e739410ba4a3d90f8fd6f921616b77f08f68&searchStr=other%2Copt%2Csentry%2Csynchronization%2Csentry-mach-release%2Cmach-release
Full log : https://firefoxci.taskcluster-artifacts.net/Icl1iAlzT8OuXIZ7aZtuLQ/0/public/logs/live_backing.log

Log snippet:

[setup 2021-03-26T04:19:12.009Z] run-task started in /builds/worker
[volume 2021-03-26T04:19:12.010Z] changing ownership of volume /builds/worker/checkouts to 1000:1000
[setup 2021-03-26T04:19:12.010Z] running as worker:worker
[task 2021-03-26T04:19:12.011Z] executing ['bash', '-cx', '/bin/bash /usr/bin/submit_sentry_release.sh']
[task 2021-03-26T04:19:12.012Z] + /bin/bash /usr/bin/submit_sentry_release.sh
[task 2021-03-26T04:19:12.019Z]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[task 2021-03-26T04:19:12.019Z]                                  Dload  Upload   Total   Spent    Left  Speed
[task 2021-03-26T04:19:12.019Z] 
[task 2021-03-26T04:19:12.254Z]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[task 2021-03-26T04:19:12.254Z] 100  4425    0  4425    0     0  18829      0 --:--:-- --:--:-- --:--:-- 18829
[task 2021-03-26T04:19:12.261Z]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[task 2021-03-26T04:19:12.261Z]                                  Dload  Upload   Total   Spent    Left  Speed
[task 2021-03-26T04:19:12.261Z] 
[task 2021-03-26T04:19:12.338Z]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[task 2021-03-26T04:19:12.338Z] 100   116  100   116    0     0   1506      0 --:--:-- --:--:-- --:--:--  1526
[task 2021-03-26T04:19:12.572Z] error: API request failed
[task 2021-03-26T04:19:12.572Z]   caused by: sentry reported an error: API key is not valid (http status: 401)
[task 2021-03-26T04:19:12.572Z] 
[task 2021-03-26T04:19:12.572Z] Add --log-level=[info|debug] or export SENTRY_LOG_LEVEL=[info|debug] to see more output.
[task 2021-03-26T04:19:12.572Z] Please attach the full debug log to all bug reports.
[task 2021-03-26T04:19:12.574Z] [with_backoff] Retrying in 1 second(s)
[task 2021-03-26T04:19:13.580Z]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[task 2021-03-26T04:19:13.580Z]                                  Dload  Upload   Total   Spent    Left  Speed
[task 2021-03-26T04:19:13.580Z] 
[task 2021-03-26T04:19:13.796Z]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[task 2021-03-26T04:19:13.796Z] 100  4425    0  4425    0     0  20581      0 --:--:-- --:--:-- --:--:-- 20581
[task 2021-03-26T04:19:13.802Z]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[task 2021-03-26T04:19:13.803Z]                                  Dload  Upload   Total   Spent    Left  Speed
[task 2021-03-26T04:19:13.803Z] 
[task 2021-03-26T04:19:13.831Z]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[task 2021-03-26T04:19:13.831Z] 100   116  100   116    0     0   4142      0 --:--:-- --:--:-- --:--:--  4142
[task 2021-03-26T04:19:13.926Z] error: API request failed
[task 2021-03-26T04:19:13.926Z]   caused by: sentry reported an error: API key is not valid (http status: 401)
[task 2021-03-26T04:19:13.926Z] 
[task 2021-03-26T04:19:13.926Z] Add --log-level=[info|debug] or export SENTRY_LOG_LEVEL=[info|debug] to see more output.
[task 2021-03-26T04:19:13.926Z] Please attach the full debug log to all bug reports.
[task 2021-03-26T04:19:13.928Z] [with_backoff] Retrying in 2 second(s)
[task 2021-03-26T04:19:15.934Z]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[task 2021-03-26T04:19:15.934Z]                                  Dload  Upload   Total   Spent    Left  Speed
[task 2021-03-26T04:19:15.934Z] 
[task 2021-03-26T04:19:16.195Z]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[task 2021-03-26T04:19:16.196Z] 100  4425    0  4425    0     0  16954      0 --:--:-- --:--:-- --:--:-- 16954
[task 2021-03-26T04:19:16.203Z]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[task 2021-03-26T04:19:16.203Z]                                  Dload  Upload   Total   Spent    Left  Speed
[task 2021-03-26T04:19:16.203Z] 
[task 2021-03-26T04:19:16.230Z]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[task 2021-03-26T04:19:16.230Z] 100   116  100   116    0     0   4296      0 --:--:-- --:--:-- --:--:--  4296
[task 2021-03-26T04:19:16.351Z] error: API request failed
[task 2021-03-26T04:19:16.351Z]   caused by: sentry reported an error: API key is not valid (http status: 401)
[task 2021-03-26T04:19:16.351Z] 
[task 2021-03-26T04:19:16.351Z] Add --log-level=[info|debug] or export SENTRY_LOG_LEVEL=[info|debug] to see more output.
[task 2021-03-26T04:19:16.351Z] Please attach the full debug log to all bug reports.
[task 2021-03-26T04:19:16.353Z] [with_backoff] Retrying in 4 second(s)
[task 2021-03-26T04:19:20.360Z]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[task 2021-03-26T04:19:20.360Z]                                  Dload  Upload   Total   Spent    Left  Speed
[task 2021-03-26T04:19:20.360Z] 
[task 2021-03-26T04:19:20.469Z]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[task 2021-03-26T04:19:20.595Z]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[task 2021-03-26T04:19:20.595Z] 100  4425    0  4425    0     0  18829      0 --:--:-- --:--:-- --:--:-- 18750
[task 2021-03-26T04:19:20.602Z]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[task 2021-03-26T04:19:20.602Z]                                  Dload  Upload   Total   Spent    Left  Speed
[task 2021-03-26T04:19:20.602Z] 
[task 2021-03-26T04:19:20.630Z]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[task 2021-03-26T04:19:20.630Z] 100   116  100   116    0     0   4142      0 --:--:-- --:--:-- --:--:--  4142
[task 2021-03-26T04:19:20.718Z] error: API request failed
[task 2021-03-26T04:19:20.718Z]   caused by: sentry reported an error: API key is not valid (http status: 401)
[task 2021-03-26T04:19:20.718Z] 
[task 2021-03-26T04:19:20.718Z] Add --log-level=[info|debug] or export SENTRY_LOG_LEVEL=[info|debug] to see more output.
[task 2021-03-26T04:19:20.718Z] Please attach the full debug log to all bug reports.
[task 2021-03-26T04:19:20.720Z] [with_backoff] Retrying in 8 second(s)
[task 2021-03-26T04:19:28.726Z]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[task 2021-03-26T04:19:28.726Z]                                  Dload  Upload   Total   Spent    Left  Speed
[task 2021-03-26T04:19:28.726Z] 
[task 2021-03-26T04:19:28.915Z]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[task 2021-03-26T04:19:28.915Z] 100  4425    0  4425    0     0  23537      0 --:--:-- --:--:-- --:--:-- 23537
[task 2021-03-26T04:19:28.922Z]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[task 2021-03-26T04:19:28.922Z]                                  Dload  Upload   Total   Spent    Left  Speed
[task 2021-03-26T04:19:28.922Z] 
[task 2021-03-26T04:19:28.990Z]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[task 2021-03-26T04:19:28.990Z] 100   116  100   116    0     0   1731      0 --:--:-- --:--:-- --:--:--  1731
[task 2021-03-26T04:19:29.063Z] error: API request failed
[task 2021-03-26T04:19:29.063Z]   caused by: sentry reported an error: API key is not valid (http status: 401)
[task 2021-03-26T04:19:29.063Z] 
[task 2021-03-26T04:19:29.063Z] Add --log-level=[info|debug] or export SENTRY_LOG_LEVEL=[info|debug] to see more output.
[task 2021-03-26T04:19:29.063Z] Please attach the full debug log to all bug reports.
[task 2021-03-26T04:19:29.065Z] [with_backoff] Unable to succeed after 5 tries, failing the job.
[taskcluster 2021-03-26 04:19:29.386Z] === Task Finished ===
[taskcluster 2021-03-26 04:19:29.386Z] Unsuccessful task run with exit code: 1 completed in 27.548 seconds

Comment 2

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1701137: Properly parse taskcluster Sentry secret

Taskcluster secrets are not provided to clients in the same
format that they are set. Rather, the YAML is rendered
to JSON, and it's wrapped in a larger object that also
contains the expires timestamp.

Correctly parse the JSON and remove the dependency on yq.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6e52d2a6585d
Properly parse taskcluster Sentry secret r=aki

Comment 4

[Mitchell Hentges [:mhentges] 🦀]

The task still failed after landing >:(
Testing in prod sucks.

Comment 5

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/6e52d2a6585d

Comment 6

[Intermittent Failures Robot]

5 failures in 3967 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 5

Platform and build breakdown:

• other: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1701137&startday=2021-03-22&endday=2021-03-28&tree=all

Comment 7

[Mitchell Hentges [:mhentges] 🦀]

The task still failed after landing >:(

That was looking at this push.
It turns out that the next push succeeded! I guess the first push didn't have my changes in it yet, which I didn't expect - I thought that merges to mozilla-central contained all un-landed changes.

Comment 8

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1701137: Sentry task should work on `try`

Automatically use the level 1 or level 3 secret
depending on context.

Comment 9

[Phabricator Automation]

Comment on attachment 9212563 [details]
Bug 1701137: Sentry task should work on try

Revision D110279 was moved to bug 1701979. Setting attachment 9212563 [details] to obsolete.

Comment 10

[Aki Sasaki (not active)]

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #7)

The task still failed after landing >:(

That was looking at this push.
It turns out that the next push succeeded! I guess the first push didn't have my changes in it yet, which I didn't expect - I thought that merges to mozilla-central contained all un-landed changes.

Ah, yeah. Sheriffs find merge candidates by which pushes look green on autoland, which is very often not the tip of autoland.

Comment 11

[Intermittent Failures Robot]

1 failures in 3125 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• other: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1701137&startday=2021-03-29&endday=2021-04-04&tree=all