Closed Bug 1701137 Opened 9 months ago Closed 8 months ago

Perma Sentry [Tier2] error: API request failed

Categories

(Firefox Build System :: Task Configuration, defect)

defect

Tracking

(firefox-esr78 unaffected, firefox86 unaffected, firefox87 unaffected, firefox88 unaffected, firefox89 fixed)

RESOLVED FIXED
89 Branch
Tracking Status
firefox-esr78 --- unaffected
firefox86 --- unaffected
firefox87 --- unaffected
firefox88 --- unaffected
firefox89 --- fixed

People

(Reporter: NarcisB, Assigned: mhentges)

References

(Regression)

Details

(Keywords: regression)

Attachments

(1 file, 1 obsolete file)

Push: https://treeherder.mozilla.org/jobs?repo=mozilla-central&selectedTaskRun=Icl1iAlzT8OuXIZ7aZtuLQ.0&revision=cad5e739410ba4a3d90f8fd6f921616b77f08f68&searchStr=other%2Copt%2Csentry%2Csynchronization%2Csentry-mach-release%2Cmach-release
Full log : https://firefoxci.taskcluster-artifacts.net/Icl1iAlzT8OuXIZ7aZtuLQ/0/public/logs/live_backing.log

Log snippet:

[setup 2021-03-26T04:19:12.009Z] run-task started in /builds/worker
[volume 2021-03-26T04:19:12.010Z] changing ownership of volume /builds/worker/checkouts to 1000:1000
[setup 2021-03-26T04:19:12.010Z] running as worker:worker
[task 2021-03-26T04:19:12.011Z] executing ['bash', '-cx', '/bin/bash /usr/bin/submit_sentry_release.sh']
[task 2021-03-26T04:19:12.012Z] + /bin/bash /usr/bin/submit_sentry_release.sh
[task 2021-03-26T04:19:12.019Z]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[task 2021-03-26T04:19:12.019Z]                                  Dload  Upload   Total   Spent    Left  Speed
[task 2021-03-26T04:19:12.019Z] 
[task 2021-03-26T04:19:12.254Z]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[task 2021-03-26T04:19:12.254Z] 100  4425    0  4425    0     0  18829      0 --:--:-- --:--:-- --:--:-- 18829
[task 2021-03-26T04:19:12.261Z]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[task 2021-03-26T04:19:12.261Z]                                  Dload  Upload   Total   Spent    Left  Speed
[task 2021-03-26T04:19:12.261Z] 
[task 2021-03-26T04:19:12.338Z]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[task 2021-03-26T04:19:12.338Z] 100   116  100   116    0     0   1506      0 --:--:-- --:--:-- --:--:--  1526
[task 2021-03-26T04:19:12.572Z] error: API request failed
[task 2021-03-26T04:19:12.572Z]   caused by: sentry reported an error: API key is not valid (http status: 401)
[task 2021-03-26T04:19:12.572Z] 
[task 2021-03-26T04:19:12.572Z] Add --log-level=[info|debug] or export SENTRY_LOG_LEVEL=[info|debug] to see more output.
[task 2021-03-26T04:19:12.572Z] Please attach the full debug log to all bug reports.
[task 2021-03-26T04:19:12.574Z] [with_backoff] Retrying in 1 second(s)
[task 2021-03-26T04:19:13.580Z]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[task 2021-03-26T04:19:13.580Z]                                  Dload  Upload   Total   Spent    Left  Speed
[task 2021-03-26T04:19:13.580Z] 
[task 2021-03-26T04:19:13.796Z]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[task 2021-03-26T04:19:13.796Z] 100  4425    0  4425    0     0  20581      0 --:--:-- --:--:-- --:--:-- 20581
[task 2021-03-26T04:19:13.802Z]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[task 2021-03-26T04:19:13.803Z]                                  Dload  Upload   Total   Spent    Left  Speed
[task 2021-03-26T04:19:13.803Z] 
[task 2021-03-26T04:19:13.831Z]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[task 2021-03-26T04:19:13.831Z] 100   116  100   116    0     0   4142      0 --:--:-- --:--:-- --:--:--  4142
[task 2021-03-26T04:19:13.926Z] error: API request failed
[task 2021-03-26T04:19:13.926Z]   caused by: sentry reported an error: API key is not valid (http status: 401)
[task 2021-03-26T04:19:13.926Z] 
[task 2021-03-26T04:19:13.926Z] Add --log-level=[info|debug] or export SENTRY_LOG_LEVEL=[info|debug] to see more output.
[task 2021-03-26T04:19:13.926Z] Please attach the full debug log to all bug reports.
[task 2021-03-26T04:19:13.928Z] [with_backoff] Retrying in 2 second(s)
[task 2021-03-26T04:19:15.934Z]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[task 2021-03-26T04:19:15.934Z]                                  Dload  Upload   Total   Spent    Left  Speed
[task 2021-03-26T04:19:15.934Z] 
[task 2021-03-26T04:19:16.195Z]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[task 2021-03-26T04:19:16.196Z] 100  4425    0  4425    0     0  16954      0 --:--:-- --:--:-- --:--:-- 16954
[task 2021-03-26T04:19:16.203Z]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[task 2021-03-26T04:19:16.203Z]                                  Dload  Upload   Total   Spent    Left  Speed
[task 2021-03-26T04:19:16.203Z] 
[task 2021-03-26T04:19:16.230Z]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[task 2021-03-26T04:19:16.230Z] 100   116  100   116    0     0   4296      0 --:--:-- --:--:-- --:--:--  4296
[task 2021-03-26T04:19:16.351Z] error: API request failed
[task 2021-03-26T04:19:16.351Z]   caused by: sentry reported an error: API key is not valid (http status: 401)
[task 2021-03-26T04:19:16.351Z] 
[task 2021-03-26T04:19:16.351Z] Add --log-level=[info|debug] or export SENTRY_LOG_LEVEL=[info|debug] to see more output.
[task 2021-03-26T04:19:16.351Z] Please attach the full debug log to all bug reports.
[task 2021-03-26T04:19:16.353Z] [with_backoff] Retrying in 4 second(s)
[task 2021-03-26T04:19:20.360Z]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[task 2021-03-26T04:19:20.360Z]                                  Dload  Upload   Total   Spent    Left  Speed
[task 2021-03-26T04:19:20.360Z] 
[task 2021-03-26T04:19:20.469Z]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[task 2021-03-26T04:19:20.595Z]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[task 2021-03-26T04:19:20.595Z] 100  4425    0  4425    0     0  18829      0 --:--:-- --:--:-- --:--:-- 18750
[task 2021-03-26T04:19:20.602Z]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[task 2021-03-26T04:19:20.602Z]                                  Dload  Upload   Total   Spent    Left  Speed
[task 2021-03-26T04:19:20.602Z] 
[task 2021-03-26T04:19:20.630Z]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[task 2021-03-26T04:19:20.630Z] 100   116  100   116    0     0   4142      0 --:--:-- --:--:-- --:--:--  4142
[task 2021-03-26T04:19:20.718Z] error: API request failed
[task 2021-03-26T04:19:20.718Z]   caused by: sentry reported an error: API key is not valid (http status: 401)
[task 2021-03-26T04:19:20.718Z] 
[task 2021-03-26T04:19:20.718Z] Add --log-level=[info|debug] or export SENTRY_LOG_LEVEL=[info|debug] to see more output.
[task 2021-03-26T04:19:20.718Z] Please attach the full debug log to all bug reports.
[task 2021-03-26T04:19:20.720Z] [with_backoff] Retrying in 8 second(s)
[task 2021-03-26T04:19:28.726Z]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[task 2021-03-26T04:19:28.726Z]                                  Dload  Upload   Total   Spent    Left  Speed
[task 2021-03-26T04:19:28.726Z] 
[task 2021-03-26T04:19:28.915Z]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[task 2021-03-26T04:19:28.915Z] 100  4425    0  4425    0     0  23537      0 --:--:-- --:--:-- --:--:-- 23537
[task 2021-03-26T04:19:28.922Z]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[task 2021-03-26T04:19:28.922Z]                                  Dload  Upload   Total   Spent    Left  Speed
[task 2021-03-26T04:19:28.922Z] 
[task 2021-03-26T04:19:28.990Z]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
[task 2021-03-26T04:19:28.990Z] 100   116  100   116    0     0   1731      0 --:--:-- --:--:-- --:--:--  1731
[task 2021-03-26T04:19:29.063Z] error: API request failed
[task 2021-03-26T04:19:29.063Z]   caused by: sentry reported an error: API key is not valid (http status: 401)
[task 2021-03-26T04:19:29.063Z] 
[task 2021-03-26T04:19:29.063Z] Add --log-level=[info|debug] or export SENTRY_LOG_LEVEL=[info|debug] to see more output.
[task 2021-03-26T04:19:29.063Z] Please attach the full debug log to all bug reports.
[task 2021-03-26T04:19:29.065Z] [with_backoff] Unable to succeed after 5 tries, failing the job.
[taskcluster 2021-03-26 04:19:29.386Z] === Task Finished ===
[taskcluster 2021-03-26 04:19:29.386Z] Unsuccessful task run with exit code: 1 completed in 27.548 seconds
Assignee: wezhou → nobody
Component: Operations: Sentry → Task Configuration
Flags: needinfo?(mhentges)
Product: Cloud Services → Firefox Build System
QA Contact: wezhou
Regressed by: 1698511
Assignee: nobody → mhentges
Status: NEW → ASSIGNED
Flags: needinfo?(mhentges)

Taskcluster secrets are not provided to clients in the same
format that they are set. Rather, the YAML is rendered
to JSON, and it's wrapped in a larger object that also
contains the expires timestamp.

Correctly parse the JSON and remove the dependency on yq.

Attachment #9211798 - Attachment description: WIP: Bug 1701137: Properly parse taskcluster Sentry secret → Bug 1701137: Properly parse taskcluster Sentry secret
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6e52d2a6585d
Properly parse taskcluster Sentry secret r=aki
Keywords: leave-open

The task still failed after landing >:(
Testing in prod sucks.

Depends on: 1701359

The task still failed after landing >:(

That was looking at this push.
It turns out that the next push succeeded! I guess the first push didn't have my changes in it yet, which I didn't expect - I thought that merges to mozilla-central contained all un-landed changes.

Status: ASSIGNED → RESOLVED
Closed: 8 months ago
Resolution: --- → FIXED

Automatically use the level 1 or level 3 secret
depending on context.

Comment on attachment 9212563 [details]
Bug 1701137: Sentry task should work on try

Revision D110279 was moved to bug 1701979. Setting attachment 9212563 [details] to obsolete.

Attachment #9212563 - Attachment is obsolete: true

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #7)

The task still failed after landing >:(

That was looking at this push.
It turns out that the next push succeeded! I guess the first push didn't have my changes in it yet, which I didn't expect - I thought that merges to mozilla-central contained all un-landed changes.

Ah, yeah. Sheriffs find merge candidates by which pushes look green on autoland, which is very often not the tip of autoland.

Target Milestone: --- → 89 Branch
You need to log in before you can comment on or make changes to this bug.