1740339 - ./mach taskgraph test-action-callback backfill fails: "Failed to trigger action for <push-id>"

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Run in a mozilla-unified folder:

export TASKCLUSTER_PROXY_URL=https://firefox-ci-tc.services.mozilla.com
eval $(./taskcluster signin)
./mach taskgraph actions > ../actions.json
./mach taskgraph test-action-callback --task-id KHJ6VY3rQS65WkqrfF2mtA --task-group-id RyBEkPCeRbe4cqNT12O4fA --parameters ../parameters.yml --input ../actions.json backfill

• --task-id is the one mentioned at the bottom left if a task is selected in Treeherder
• --task-group-id is the associated task group: Open the task link from Treeherder and then on "Task Group" in the navigation at the top.
• parameters.yml gets downloaded from the Gecko Decision Task for the push (haven't investigated if necessary or automatically identified and downloaded).

Backfills fail with:

 0:17.16 https://firefox-ci-tc.services.mozilla.com:443 "GET /api/index/v1/task/gecko.v2.autoland.pushlog-id.152839.decision HTTP/1.1" 200 171
 0:17.23 https://sentry.prod.mozaws.net:443 "POST /api/525/store/ HTTP/1.1" 200 41
 0:17.35 https://firefox-ci-tc.services.mozilla.com:443 "GET /api/queue/v1/task/f_KNg621R16xlFCjgZ6B8Q/artifacts/public/actions.json HTTP/1.1" 303 126
 0:17.39 https://firefoxci.taskcluster-artifacts.net:443 "GET /f_KNg621R16xlFCjgZ6B8Q/0/public/actions.json HTTP/1.1" 200 39939
 0:17.40 credentials key scrubbed from logging output
 0:17.40 {'rootUrl': 'https://firefox-ci-tc.services.mozilla.com', 'maxRetries': 5, 'signedUrlExpiration': 900}
 0:17.40 Using method(v1, v2, payload) calling convention
 0:17.40 Found a positional argument: project-gecko
 0:17.40 Found a positional argument: in-tree-action-3-backfill/73b81e92a6
 0:17.40 After processing positional arguments, we have: {'hookGroupId': 'project-gecko', 'hookId': 'in-tree-action-3-backfill/73b81e92a6'}
 0:17.40 After keyword arguments, we have: {'hookGroupId': 'project-gecko', 'hookId': 'in-tree-action-3-backfill/73b81e92a6'}
 0:17.40 Full URL used is: https://firefox-ci-tc.services.mozilla.com/api/hooks/v1/hooks/project-gecko/in-tree-action-3-backfill%2F73b81e92a6/trigger
 0:17.40 Not using hawk!
 0:17.40 Making attempt 0
 0:17.40 Making a POST request to https://firefox-ci-tc.services.mozilla.com/api/hooks/v1/hooks/project-gecko/in-tree-action-3-backfill%2F73b81e92a6/trigger
 0:17.40 HTTP Headers: {'Content-Type': 'application/json'}
 0:17.40 HTTP Payload: {"decision":{"action":{"cb_name":"backfill-task","description":"This action is normally scheduled by (limit 100 char)
 0:17.40 Starting new HTTPS connection (1): firefox-ci-tc.services.mozilla.com:443
 0:17.67 https://firefox-ci-tc.services.mozilla.com:443 "POST /api/hooks/v1/hooks/project-gecko/in-tree-action-3-backfill%2F73b81e92a6/trigger HTTP/1.1" 403 None
 0:17.67 Received HTTP Status:    403
 0:17.67 Received HTTP Headers: {'Server': 'openresty', 'Date': 'Tue, 09 Nov 2021 19:45:48 GMT', 'Content-Type': 'application/json; charset=utf-8', 'Vary': 'Accept-Encoding', 'Content-Security-Policy': "report-uri /__cspreport__;default-src 'none';frame-ancestors 'none';", 'x-content-type-options': 'nosniff', 'x-for-trace-id': '1e68f06fbb5a88b615bc16a357b6acf0', 'x-for-request-id': '6e62efef-f00d-415e-b657-1ffa2b87d050', 'Access-Control-Allow-Origin': '*', 'Access-Control-Max-Age': '900', 'Access-Control-Allow-Methods': 'OPTIONS,GET,HEAD,POST,PUT,DELETE,TRACE,CONNECT', 'Access-Control-Request-Method': '*', 'Access-Control-Allow-Headers': 'X-Requested-With,Content-Type,Authorization,Accept,Origin,Cache-Control', 'Cache-Control': 'no-store no-cache must-revalidate', 'ETag': 'W/"f01-dxdhj1lClegHa/JcAByItmMxZNU"', 'Strict-Transport-Security': 'max-age=31536000', 'Content-Encoding': 'gzip', 'Via': '1.1 google', 'Alt-Svc': 'clear', 'Transfer-Encoding': 'chunked'}
 0:17.67 Failed to trigger action for 152839

Unnecessary steps:

export TASKCLUSTER_PROXY_URL=https://firefox-ci-tc.services.mozilla.com
eval $(./taskcluster signin)

Compare with ./mach taskgraph test-action-callback --task-id KHJ6VY3rQS65WkqrfF2mtA --task-group-id RyBEkPCeRbe4cqNT12O4fA --parameters ../parameters.yml --input ../actions.json retrigger.

The backfill code doesn't pass through a check for a testing environment like the retrigger one does.

Comment 1

[Aki Sasaki (not active)]

I fixed test-action-callback for relpro here; we may need something similar for the backfill task.

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The backfill workflow has two stages (add backfill-task to the pushes which actually add the requested tasks) and uses a hook for this, mocking the second step requires more work.

Comment 3

[Heitor Neiva [:hneiva]]

Attachment: Bug 1740339 - Skip triggering actions when testing action-callbacks r?aki

Added test mode check before actually triggering actions

Comment 4

[Heitor Neiva [:hneiva]]

@aryx Please check the linked PR above and let us know if it helps.

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Thanks. This works as expected. Do you have a recommendation how to test changes which use a hook, e.g. the code for backfills?

Comment 6

[Aki Sasaki (not active)]

Hooks essentially bake in the .taskcluster.yml from a revision, e.g. https://firefox-ci-tc.services.mozilla.com/hooks/project-gecko/in-tree-action-1-backfill%2Fd48859b560, which then calls ./mach taskgraph action-callback, so ./mach taskgraph test-action-callback should also test actions which use a hook.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/715d84bd5f87
Skip triggering actions when testing action-callbacks r=aki

Comment 8

[Cristina Cozmuta (:CrissCozmuta)]

https://hg.mozilla.org/mozilla-central/rev/715d84bd5f87