1631834 - mobile: Allow action tasks to be executed

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Johan Lorenzo [:jlorenzo]]

Bug 1568466 introduced action tasks for release-promotion on fenix. An unwanted side-effect was that it changed the way Taskcluster reruns task on all mobile projects.

Aki started the work in [1] (and followups) but patches were backed out. :Aryx and :sebastian both expressed their will to be able to rerun tasks.

[1] https://github.com/mozilla-mobile/reference-browser/pull/1061

Comment 1

[Johan Lorenzo [:jlorenzo]]

First failure on Fenix:

[task 2020-04-21T16:00:20.389Z] Traceback (most recent call last):
[task 2020-04-21T16:00:20.389Z]   File "/builds/worker/.local/lib/python2.7/site-packages/taskgraph/main.py", line 389, in action_callback
[task 2020-04-21T16:00:20.389Z]     test=False,
[task 2020-04-21T16:00:20.390Z]   File "/builds/worker/.local/lib/python2.7/site-packages/taskgraph/actions/registry.py", line 324, in trigger_action_callback
[task 2020-04-21T16:00:20.390Z]     cb(Parameters(**parameters), graph_config, input, task_group_id, task_id)
[task 2020-04-21T16:00:20.391Z]   File "/builds/worker/.local/lib/python2.7/site-packages/taskgraph/actions/retrigger.py", line 208, in rerun_action
[task 2020-04-21T16:00:20.391Z]     parameters, graph_config)
[task 2020-04-21T16:00:20.391Z]   File "/builds/worker/.local/lib/python2.7/site-packages/taskgraph/actions/util.py", line 42, in fetch_graph_and_labels
[task 2020-04-21T16:00:20.392Z]     decision_task_id = find_decision_task(parameters, graph_config)
[task 2020-04-21T16:00:20.392Z]   File "/builds/worker/.local/lib/python2.7/site-packages/taskgraph/util/taskgraph.py", line 31, in find_decision_task
[task 2020-04-21T16:00:20.392Z]     parameters['head_rev']))
[task 2020-04-21T16:00:20.392Z]   File "/builds/worker/.local/lib/python2.7/site-packages/taskgraph/util/taskcluster.py", line 169, in find_task_id
[task 2020-04-21T16:00:20.392Z]     raise KeyError("index path {} not found".format(index_path))
[task 2020-04-21T16:00:20.392Z] KeyError: u'index path mobile.v2.fenix.revision.2a7c73e1387e849a3606eb3c809cb5e26a452778.taskgraph.decision not found'

https://firefox-ci-tc.services.mozilla.com/tasks/SLJe45ucQPiBRmZxLo9QaQ/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FSLJe45ucQPiBRmZxLo9QaQ%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive.log#L2918

Comment 2

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1631834 - Add trust-domain scopes to reference-browser and android-components r=mtabara

Comment 3

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1631834 - part 2: Enable action tasks on android-components and firefox-ios r=mtabara

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/8291dcfc0287
part 2: Enable action tasks on android-components and firefox-ios r=tomprince

Comment 5

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1631834 - part 3: Allow action tasks on Github PRs r=tomprince

Comment 6

[Johan Lorenzo [:jlorenzo]]

Attachment: [fenix] part 1: Add revision route to decision task

Comment 7

[Johan Lorenzo [:jlorenzo]]

Attachment: [fenix] part 2: Express who triggered the action task

Comment 8

[Johan Lorenzo [:jlorenzo]]

Attachment: [reference-browser] Allow action tasks to be executed

Comment 9

[Johan Lorenzo [:jlorenzo]]

Attachment: [reference-browser] part 2: Do not provide taskId to action tasks and make .taskcluster.yml closer to fenix

Comment 10

[Johan Lorenzo [:jlorenzo]]

Attachment: [reference-browser] part 3: Install taskgraph in action task too

Comment 11

[Johan Lorenzo [:jlorenzo]]

Attachment: [reference-browser] part 4: Use taskgraph by providing full path

Comment 12

[Johan Lorenzo [:jlorenzo]]

Attachment: [reference-browser] part 5: Fix typo in path

Comment 13

[Johan Lorenzo [:jlorenzo]]

Attachment: [android-components] part 1: Allow action tasks to be executed

Comment 14

[Johan Lorenzo [:jlorenzo]]

Attachment: [android-components] part 2: Do not provide taskId on action tasks

Comment 15

[Johan Lorenzo [:jlorenzo]]

Attachment: [android-components] part 3: Install taskgraph in action task too

Comment 16

[Johan Lorenzo [:jlorenzo]]

Attachment: [android-components] part 4: Use taskgraph by providing full path

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/ci-configuration/rev/d787e8475117
part 3: Allow action tasks on Github PRs r=tomprince

Comment 18

[Johan Lorenzo [:jlorenzo]]

Attachment: Hook missing scope

Good news first: level 3 tasks can be rerun, retriggered, and cancelled. I tested them out on fenix, reference-browser, and android-components. :Aryx tested fenix and was able to rerun too.

The last piece I wanted to add was level 1 tasks and more precisely, tasks on pull requests. The last ci-config patch put me closer but we're still a long way behind it. As a matter of fact, I hit this scope error (see screenshot). Basically, we need to grant the in-tree action roles this kind of scope:

assume:repo:github.com/<fork-owner>/<repo-name>:action:generic.

Sadly, there is no way to predict each possible owner. I wanted to use the <..> substitute but since the role is hook-id:project-mobile/in-tree-action-1-generic/* (* being the first 10 digits of the sha256 sum of .taskcluster.yml), <..> will be resolved into: assume:repo:github.com/<first-10-digits>:action:generic.

I don't see a simple way to solve this problem. But that's not all!

I manually granted my fork and I stumbled upon this new error:

[task 2020-05-27T09:29:11.943Z] Traceback (most recent call last):
[task 2020-05-27T09:29:11.943Z]   File "/builds/worker/.local/lib/python2.7/site-packages/taskgraph/main.py", line 390, in action_callback
[task 2020-05-27T09:29:11.943Z]     test=False,
[task 2020-05-27T09:29:11.943Z]   File "/builds/worker/.local/lib/python2.7/site-packages/taskgraph/actions/registry.py", line 324, in trigger_action_callback
[task 2020-05-27T09:29:11.944Z]     cb(Parameters(**parameters), graph_config, input, task_group_id, task_id)
[task 2020-05-27T09:29:11.944Z]   File "/builds/worker/.local/lib/python2.7/site-packages/taskgraph/actions/retrigger.py", line 208, in rerun_action
[task 2020-05-27T09:29:11.945Z]     parameters, graph_config)
[task 2020-05-27T09:29:11.945Z]   File "/builds/worker/.local/lib/python2.7/site-packages/taskgraph/actions/util.py", line 42, in fetch_graph_and_labels
[task 2020-05-27T09:29:11.945Z]     decision_task_id = find_decision_task(parameters, graph_config)
[task 2020-05-27T09:29:11.945Z]   File "/builds/worker/.local/lib/python2.7/site-packages/taskgraph/util/taskgraph.py", line 31, in find_decision_task
[task 2020-05-27T09:29:11.945Z]     parameters['head_rev']))
[task 2020-05-27T09:29:11.945Z]   File "/builds/worker/.local/lib/python2.7/site-packages/taskgraph/util/taskcluster.py", line 171, in find_task_id
[task 2020-05-27T09:29:11.946Z]     raise KeyError("index path {} not found".format(index_path))
[task 2020-05-27T09:29:11.946Z] KeyError: u'index path mobile.v2.reference-browser.revision.d495082f3c0ae41072b78575a0f1948c4d2c6226.taskgraph.decision not found'

Taskgraph is looking for an index that doesn't exist because revision d495082f is the tip of my PR branch. So we would need to create a dedicated index namespace for untrusted PRs. While feasible, I don't think it's a priority. PRs can somehow be retriggered by closing and reopening them (bug 1631014). It's kind of waste of resource because the full graph is spinned, but at least there's a way of rerunning intermittent jobs.

I'm going to track the work in a separate bug because this current bug was mainly for sheriffs to be able to rerun level 3 tasks.

Comment 19

[Tom Prince [:tomprince]]

(In reply to Johan Lorenzo [:jlorenzo] from comment #18)

The last piece I wanted to add was level 1 tasks and more precisely, tasks on pull requests. The last ci-config patch put me closer but we're still a long way behind it. As a matter of fact, I hit this scope error (see screenshot). Basically, we need to grant the in-tree action roles this kind of scope:

assume:repo:github.com/<fork-owner>/<repo-name>:action:generic

Note that this role is under our control, so we can change it to be something that only involves the base repo. That said, we should be careful to ensure that the action on pull-requests only has the pull-request scopes, and not the scopes for pushes etc. (which the current action roles do)

Comment 20

[Johan Lorenzo [:jlorenzo]]

Attachment: [firefox-ios] Add missing route used by taskgraph

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b63d3447f180
[components] Bug 1631834 - Allow action tasks to be executed
https://hg.mozilla.org/integration/autoland/rev/cbcd9eb4548e
[components] Bug 1631834 - part 2: Do not provide taskId on action tasks
https://hg.mozilla.org/integration/autoland/rev/67c8cdb0aa64
[components] Bug 1631834 - part 3: Install taskgraph in action task too
https://hg.mozilla.org/integration/autoland/rev/56669f873e1f
[components] Bug 1631834 - part 4: Use taskgraph by providing full path
https://hg.mozilla.org/integration/autoland/rev/269cb99b895a
[fenix] Bug 1631834 - Add revision route to decision task (https://github.com/mozilla-mobile/fenix/pull/10644)
https://hg.mozilla.org/integration/autoland/rev/0e2690f3a9ba
[fenix] Bug 1631834 - part 2: Express who triggered the action task (https://github.com/mozilla-mobile/fenix/pull/10650)