1491307 - hook-based retriggers failing for mochitests

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Bug 1470622 switched to hook-based actions for Treeherder (e.g. for retriggers).

Sometimes these retriggers fail, even after logging out and in and reloading treeherder:

See e.g. https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=success,pending,running,testfailed,busted,exception&revision=28baa3df5f0f530faee1900f43c463f6b77b985c&searchStr=retrigger

If treeherder had to roll back, there would be errors from buildbot shutdown.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

https://tools.taskcluster.net/groups/DFm69ENRTIGCxEJwYa_GeQ/tasks/By_RQ7A-Q6yWmo7xktPUzg/runs/0/logs/public%2Flogs%2Flive.log
[task 2018-09-14T11:08:16.263Z] Traceback (most recent call last):
[task 2018-09-14T11:08:16.263Z]   File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 252, in action_callback
[task 2018-09-14T11:08:16.263Z]     test=False)
[task 2018-09-14T11:08:16.263Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/actions/registry.py", line 325, in trigger_action_callback
[task 2018-09-14T11:08:16.263Z]     cb(Parameters(**parameters), graph_config, input, task_group_id, task_id, task)
[task 2018-09-14T11:08:16.263Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/actions/retrigger.py", line 131, in mochitest_retrigger_action
[task 2018-09-14T11:08:16.263Z]     custom_mach_command += [input['path']]
[task 2018-09-14T11:08:16.263Z] KeyError: u'path'

It looks like the issue here is that these are mochitest retriggers, which expect input as to what mochitest to run, etc.  Treeherder isn't giving them that input.

Probably:
 * We should define a minimal input for all of the pre-defined action names (including retrigger). Probably {} for most.
 * We should adjust all of the retrigger options to accept that input and do a basic retrigger

Hassan, does that seem like a good fix?

So to be clear, this affects only mochitests.

Comment 2

[Hassan Ali (:hassan)]

Seems like a good fix.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: Bug 1491307: make retrigger-mochitest its own action

Even with no options, this is not a simple retrigger -- it adds voluminous
debug output.  So let's keep it as a custom action.

Comment 5

[Tom Prince [:tomprince]]

Comment on attachment 9009152 [details]
Bug 1491307: make retrigger-mochitest its own action

Tom Prince [:tomprince] has approved the revision.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=5571a8210e4bca287c120d61ce5de176be9175a5

Comment 7

[[github robot]]

Commits pushed to master at https://github.com/taskcluster/taskcluster-docs

https://github.com/taskcluster/taskcluster-docs/commit/b587d78210dd76f82715fa7e17c5124521c59f57
Bug 1491307 - indicate that default actions get {}

https://github.com/taskcluster/taskcluster-docs/commit/e143f15cd9afd181c7bebdd280d052a3f302011c
Merge pull request #282 from djmitche/bug1491307

Bug 1491307 - indicate that default actions get {}

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/841dccc87719
make retrigger-mochitest its own action r=tomprince

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/28de549aaae2
make retrigger-mochitest its own action r=tomprince a=mochitest-retrigger-fix

Comment 10

[Raul Gurzau (:RaulG)]

https://hg.mozilla.org/mozilla-central/rev/28de549aaae2

Comment 11

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/releases/mozilla-beta/rev/dd935a08ff4a

Comment 12

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/841dccc87719

Comment 14

[Ryan VanderMeulen [:RyanVM]]

IIUC, this bug is the reason that we can't currently cancel jobs on mozilla-release or mozilla-esr60 from Treeherder. I took a look at backporting, but there's a number of dependencies and conflicts. Dustin, can you please sort out what needs uplifting to get this working on the remaining supported branches?

Comment 15

[Ryan VanderMeulen [:RyanVM]]

FTR, here's the error I'm getting (on Beta too, actually):
Taskcluster: No such hook --- * method: triggerHook * errorCode: ResourceNotFound * statusCode: 404 * time: 2018-09-20T20:24:32.941Z

Comment 16

[Dustin J. Mitchell [:dustin] (he/him)]

Fixed that with a ci-admin run.

Comment 17

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-release/rev/23c987fff843

Comment 18

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/0b5e251fa609