Closed Bug 1491307 Opened 2 years ago Closed 2 years ago

hook-based retriggers failing for mochitests


(Firefox Build System :: Task Configuration, task, major)

Not set


(firefox-esr60 fixed, firefox62 fixed, firefox63 fixed, firefox64 fixed)

Tracking Status
firefox-esr60 --- fixed
firefox62 --- fixed
firefox63 --- fixed
firefox64 --- fixed


(Reporter: aryx, Assigned: dustin)




(1 file)

Bug 1470622 switched to hook-based actions for Treeherder (e.g. for retriggers).

Sometimes these retriggers fail, even after logging out and in and reloading treeherder:

See e.g.,pending,running,testfailed,busted,exception&revision=28baa3df5f0f530faee1900f43c463f6b77b985c&searchStr=retrigger

If treeherder had to roll back, there would be errors from buildbot shutdown.
[task 2018-09-14T11:08:16.263Z] Traceback (most recent call last):
[task 2018-09-14T11:08:16.263Z]   File "/builds/worker/checkouts/gecko/taskcluster/", line 252, in action_callback
[task 2018-09-14T11:08:16.263Z]     test=False)
[task 2018-09-14T11:08:16.263Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/actions/", line 325, in trigger_action_callback
[task 2018-09-14T11:08:16.263Z]     cb(Parameters(**parameters), graph_config, input, task_group_id, task_id, task)
[task 2018-09-14T11:08:16.263Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/actions/", line 131, in mochitest_retrigger_action
[task 2018-09-14T11:08:16.263Z]     custom_mach_command += [input['path']]
[task 2018-09-14T11:08:16.263Z] KeyError: u'path'

It looks like the issue here is that these are mochitest retriggers, which expect input as to what mochitest to run, etc.  Treeherder isn't giving them that input.

 * We should define a minimal input for all of the pre-defined action names (including retrigger). Probably {} for most.
 * We should adjust all of the retrigger options to accept that input and do a basic retrigger

Hassan, does that seem like a good fix?

So to be clear, this affects only mochitests.
Component: Hooks → Task Configuration
Flags: needinfo?(helfi92)
Product: Taskcluster → Firefox Build System
Assignee: nobody → dustin
Seems like a good fix.
Flags: needinfo?(helfi92)
Duplicate of this bug: 1491287
Summary: new treeherder hook-based retriggers often failing → hook-based retriggers failing for mochitests
Even with no options, this is not a simple retrigger -- it adds voluminous
debug output.  So let's keep it as a custom action.
Comment on attachment 9009152 [details]
Bug 1491307: make retrigger-mochitest its own action

Tom Prince [:tomprince] has approved the revision.
Attachment #9009152 - Flags: review+
Pushed by
make retrigger-mochitest its own action r=tomprince
Pushed by
make retrigger-mochitest its own action r=tomprince a=mochitest-retrigger-fix
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla64
Duplicate of this bug: 1491534
IIUC, this bug is the reason that we can't currently cancel jobs on mozilla-release or mozilla-esr60 from Treeherder. I took a look at backporting, but there's a number of dependencies and conflicts. Dustin, can you please sort out what needs uplifting to get this working on the remaining supported branches?
Flags: needinfo?(dustin)
FTR, here's the error I'm getting (on Beta too, actually):
Taskcluster: No such hook --- * method: triggerHook * errorCode: ResourceNotFound * statusCode: 404 * time: 2018-09-20T20:24:32.941Z
Fixed that with a ci-admin run.
Flags: needinfo?(dustin)
You need to log in before you can comment on or make changes to this bug.