Closed Bug 1491307 Opened 2 years ago Closed 2 years ago

hook-based retriggers failing for mochitests

Categories

(Firefox Build System :: Task Configuration, task, major)

task
Not set
major

Tracking

(firefox-esr60 fixed, firefox62 fixed, firefox63 fixed, firefox64 fixed)

RESOLVED FIXED
mozilla64
Tracking Status
firefox-esr60 --- fixed
firefox62 --- fixed
firefox63 --- fixed
firefox64 --- fixed

People

(Reporter: aryx, Assigned: dustin)

References

Details

Attachments

(1 file)

Bug 1470622 switched to hook-based actions for Treeherder (e.g. for retriggers).

Sometimes these retriggers fail, even after logging out and in and reloading treeherder:

See e.g. https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=success,pending,running,testfailed,busted,exception&revision=28baa3df5f0f530faee1900f43c463f6b77b985c&searchStr=retrigger

If treeherder had to roll back, there would be errors from buildbot shutdown.
https://tools.taskcluster.net/groups/DFm69ENRTIGCxEJwYa_GeQ/tasks/By_RQ7A-Q6yWmo7xktPUzg/runs/0/logs/public%2Flogs%2Flive.log
[task 2018-09-14T11:08:16.263Z] Traceback (most recent call last):
[task 2018-09-14T11:08:16.263Z]   File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 252, in action_callback
[task 2018-09-14T11:08:16.263Z]     test=False)
[task 2018-09-14T11:08:16.263Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/actions/registry.py", line 325, in trigger_action_callback
[task 2018-09-14T11:08:16.263Z]     cb(Parameters(**parameters), graph_config, input, task_group_id, task_id, task)
[task 2018-09-14T11:08:16.263Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/actions/retrigger.py", line 131, in mochitest_retrigger_action
[task 2018-09-14T11:08:16.263Z]     custom_mach_command += [input['path']]
[task 2018-09-14T11:08:16.263Z] KeyError: u'path'

It looks like the issue here is that these are mochitest retriggers, which expect input as to what mochitest to run, etc.  Treeherder isn't giving them that input.

Probably:
 * We should define a minimal input for all of the pre-defined action names (including retrigger). Probably {} for most.
 * We should adjust all of the retrigger options to accept that input and do a basic retrigger

Hassan, does that seem like a good fix?

So to be clear, this affects only mochitests.
Component: Hooks → Task Configuration
Flags: needinfo?(helfi92)
Product: Taskcluster → Firefox Build System
Assignee: nobody → dustin
Seems like a good fix.
Flags: needinfo?(helfi92)
Duplicate of this bug: 1491287
Summary: new treeherder hook-based retriggers often failing → hook-based retriggers failing for mochitests
Even with no options, this is not a simple retrigger -- it adds voluminous
debug output.  So let's keep it as a custom action.
Comment on attachment 9009152 [details]
Bug 1491307: make retrigger-mochitest its own action

Tom Prince [:tomprince] has approved the revision.
Attachment #9009152 - Flags: review+
Pushed by dmitchell@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/841dccc87719
make retrigger-mochitest its own action r=tomprince
Pushed by rgurzau@mozilla.com:
https://hg.mozilla.org/mozilla-central/rev/28de549aaae2
make retrigger-mochitest its own action r=tomprince a=mochitest-retrigger-fix
https://hg.mozilla.org/mozilla-central/rev/28de549aaae2
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla64
Duplicate of this bug: 1491534
IIUC, this bug is the reason that we can't currently cancel jobs on mozilla-release or mozilla-esr60 from Treeherder. I took a look at backporting, but there's a number of dependencies and conflicts. Dustin, can you please sort out what needs uplifting to get this working on the remaining supported branches?
Flags: needinfo?(dustin)
FTR, here's the error I'm getting (on Beta too, actually):
Taskcluster: No such hook --- * method: triggerHook * errorCode: ResourceNotFound * statusCode: 404 * time: 2018-09-20T20:24:32.941Z
Fixed that with a ci-admin run.
Flags: needinfo?(dustin)
You need to log in before you can comment on or make changes to this bug.