1288404 - Taskcluster re-triggers are not working

Status: VERIFIED FIXED

(Taskcluster :: General, defect)

Description

[Andrew Halberstadt [:ahal]]

STR:
1) Attempt to re-trigger any taskcluster job
2) Notice "Re-trigger request sent" popup
3) No new jobs show up on treeherder

I chatted with Dustin about this yesterday, and he says he saw the re-trigger request go through in the taskcluster logs and didn't notice any errors. So this is possibly a treeherder bug, though I won't rule out taskcluster yet either.

Comment 1

[Ed Morley [:emorley]]

Could you link to the task inspector page for a job that was retriggered and successfully completed in TC, but didn't show up in Treeherder? With that, we can confirm that the job was scheduled correctly and try to work out why the data either wasn't submitted by Taskcluster to Treeherder, or why the Treeherder pulse ingestion process isn't working in this case.

Thanks :-)

Comment 2

[Andrew Halberstadt [:ahal]]

Sure, https://tools.taskcluster.net/task-inspector/#fhDGqBd7TpWlWblmqhORow/0

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

From bug 1288215: "like https://treeherder.mozilla.org/#/jobs?repo=try&revision=3b80f7f156a93700348f4a023756457475b4b7a9&selectedJob=24294352 retriggered the arm job a couple of times but i don't retrigger anymore :("

Looking in the mozilla-taskcluster logs, I don't see any retrigger requests for taskId I3PEAfYVTy-cHgfeohfpiQ

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Sorry, I meant bug 1288332.

Watching https://tools.taskcluster.net/pulse-inspector/#!((exchange:exchange/treeherder/v1/job-actions,routingKeyPattern:taskcluster.try.retrigger)) and retriggering that arm job, I do see a message:

build_system_type:taskcluster
job_id:24288960
project:try
version:1
requester:[e-mail]
action:retrigger
job_guid:2373c401-f615-4f2f-9c1e-07dea217e989/0

That job_guid is a cleverly re-encoded version of the taskId (?!!@?#$%!)

> slugid.encode('2373c401-f615-4f2f-9c1e-07dea217e989')
'I3PEAfYVTy-cHgfeohfpiQ'

Mozilla-taskcluster fetches this task -- I see the superagent logging from that request.  It then checks that this is a "scheduled" task:

    // Must be a scheduled task and have a task group ID
    if (task.schedulerId !== SCHEDULER_TYPE || !task.taskGroupId) {
      return
    }  

SCHEDULER_TYPE is 'task-graph-scheduler', and the schedulerId for this task is '-'.

I think that check is outdated now.  I don't know why it's even there.
and modern tasks aren't such.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

https://github.com/taskcluster/mozilla-taskcluster/pull/88

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

OK, deployed the change, and I successfully retriggered
  https://treeherder.mozilla.org/#/jobs?repo=try&fromchange=7ff12b3f03184cdfac8c2764aa2b608e691330a5&selectedJob=24311850

Andrew, Tomcat, please confirm?

Comment 8

[Andrew Halberstadt [:ahal]]

Yep, looks like it's working again! Thanks.