1255406 - "Retrigger all" on a bunch of pinned jobs doesn't always retrigger them all

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect)

Description

[Kartikaya Gupta (email:[e-mail])]

This is happened to me a few times now - I pin a bunch of jobs, use the "retrigger all" button, but only some of the jobs get retriggered. In the latest instance, I wanted to retrigger all the talos desktop jobs on the latest m-c push, so I:
1) went to TH for that cset
2) filtered by talos [1]
3) used the "filters > pin all showing" button to add them to the pinboard
4) tried to remove the android talos tests (tpn was easy, had to hunt for the right tsvg jobs based on the runtime in the tooltip), and
5) Used the "retrigger all" item. Repeat to retrigger 5 times.

A few minutes later I see that the OSX jobs got retriggered but none of the others did. I also looked in the BuildAPI page to make sure it wasn't just the TH display, but no - only OS X jobs listed there as pending.

I'm going to manually retrigger the things I need so if you go back to that push later it will probably look different.

[1] https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=dd1abe874252&filter-searchStr=talos&group_state=expanded

Comment 1

[William Lachance (:wlach)]

Hey Armen, do you have any idea what might have happened here?

Comment 2

[Armen [:armenzg]]

I never worked in the work behind it.
Maybe Vaibhav would know.
I don't recall this being one of the actions that pulse_actions took care of.
We should really document these when I come back.

If you still want my help to investigate please do not clear the NI flag.
-------------------------
I will be away until March 23rd.
Please find another reviewer before that, otherwise, I will get to it at that point.
My apologies for the inconvenience.

Comment 3

[Armen [:armenzg]]

kats, where do you see the "retrigger all" item?
I can't see it in the UI:
http://people.mozilla.org/~armenzg/sattap/240897f5.png

I don't think pulse_actions takes care of these kind of requests.
I only see this on the 10th from you:
> pulse_actions.handlers.treeherder_runnable\t INFO:\t New jobs requested by [e-mail] for https://treeherder.mozilla.org/#/jobs?repo=try&revision=b2b111fece96 

Treeherder is calling Buildapi directly without going through pulse_actions.
Loading one of the re-trigger shows self-server request by kgupta (pulse_actions leaves a note to make it easy to spot them):
https://secure.pub.build.mozilla.org/buildapi/self-serve/mozilla-central/build/102357942

Comment 4

[Kartikaya Gupta (email:[e-mail])]

Attachment: Retrigger all screenshot

The "retrigger all" item is in the dropdown after you have some stuff in the pinboard. See attached screenshot.

Comment 5

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #4)
> Created attachment 8733860 [details]
> Retrigger all screenshot
> 
> The "retrigger all" item is in the dropdown after you have some stuff in the
> pinboard. See attached screenshot.

Which calls https://github.com/mozilla/treeherder/blob/766838a5c62444776391548d354eb7885a7c4c9c/ui/plugins/pinboard.js#L97 when clicked. Which passes the pinned jobs to https://github.com/mozilla/treeherder/blob/4a6a0768f8169ad70a9a854ae02e3ca4a0be8349/ui/plugins/controller.js#L361

Comment 6

[Armen [:armenzg]]

I think this calls BuildAPI directly (if so, nothing for me to look at in here):
> return thBuildApi.retriggerJob($scope.repoName, requestId);
https://github.com/mozilla/treeherder/commit/e3b276268ce7112ceb6b585fe45291e8f5c7afd0

Comment 7

[Cameron Dawson [:camd]]

I'll take a look and see if I can determine what TH may be doing wrong here...

Comment 8

[Cameron Dawson [:camd]]

Heh.  So I pinned probably around 30 jobs, and did a retrigger all (on that same push, so sorry for the noise).  It only retriggered the first 10 of them.  I could see in the console that it only made the call 10 times.

I can't quite tell WHY that is yet.  But I think we're on to something.  :)

Comment 9

[Cameron Dawson [:camd]]

I should say also that id DID trigger windows as well as mac pushes, so platform isn't the issue...

Comment 10

[Botond Ballo [:botond]]

(In reply to Cameron Dawson [:camd] from comment #8)
> Heh.  So I pinned probably around 30 jobs, and did a retrigger all (on that
> same push, so sorry for the noise).  It only retriggered the first 10 of
> them.  I could see in the console that it only made the call 10 times.
> 
> I can't quite tell WHY that is yet.  But I think we're on to something.  :)

Yeah, I've seen this behaviour a lot. Something is limiting the retriggers to the first 10, or something like that.

Comment 11

[Cameron Dawson [:camd]]

Yeah, so the /artifacts endpoint has a default of returning only the first 10 artifacts unless the "count" param is passed.  Rats.  But an easy fix.  :)  I'll write up a PR tomorrow.

Comment 12

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:bug1255406 > mozilla:master

Comment 13

[William Lachance (:wlach)]

Comment on attachment 8735648 [details] [review]
[treeherder] mozilla:bug1255406 > mozilla:master

Thanks!

Comment 14

[GitHub Autolander Bot]

Pull request has landed in master: https://github.com/mozilla/treeherder/commit/a86d241370f75726411b04236e34d883c97cf967

Comment 15

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/606864b79000b5380b6e9a203390407a103a3c30
Bug 1255406 - "Retrigger all" on a >10 pinned jobs only retriggers 10

The /artifacts endpoint defaults to only returning the first 10 items
unless a count is passed in.  We weren’t passing in that param, so we’d
only get back the first 10 (we need them for the buildbot request_ids).

This fixes it by passing in a “count” param equal to the number of jobs
we want to retrigger.

Comment 16

[Cameron Dawson [:camd]]

not sure why that was unresolved