690672 - self-serve should be able to disable coalescing (aka retriggering builds manually should never coalesce)

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Phil Ringnalda (:philor)]

I'm not sure whether it should always avoid coalescing, or should just have the option of avoiding it, but for my use of self-serve, which is primarily "I need to know which of these three pushes to back out because it caused this failure that happens on between one in four and two in four runs of a suite, so I need to have four runs of the suite on each of two or three pushes" coalescing makes the task absolutely miserable. Not only do I have to only trigger one run at a time, having to wait until it's picked up before triggering a second one (third, fourth, twelfth), but every time between the trigger and the pickup I have to worry about the way that any more recent push might finish a build and schedule the same suite and coalesce away my job.

Comment 1

[Phil Ringnalda (:philor)]

Probably I do want an option, since really that "somebody's getting backed out" use is when I really desperately want it and want it now, but the vast majority of my actual use by number of jobs is retriggering Android jobs from within tbpl, where I generally don't much care whether or not it gets coalesced, just that it runs again after that infra failure, and gets a chance to get as far as a code failure instead. So an option (somehow jammed) in buildapi/self-serve/, where I usually am for regression hunting since I'm having to trigger things that got coalesced the first run, too, would work fine.

Comment 2

[Chris AtLee [:catlee]]

I think this could work:

* self-serve accepts optional parameter to disable coalescing; sets this as a buildset property on the new build request (e.g. 'mergeable': False)

* modify our mergeRequests function (http://hg.mozilla.org/build/buildbotcustom/file/74df664b9053/misc.py#l348) to check for this property and return False if either request has it set

Comment 3

[Ed Morley [:emorley]]

This bites myself, philor and RyanVM quite a lot.
It would be great if manually retriggered jobs would never get coalesced :-)

Comment 4

[Ed Morley [:emorley]]

Thinking about comment 1 - I'm not that sure we really do need an option any more (given the reduced Android failure rate) - which would simplify the implementation (we'd otherwise need two sets of rebuild columns on the self-serve columns, or else have to tuck the option away somewhere, or ...?) and mean I can still use TBPL's UI.

I'm trying to pick through the buildbotcustom/buildbot repo now, but I know absolutely nothing about them. Is there an easy way to tell from http://hg.mozilla.org/build/buildbotcustom/file/74df664b9053/misc.py#l348 whether the request came from self-serve as opposed to the schedulers? If so we could just never merge if it was a self-serve request and resolve this pretty easily.

Comment 5

[Ed Morley [:emorley]]

Attachment: Don't coalesce on retriggers

I'm a complete buildbot (and python) n00b, so this is probably wrong - but hey :-)

Comment 6

[Chris AtLee [:catlee]]

Comment on attachment 653782 [details] [diff] [review]
Don't coalesce on retriggers

Review of attachment 653782 [details] [diff] [review]:
-----------------------------------------------------------------

Yeah, this is the right place to do it. There are some other strings that show up in the reason field that we may want to use as well. E.g.
"Requested via self-serve by [e-mail]" for new builds rather than rebuilds

Alternatively, we could make sure that all actions performed via self-serve include "self-serve" in the reason. e.g. http://hg.mozilla.org/build/buildapi/file/6a71c42d0895/buildapi/scripts/selfserve-agent.py#l237

Comment 7

[Ed Morley [:emorley]]

Attachment: Don't coalesce on retriggers v2

Good suggestion, I've filed bug 786290 to add a new 'Self-serve' prefix.

Patch updated to use this new string.

Comment 8

[Ed Morley [:emorley]]

Comment on attachment 656025 [details] [diff] [review]
Don't coalesce on retriggers v2

Thank you for the review :-)

http://hg.mozilla.org/build/buildbotcustom/rev/08ab818fd5a1

Comment 9

[Armen [:armenzg]]

Live in production.

Comment 10

[Ed Morley [:emorley]]

I tested this earlier but the builds still seemed to coalesce.
Philor/Ryan, could you keep an eye out and see if it is working next time we have a retrigger-party? :-)

Comment 11

[Chris AtLee [:catlee]]

If/when this happens again, if you could point to particular revisions and builder names, that would help a lot!

Comment 12

[Ed Morley [:emorley]]

(In reply to Chris AtLee [:catlee] from comment #11)
> If/when this happens again, if you could point to particular revisions and
> builder names, that would help a lot!

Will do :-)

(In this instance I was mid-eating dinner so ended up losing track of which tab/revision/..., but I'll pay more attention next time)

Comment 13

[Ed Morley [:emorley]]

Seems to be working fine now :-)