684436 - Stop merging builds for pushes within 3 minutes of each other

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Ed Morley [:emorley]]

Tonight makes two nights in a row where mozilla-inbound has had bustage, but yet excessive numbers of changesets have had to be backed out, since their builds have been combined (due to being within 3 minutes of each other).

Mak, myself and others certain others sherrifing m-i feel that the advantages of coalescing builds within 3 minutes is vastly outweighed by the confusion of blame.

Yes, it may rarely save one extra build, but more frequently it results in numerous extra builds after backing out one changeset after another, in order to find the cause of bustage.

Please can this be stopped.

Comment 1

[(no longer active)]

I've asked this a number of times in the past.  Releng has always said that the extra load might be too high for them to handle.  Is there any way for us to measure exactly how much this is saving us, and what would happen if we avoided job coalescing at least on mozilla-inbound?

Comment 2

[Chris AtLee [:catlee]]

(In reply to Ehsan Akhgari [:ehsan] from comment #1)
> I've asked this a number of times in the past.  Releng has always said that
> the extra load might be too high for them to handle.  Is there any way for
> us to measure exactly how much this is saving us, and what would happen if
> we avoided job coalescing at least on mozilla-inbound?

FTR, I think it should be fine. Also, we're likely to coalesce changes that happen within the same poll interval, which is 60s right now.

Comment 3

[(no longer active)]

(In reply to Chris AtLee [:catlee] from comment #2)
> (In reply to Ehsan Akhgari [:ehsan] from comment #1)
> > I've asked this a number of times in the past.  Releng has always said that
> > the extra load might be too high for them to handle.  Is there any way for
> > us to measure exactly how much this is saving us, and what would happen if
> > we avoided job coalescing at least on mozilla-inbound?
> 
> FTR, I think it should be fine. Also, we're likely to coalesce changes that
> happen within the same poll interval, which is 60s right now.

By "it", do you mean not coalescing jobs in the 3-minute window?

Comment 4

[Chris AtLee [:catlee]]

Attachment: turn off treeStableTimer

Comment 5

[Armen [:armenzg]]

Merged to production and reconfigured.

Comment 6

[[e-mail] (:bhearsum)]

This landed in production on the 9th.