507545 - Reduce frequency of "idle" builds

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

We currently trigger new build every 2 hours, regardless of whether there are any changes or not. This triggers new opt/debug/leak builds, with cascading runs of unittests and talos, all across all o.s. This is across most(all?) active code lines.

This was originally enabled as a workaround to stop quieter branches having columns go empty and fall off the tinderbox waterfall. However, as people move to tinderbox pushlog, there is less need for this. For those still using tinderbox, we can reduce the frequence of pushed jobs like this to 12 hours, which significantly reduces load, yet still keeps the columns from going hidden on the waterfall.

(In group meeting discussion, we decided to not turn these "idle" builds off completely, just yet. We need to wait until after tinderboxpushlog changes from getting data from the tinderbox waterfall using JSON, over to getting data from our still-under-development db behind buildbot.)

Comment 1

[Chris AtLee [:catlee]]

(In reply to comment #0)
> We currently trigger new build every 2 hours, regardless of whether there are
> any changes or not. This triggers new opt/debug/leak builds, with cascading
> runs of unittests and talos, all across all o.s. This is across most(all?)
> active code lines.

This isn't quite accurate.  On mozilla-central and mozilla-1.9.1 we trigger a new build if none has been triggered in 2 hours.  On the project branches this is set to 10 hours.

But yeah, backing everything off to 12 hours sounds great.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

From prior experience you have to set the period to less than 12 hours to avoid dropping. We get nagios reporting that TraceMonkey machines have dropped and when you go look the periodic builds have started but not completed.

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #2)
> From prior experience you have to set the period to less than 12 hours to avoid
> dropping. We get nagios reporting that TraceMonkey machines have dropped and
> when you go look the periodic builds have started but not completed.

Am I trying to make sure something starts on waterfall within 12 hours? (set idle=11) or does it need to complete on waterfall within 12 hours? (set idle=9)

Anything larger then "2" will be an improvement. :-)

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

The latter I think. We have 10 hours on TraceMonkey now and get drops.

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #4)
> The latter I think. We have 10 hours on TraceMonkey now and get drops.

ok lets do that, idle=9 hours is fine with me also.

Comment 6

[Chris Cooper [:coop] (he/him)]

Attachment: Bump idle timeout to 9 hours for all branches

I've also changed the idle timeout for all the project branches since Nick indicated that TM was seeing occasional drops.

Comment 7

[Chris Cooper [:coop] (he/him)]

Comment on attachment 392813 [details] [diff] [review]
Bump idle timeout to 9 hours for all branches

http://hg.mozilla.org/build/buildbot-configs/rev/18bf2f7e98de

Comment 8

[Chris Cooper [:coop] (he/him)]

production-master reconfig-ed.