539105 - Reduce talos load from 3.0 builds

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Lukas Blakk [:lsblakk] use ?needinfo]

Currently the en-US builds are running continuously under the tinderbox client, this is creating a large, unnecessary workload for Talos.  Windows isn't a concern here, so just Linux and Mac building less frequently would be a big win.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

How long do we want to wait after each build ? 30 mins ? 60 mins ?

The linux builder is just doing opt, no-op build is about 6 minutes.

Mac is doing opt then debug, then it would wait, then repeat from the top. No-op build times are ~ 13 and 8 minutes respectively.

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(cc-ing beltzner, dveditz for comment)

Win32 opt builds take approx 3hrs; its so slow that, aiui, we have 2 machines for win32... one doing debug, one doing opt.

How about delaying linux, mac so they run as (in)frequently as win32, meaning run every 3 hours? That brings the number of builds for each OS closer to parity, and at the same time significantly reduces the load on Talos.

Comment 3

[Daniel Veditz [:dveditz]]

three hour turn-around is terrible and I'm never going to agree to that. Where do you see 3 hours anyway? The slow build (fx-win32-tbox Depend Nightly) takes at most 1:20, and the faster "fxdbug-win32-tb Depend Debug + Leak Test" takes about 20 minutes.

The real problem is that Talos is getting too many new builds, not that tinderbox clients are making them. Instead of slowing down the builds you could alternatively throw away the bits most of the time rather than feed Talos after every one.

If a build an hour is still too much for Talos then I'd really rather you take the second approach than delay the builds even more.

Comment 4

[Chris AtLee [:catlee]]

Attachment: add delay to ftp poller

Not sure if this will work or not, haven't had time to test it.  Basically, we add a minimum time between builds in the ftp poller.  So any new builds that are detected within an hour of the last one submitted are ignored.

I need access to talos staging to test this.

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #4)
> Created an attachment (id=421453) [details]
> I need access to talos staging to test this.
ping?

Comment 6

[Chris AtLee [:catlee]]

Comment on attachment 421453 [details] [diff] [review]
add delay to ftp poller

Seems to be working in staging.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 421453 [details] [diff] [review]
add delay to ftp poller

Looks fine to me. When I looked at moving the builds to buildbot-driven it would have been quite a bit more effort than this.

Comment 8

[Chris AtLee [:catlee]]

Comment on attachment 421453 [details] [diff] [review]
add delay to ftp poller

changeset:   2033:8219c9dd5d55

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Also for talos-r3

This is the same patch as attachment 421453 [details] [diff] [review] but applied to the talos-r3 directory, no fuzz was required. Diffing talos-pool and talos-r3 gives only sensible-looking changes in config.py from the different slave pools, and the ports in master.cfg. We'll need to add Lorentz to talos-r3 at some point though.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 428081 [details] [diff] [review]
Also for talos-r3

http://hg.mozilla.org/build/buildbot-configs/rev/7c5d4d938506

reconfig'd talos-r3 master.