516127 - Find some way to get builds going quicker after a push

Status: RESOLVED FIXED

(Mozilla Messaging Graveyard :: Release Engineering, defect)

Description

[Phil Ringnalda (:philor)]

We build wonderfully fast, which is great for getting off the hook after you push, but... I pushed something at 19:43, and got 3.0 builds (what I'm interested in, tonight and mostly for now) starting at 20:36 for Windows, 20:50 for Linux, and 21:50 for Mac. I know there's been talk about the Mac problems that I haven't been paying enough attention to, but even the 45 minutes before anything at all starts building is pretty painful, much less the 2+ hours before even starting to get ready to be able to go do something else. If just throwing money at the problem will help to make it go away, davida said he's happy to do that. Right now, I pushed something else at 22:05, and at 22:31 nothing's building it, so I strongly suspect I'll be giving up and going to bed before I'm actually off the hook, which isn't the sort of behavior we really want to push ourselves into.

Comment 1

[David Ascher (:davida)]

Asuth points out that maybe the 3.1 builders are stealing builders from 3.0, which might be a quick tweak.  But I'm also happy to just buy more if that's the way to faster responses.

Comment 2

[Phil Ringnalda (:philor)]

Maybe the tinderboxes just hate me: after my 19:43 push, bienvenu pushed at 20:08, and 3.1 started Windows and Linux builds at 20:09.

By way of unfair comparison, since they have a trillion builders and don't have our multi-repo-per-build issue, for a mozilla-central push rather than 25 or 45 or 67 minutes before something starts, it's a flat 3 minutes. Takes forever once it starts, but it *will* start 3 minutes later.

Comment 3

[Philippe M. Chiasson (:gozer)]

(In reply to comment #2)
> Maybe the tinderboxes just hate me: after my 19:43 push, bienvenu pushed at
> 20:08, and 3.1 started Windows and Linux builds at 20:09.

No, they don't hate you. The issue is a different one for each platforms, and has to do with our multi-branch setup. Also, all this has been made worse by the l10n-repack jobs that now run across both branches, and generate 10s of extra builds that add latency to the process. This is what is making this issue much more visible than it used to be.

First, on Linux, that's because we only have 3 builders, fast ones, but only 3, so they are almost never idle when there is tree activity, so when you push, you end up waiting for a build in progress to complete before it can pick yours up.
I can help that significantly by throwing a few more Linux builders in the mix (we have resources for that).

Second, on Mac, that's a bigger issue. We have a single Xserve that running pretty much at full capacity, and as a result, the builders there are much slower. That, and the fact that there are only 4, plus the l10n-repack jobs, and you end up again with builders that are almost always busy and not idling, waiting to pick up new changes right away.

I am awaiting news from MoCo about taking over the old Thunderbird Xserve, and there is also the Sunbird Xserve, to try and increase capacity there. It's currently the biggest pain point, really. I am also thinking of the possibility of adding a few Minis to our build pool to help there as well. Once Beta4 is out, I'll be picking that up.

Third, on Windows, we are in much better shape, with the most builders (8 total) for 6 regular jobs. Lately, that has worsened a little because of the l10n jobs, but by less.

> By way of unfair comparison, since they have a trillion builders and don't have
> our multi-repo-per-build issue, for a mozilla-central push rather than 25 or 45
> or 67 minutes before something starts, it's a flat 3 minutes.

Got to be watching hg with a 3 minute tree stable timer.

> Takes forever
> once it starts, but it *will* start 3 minutes later.

Yeah, that's because they have more buildslaves than build jobs most of the time, so there are always (or very often) idle builders just waiting for something to do.

I was hoping to improve things by using buildbot's prioritization features, but it's not quite smart enough to allow for prioritization without starving other builders, so that only works for release/nightly builds.

Another possible solution would be to limit l10n repacks jobs to, say, a single builder per platform, to reduce the impact it has on latency, at the expense of producing the l10n results later.

Of course, the simple, ideal solution is to throw more resources at the problem, and that's what I am already planning to do. (I added 3 new win32 builders last week, for instance)

Comment 4

[Philippe M. Chiasson (:gozer)]

We have more resources now, and are planning to throw even more builders in, so closing this. If there is a more specific problem in the future, can always file new bugs.