493942 - are there mac build/leak/unittest slaves missing?

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Right now there seem to be an insufficient number of Mac slaves in the build/leak/unittest pool, and this is causing most pushes to mozilla-central to be untested on Mac.  For example, for the 5 pushes to mozilla-central from 4:08 to 5:10 this morning, there is so far only one build run (4:32), one leak run (4:32), and one unit test run (5:10), whereas for Windows and Linux each of the commits had all three of build/leak/unittest going almost immediately.

A little while ago I counted only 7 active mac slaves (3 on Firefox3.5, 2 on TraceMonkey, and 2 on MozillaTry).  I thought we had a lot more than this.  Are some of them going unused for some reason?

Comment 1

[Axel Hecht [:Pike]]

I glanced over the slaves for the l10n pool, and I've seen a few "shark" builds on various branches. Not sure if those are new, or lengthier or anything, just mentioning because I wasn't aware we had those.

Comment 2

[Chris AtLee [:catlee]]

We have 8 mac slaves total in the build/leak/unittest pool, 4 minis and 4 xerves.

Around this time of day all the l10n repacks happen, so that ties up some of the slaves for a while.

Comment 3

[Chris AtLee [:catlee]]

(In reply to comment #2)
> We have 8 mac slaves total in the build/leak/unittest pool, 4 minis and 4
> xerves.

Sorry, I mis-counted.  We have 10 slaves: 4 minis and 6 xserves.

Comment 4

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #1)
> I glanced over the slaves for the l10n pool, and I've seen a few "shark" builds
> on various branches. Not sure if those are new, or lengthier or anything, just
> mentioning because I wasn't aware we had those.

Yep, shark builds were enabled in March, see details in bug#448727. Nothing new there.

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Not "missing", just overrun and now jobs are having long wait times in this recent surge of activity. If you look in dev-tree-management, you can see wait times for production build/unittest machines were great until the last few days. 

More mac slaves already arrived, and are being brought online, which should help.

Comment 6

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Comment#5 here explains what happened about slaves being overrun when in peak-load situations. To make this better going forward, bug#500184 is tracking getting new minis powered/racked and then imaged up.

Nothing left to track in this bug. Not sure how best to close this, so going for WORKSFORME.