715337 - [tracker] retire rev2 minis in sjc1

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task, P1)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

All of the minis in sjc1 (which are all rev2, to my knowledge, and certainly not rev3 or higher) cannot move to scl3, and need to be retired before we leave sjc1 on April 30.

Yes, Virginia, there is a contingency plan if that's not possible, but it's a contingency plan, not the current plan of record.  It's expensive, and a waste of a lot of people's time.

This includes talos-rev2-tiger* (c.f. bug 628405) and {moz2-darwin*,try-mac64*}.

Replacing these systems requires some work from developers and potentially some product support changes.  More details to follow.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

So we have a few things to worry about here.

== for mozilla-central ==

The product targets 10.5-10.7, with a fat binary containing 32-bit (runs on 10.5) and 64-bit (runs on 10.6, 10.7).

There's work underway (bug 674655) to support building for 10.5 using the 10.6 SDK, which would eliminate the dependency on the 10.5 SDK, which is only available in XCode 3.x, which only runs on 10.6.  Once this is done, with a bit of extra work, we could be building for 10.5, using the 10.6 SDK, on 10.7 rev5 minis in scl3.

This also depends on releng to spec rev5 minis (bug 688632), get them delivered and installed, and get a builder image running on them in the short time remaining (although it's worth noting we've been aware of this problem for several months now, so I don't feel too bad about the short timeframe and the cut corners it will likely require).

So I think that we can get mozilla-central off of the these rev2 minis and xserves in a timely fashion.  I would want confident "yes"s on the build-system issues before January 31 to avoid the expensive contingency plan.

== for Firefox 3.6.x ==

3.6.x still supports 10.4 and powerpc.  There's just no way to build these on new hardware, and anyway adjusting 3.6.x build processes is not a good use of Mozilla developer resources.

So the first option (hope?) is that 3.6.x gets EOL'd before we have to had over the keys to sjc1 (the old colo).  https://wiki.mozilla.org/Enterprise/Firefox/ExtendedSupport:Proposal is a DRAFT, but says that the first ESR will be Firefox 11 (I assume 10 is a typo) on April 24, 2012.  Which gives us an easy 6 days to unplug and destroy the minis and xserves -- NO PROBLEM!  OK, maybe that's not so realistic.

The second option is to select a subset of the hardware currently in sjc1 -- representative enough to do 3.6.x builds, tests, and releases, and including the talos-rev2-tiger machines -- and move those to mtv1 for a while.  The idea makes me wretch a little, but this hardware should have a *very* limited lifetime anyway.  This is a much cheaper and easier contingency plan.

Comment 2

[Armen [:armenzg]]

bug 717102 is to decide which rev5 to buy (added as dep).

bug 698827 will soon switch the 10.5 leak builds to 10.6

We also have mobile nightly & repacks running on darwin9 slaves. We will want to switch this to Lion (hopefully they will soon de-supported).. aki probably would know if this is to be disabled soon.

Comment 3

[Armen [:armenzg]]

If I have my numbers right:
* We have 165 darwin build slaves on SJC
* 147 of them are rev2 minis
* 71 of them are darwin9 minis

We also have the tiger slaves for 3.6.

Comment 4

[John Hopkins (:jhopkins)]

Thunderbird builds can use its existing small number of minis for 3.1.x builds as a contingency plan.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

John, those minis are currently in sjc1, so we'll need more detail on that contingency plan.  If you feel such a plan is necessary, please open a new bug blocking this one to sort it out?

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

bm-parallels is now in scl3 and with it bm-deploystudio, so we can no longer re-image rev2 minis.  Their days are numbered (at 48 as of today) anyway.

Comment 7

[Amy Rich [:arr] [:arich]]

We are keeping 20 minis in scl3 to replace ones that die in mtv1.  All other minis have been donated to ctalbert.  All minis in sjc1 have been deactivated and are scheduled for the 5/8 train.

Comment 8

[Mike Taylor [:bear]]

removing 674655 from the dependency list - - it's a dev only bug

Comment 9

[Amy Rich [:arr] [:arich]]

The spares are in scl3 if we need them.