548109 - Reimage 50 old talos-rev2 minis as 10.6 build slaves

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

According to inventory, these should be called: 

moz2-darwinX-slave01.build.mozilla.org
...
moz2-darwinX-slave40.build.mozilla.org


please add following to dns, inventory, nagios, etc.

For now I'm leaving this bug in RelEng while we monitor the transition, and also finish up the new 10.6 moz2-darwin10-ref, hence blocked on bug#537065.  If all goes well, I'll push this bug to IT in a few days.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

rev2 machines now being recycled. Only remaining blocker here is waiting for a working 10.6 moz2-darwin10-ref.

Comment 2

[Armen [:armenzg]]

I believe this should be blocked on bug 545539 (64bit ref image bug) instead of bug 537065 (rev3 machines)

Comment 3

[Armen [:armenzg]]

hrmm it sounds like these are rev3 machines instead of rev2.
Are both rev3 and rev2 going to use the same ref-image?

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

pushing to bear to track. Once bug#545539 is done, we can push this to IT.

Comment 6

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Tweaking summary as part of WONTFIXing bug#548231.

All these minis will come up in staging initially. We will probably push 40 to production, and leave 10 in staging, but RelEng can decide that at the time.

Comment 7

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

jabba: 

Here's the bug you were looking for. Just two imaged slaves please. If they run smoothly in staging, we'll give the go to start imaging the rest of them.

Comment 8

[Justin Dow [:jabba]]

I've got slave01 and slave02 online, in DHCP, in DNS and in Nagios. Let me know when you want me to proceed on the rest of them.

Comment 9

[Justin Dow [:jabba]]

I just wanted to clarify what the names of these should be. Is it darwinX or darwin10?

Comment 10

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #9)
> I just wanted to clarify what the names of these should be. Is it darwinX or
> darwin10?

I believe the machine names proposed in comment#0 are correct. alice: can you clarify? iirc, the suggestion was darwinX because it sorted correctly along with the darwin8, darwin9 machines. Out of curiosity, I wonder what to do with next OS - maybe call those slaves darwinXI ?

Comment 11

[[e-mail] (:bhearsum)]

(In reply to comment #10)
> (In reply to comment #9)
> > I just wanted to clarify what the names of these should be. Is it darwinX or
> > darwin10?
> 
> I believe the machine names proposed in comment#0 are correct. alice: can you
> clarify? iirc, the suggestion was darwinX because it sorted correctly along
> with the darwin8, darwin9 machines.

If that's the case, we need to fix the ref platform to be darwinX, too. Why is sorting in order important? darwin10 is more consistent with existing hostnames.

Comment 12

[Justin Dow [:jabba]]

I think darwin10 is slightly better, since it ends the confusion of whether X is a roman numeral or a place holder. If sorting is an issue, we could also switch to hex and call them darwinA and future versions could be darwinB, darwinC, etc. I think darwin10 is probably the correct choice. Just my two cents.

Comment 13

[Chris AtLee [:catlee]]

/me agrees that darwin10 makes more sense.

Comment 14

[[e-mail] (:bhearsum)]

Go ahead and shut down the ref machine, pull the image, and image all of the minis!

Comment 15

[[e-mail] (:bhearsum)]

Attachment: [checked in] add 10.6 slaves to puppet manifests

I randomly picked 01 - 04 to be staging slaves.

Comment 16

[Justin Dow [:jabba]]

The following are done and online:

moz2-darwin10-slave01
moz2-darwin10-slave02

Comment 17

[[e-mail] (:bhearsum)]

Comment on attachment 435005 [details] [diff] [review]
[checked in] add 10.6 slaves to puppet manifests

changeset:   118:b56cc318fd8f

Comment 18

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

from irc, jabba notes that we have 7 dead rev2 minis, leaving 43 available for imaging. The 4 available minis from bug#549570 helps backfill a little, bringing us back up to 47 available.

Comment 19

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #18)
> from irc, jabba notes that we have 7 dead rev2 minis, leaving 43 available for
> imaging. The 4 available minis from bug#549570 helps backfill a little,
> bringing us back up to 47 available.

...and adding the 3 available minis from bug#549551 brings us back up to 50.

Comment 20

[Justin Dow [:jabba]]

moz2-darwin10-slave01
...
...
moz2-darwin10-slave50

are all online, in DNS, DHCP and Nagios.
slaves 01, 02, and 40 through 50 are physically located at 650 Castro, the rest are at MPT.