635964 - re-purpose 16 darwin9 minis for darwin10 service

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P4)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

The following slaves were killed in bug 604497 (which is secret for unrelated reasons), and will be brought back up as 10.6 builders and try systems.

moz2-darwin9-slave29
moz2-darwin9-slave30
moz2-darwin9-slave31
moz2-darwin9-slave32
moz2-darwin9-slave33
moz2-darwin9-slave34
moz2-darwin9-slave35
moz2-darwin9-slave36
moz2-darwin9-slave37
try-mac-slave20
try-mac-slave21
try-mac-slave22
try-mac-slave23
try-mac-slave24
try-mac-slave25
try-mac-slave26
try-mac-slave27
try-mac-slave28
try-mac-slave29

Spencer is currently cracking them and adding RAM in the IT area.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

As well as the RAM upgrade, these machines need OS 10.6 reimaging, and being renamed to:

moz2-darwin10-slave{53...61}
try-mac64-slave(27...36}

Comment 2

[Spencer Hui]

Following mac minis re-imaged and had their ram upgraded from 2gb to 4 gb

QM-PLEOPARD-TRY12
QM-PLEOPARD-TRY13
QM-PLEOPARD-TRY14
QM-PLEOPARD-TRY15
QM-PLEOPARD-TRY16
QM-PUBUNTU-TRY12 could not upgrade ram(screw is stripped)
QM-PUBUNTU-TRY13
QM-PUBUNTU-TRY14
QM-PUBUNTU-TRY15
QM-PUBUNTU-TRY16
MOZ2-DRAWIN9-SLAVE30
MOZ2-DRAWIN9-SLAVE29
MOZ2-DRAWIN9-SLAVE31
MOZ2-DRAWIN9-SLAVE32
MOZ2-DRAWIN9-SLAVE33
MOZ2-DRAWIN9-SLAVE34
MOZ2-DRAWIN9-SLAVE35
MOZ2-DRAWIN9-SLAVE36
MOZ2-DRAWIN9-SLAVE37
ZACK-TESTING

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

(In reply to comment #2)
> Following mac minis re-imaged and had their ram upgraded from 2gb to 4 gb

I'm assuming these are old names for the machines?

> QM-PLEOPARD-TRY12
> QM-PLEOPARD-TRY13
> QM-PLEOPARD-TRY14
> QM-PLEOPARD-TRY15
> QM-PLEOPARD-TRY16
> QM-PUBUNTU-TRY12 could not upgrade ram(screw is stripped)
> QM-PUBUNTU-TRY13
> QM-PUBUNTU-TRY14
> QM-PUBUNTU-TRY15
> QM-PUBUNTU-TRY16

So the above came from another pool of unused minis?

> MOZ2-DRAWIN9-SLAVE30
> MOZ2-DRAWIN9-SLAVE29
> MOZ2-DRAWIN9-SLAVE31
> MOZ2-DRAWIN9-SLAVE32
> MOZ2-DRAWIN9-SLAVE33
> MOZ2-DRAWIN9-SLAVE34
> MOZ2-DRAWIN9-SLAVE35
> MOZ2-DRAWIN9-SLAVE36
> MOZ2-DRAWIN9-SLAVE37

Presumably those are the same moz2-darwin9-slaveNN mentioned in comment 0.

> ZACK-TESTING

and this was in our inventory already.

These have been reimaged, but what names have they been given?  And what happened to the try-mac-slaveNN mentioned in comment 0?  Did you do any additional diagnostics on try-mac-slave28?

Comment 4

[Zandr Milewski [:zandr]]

The QM- hostnames were on the front of the machines, those are actually the try-mac-slave machines listed in comment 0.

Spencer, you'll find them in inventory under the try-mac names that are on the backs of the machines. Please give them new names per comment 1 and update inventory accordingly.

As to try-mac-slave28, let's see what happens post-imaging.

Comment 5

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to comment #4)
> As to try-mac-slave28, let's see what happens post-imaging.

iirc, this slave was also causing grief before imaging.  Maybe it needs to go to greener pastures?

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

zandr/ssh are on top of it.

Comment 7

[Spencer Hui]

    all macs have been re-named, re-labeled  and updated in inventory 
    moz2-darwin10-slave{53...61} 
    try-mac64-slave(27...36}
    John said he had trouble trying to ssh into TRY-MAC64-SLAVE35

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Thanks, spencer!

Now we need to bring these back up and attach them to masters - a slaveduty job.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

oops, back to the releng component for bear.

Comment 10

[Zandr Milewski [:zandr]]

Power and network would be a good first step, actually. /me goes back to the server closet to make that happen.

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

Is nagios updated to monitor these new slaves?  Can we set that up once they're powered up?  New bug, or on this one?

Comment 13

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Found in triage with zandr:


(In reply to comment #10)
> Power and network would be a good first step, actually. /me goes back to the
> server closet to make that happen.

1) instead of putting in new server room, will just put back on shelf in QA lab, and connect to power/network there.

2) nagios not yet done.

Not sure why this in RelEng, moving to IT to resolve.

Comment 14

[Zandr Milewski [:zandr]]

Powered up and networked in a much neater AFK.

It's friday at 5, I'll investigate dns/dhcp later.

Comment 15

[Zandr Milewski [:zandr]]

I started looking at these Friday night, and inventory/dns/hostname are inconsistent, with at least two missing from inventory altogether.

Spencer, could you verify each of 

moz2-darwin10-slave{53...61} 
    try-mac64-slave(27...36}

And make sure that each machine is:

* in inventory with a correct IP address
* in DNS with a correct IP address and hostname
* has its hostname set to match
* is labeled to match the above.

Since everything else is software, I'd start by comparing labels to MAC addresses on the physical boxes, and work through DHCP to DNS to inventory.

Comment 16

[Zandr Milewski [:zandr]]

OK, verification complete.

Inventory/DNS/DHCP updated and verified against hostname and physical labeling.

Over to Releng for setup of new slaves:

moz2-darwin10-slave53-61
try-mac64-slave27,29-36

NB: There is no try-mac-64-slave28. That machine (the former try-mac-slave29) has a stripped screw that is currently preventing upgrade.

Comment 17

[Dustin J. Mitchell [:dustin] (he/him)]

This is still waiting for setup.  Bump the priority if we're seeing wait times that suggest these slaves will be useful.

Comment 18

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: m635964-puppet-manifests-r1.patch

I'll get puppet set up, and leave the re-mastering to later.

Comment 19

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: m635964-puppet-manifests-r2.patch

New version with the old machines removed

Comment 20

[Dustin J. Mitchell [:dustin] (he/him)]

OK, these machines are all talking to mv-production-puppet now, but not doing any buildbottish work.

Comment 21

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: m635964-buildbot-configs-r1.patch

This should add these machines to the staging configuration, for testing purposes.

Comment 22

[Dustin J. Mitchell [:dustin] (he/him)]

I'm seeing

Mar 25 12:51:07 production-puppet puppetmasterd[19135]: Allowing unauthenticated client try-mac64-slave38.build.mozilla.org(10.250.48.198) access to puppetca.getcert
Mar 25 12:51:07 production-puppet puppetmasterd[19135]: Certificate request does not match existing certificate; run 'puppetca --clean try-mac-slave28.build.mozilla.org'.

In the mpt-production-puppet logs.  Is this the missing try-mac64-slave28?  Or the previously-problematic try-mac-slave28?  Or both?

198.48.250.10.in-addr.arpa domain name pointer try-mac64-slave38.build.mozilla.org.

No forward DNS for either hostname

? (10.250.48.198) at 00:16:CB:B0:75:23 [ether] on eth0

Comment 23

[Dustin J. Mitchell [:dustin] (he/him)]

moz2-darwin10-slave53-61 are up and running on sm02 (via slave alloc, at that!)

Comment 24

[Dustin J. Mitchell [:dustin] (he/him)]

try-mac64-slave27,29-36 are up and running on sm02 as well.

Comment 25

[Amy Rich [:arr] [:arich]]

/etc/hosts entries for the re-tasked hosts removed from bm-admin01, and their nagios configuration modified.

Comment 26

[Dustin J. Mitchell [:dustin] (he/him)]

Let's hold off on moving these to production for a moment - joduinn is suggesting re-re-purposing them for developers, since we're now over-capacity on these builders.

Comment 27

[Aki Sasaki (not active)]

Attachment: new try mac64 slaves

Comment 28

[Aki Sasaki (not active)]

Comment on attachment 523403 [details] [diff] [review]
new try mac64 slaves

http://hg.mozilla.org/build/buildbot-configs/rev/9eaa42cbbc7c

Comment 29

[Dustin J. Mitchell [:dustin] (he/him)]

As of bug 647051, this bug is now only about

moz2-darwin10-slave53
moz2-darwin10-slave54
moz2-darwin10-slave55
moz2-darwin10-slave56
try-mac64-slave27
try-mac64-slave29
try-mac64-slave30
try-mac64-slave31

which are all still in AFK, powered on, and in staging.

Comment 30

[Aki Sasaki (not active)]

Attachment: adjust try macs, add moz2-darwin10 macs

Due to bug 647051.

Comment 31

[Aki Sasaki (not active)]

Comment on attachment 523641 [details] [diff] [review]
adjust try macs, add moz2-darwin10 macs

http://hg.mozilla.org/build/buildbot-configs/rev/912ae2b49222

Comment 32

[Dustin J. Mitchell [:dustin] (he/him)]

aki, if I switch the ssh keys and select a production pool in slavealloc, are the production masters ready to welcome these new slaves?

Comment 33

[Dustin J. Mitchell [:dustin] (he/him)]

In production now, via slavealloc (keys changed too, don't worry):

moz2-darwin10-slave53
moz2-darwin10-slave54
moz2-darwin10-slave55
moz2-darwin10-slave56
try-mac64-slave27
try-mac64-slave29
try-mac64-slave30
try-mac64-slave31

Comment 34

[Zandr Milewski [:zandr]]

? (10.250.48.198) at 00:16:CB:B0:75:23 [ether] on eth0

This is physically labeled as try-mac64-slave28, and forward DNS is correct for this host. Reverse, not so much. Fixed.

Comment 35

[Dustin J. Mitchell [:dustin] (he/him)]

^^ copied to bug 652983

Comment 36

[Armen [:armenzg]]

I am removing from our configs and from slave-alloc many of these slaves in bug 700705.