461228 - backup bm-xserve01 and then repurpose as 2nd PPC builder slave

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Summarising a discussion we've had in irc a few times, and just had again today. Please update if I missed something:

We dont have duplicate PPC xserves in production. If any one production PPC xserve dies, we'll have to close the mozilla1.8 tree, and block schedule security releases until we have the replacement PPC xserve configured and ready for use. Apple has discontinued PPC xserves, so we cant just buy one.  

To prevent this situation, we have another PPC xserve (bm-xserve01), which I believe is idle, and which I'd like to setup as a 2nd PPC builder slave. 

afaik, during this year, bm-xserve01 has been used for:
- partner build repacks. however, Kev has now moved repack work to dm-partnerdist01. There are still some files lingering here from those repacks, so Kev will manually cleanup and confirm in this bug when all his stuff is off.
- central mac update of all macs. This hasnt been used in last year/two and unclear if it ever worked. Also unclear if it works for only xserves or also for mac minis.
- occasional random mac script testing. This can be done on any other mac, like a mac mini or one of our staging xserves. 

I'm filing this bug to see if anyone can think of reason to keep bm-xserve01. If there's no objections, we'll go ahead with the backup and repurpose.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

no objections from anyone else, so we're going ahead with this as soon as Kev is finished copying off his files.

Comment 2

[Kev Needham [:kev]]

k, just need to double-check I have everything. will confirm by cob tomorrow.

Comment 3

[[e-mail] (:bhearsum)]

It looks like bm-xserve02 has had a hardware failure (bug 462693), we should probably repurpose this machine ASAP to fill that void. We're missing Tb 1.8 builds (important) and tb/fx 1.8 l10n currently.

Comment 4

[Kev Needham [:kev]]

will double-check and clear everything out in the next 30 minutes.

Comment 5

[Kev Needham [:kev]]

I have everything moved off, but have no access to the apache/php configs for. Ben, could you backup thunder's homedir and the php/cake install/config?

Comment 6

[[e-mail] (:bhearsum)]

(In reply to comment #5)
> I have everything moved off, but have no access to the apache/php configs for.
> Ben, could you backup thunder's homedir and the php/cake install/config?

Sure, where do you want me to put them?

Comment 7

[Kev Needham [:kev]]

in /tmp is fine. just let me know when they're available and I'll move them.

additionally, there is a mysql db running. I have taken an export of partner_repack, but if there's anything else, you may want to back it up, too.

Comment 8

[[e-mail] (:bhearsum)]

Alright, I copied ~thunder and /etc/httpd into /tmp/backup - you should have access to that entire directory. Other than inside of thunder's homedir I couldn't find any obvious references to php or cake.

AFAIK we haven't use mysql for anything on this machine, nothing to backup there.

Comment 9

[Kev Needham [:kev]]

Everything should be off now. Double-checked /var/private/mysql and there doesn't seem to be any other dbs.

Comment 10

[[e-mail] (:bhearsum)]

Thanks Kev.

Phong, do we have a 10.4 image we can try to image this machine with?

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

We should have an image we used when setting up all these PPC machines into the release automation. Originally we had an image of 02 (bug 391742, bug 391498), and more recently bm-xserve04 was imaged from 03 (bug 425052).

Comment 12

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Moving to correct component.

(In reply to comment #11)
> We should have an image we used when setting up all these PPC machines into the
> release automation. Originally we had an image of 02 (bug 391742, bug 391498),
> and more recently bm-xserve04 was imaged from 03 (bug 425052).

Phong, can you please backup first in case we need to restore this machine?

Comment 13

[Phong Tran [:phong]]

backed up to external drive.  which image should I restore this from?

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

The same image you're (attempting) to put on bm-xserve02, thanks.

Comment 15

[Phong Tran [:phong]]

re-imaged with BM-XSERVE02 image from April of this year.