Closed Bug 612452 Opened 14 years ago Closed 14 years ago

move remaining minis to scl1 redux

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jhford, Assigned: zandr)

References

Details

We should finish moving all remaining minis to the colo.

Lets start by figuring out which slaves are in MTV still and create batches to take them to the colo.
Based on conversations with jhford, it appears that these are the remaining machines:

talos-rev2-tiger15      IN  CNAME   talos-rev2-tiger15.build.mtv1.mozilla.com.
talos-rev2-x6408        IN  CNAME   talos-rev2-x648.build.mtv1.mozilla.com.
talos-r3-w7-001         IN  CNAME   talos-r3-w7-001.build.mtv1.mozilla.com.
talos-r3-w7-ref         IN  CNAME   talos-r3-w7-ref.build.mtv1.mozilla.com.
talos-r3-fed-ref        IN  CNAME   talos-r3-fed-ref.build.mtv1.mozilla.com.
talos-r3-fed64-ref      IN  CNAME   talos-r3-fed64-ref.build.mtv1.mozilla.com.
talos-r3-leopard-013    IN  CNAME   talos-r3-leopard-013.build.mtv1.mozilla.com.
talos-r3-leopard-014    IN  CNAME   talos-r3-leopard-14.build.mtv1.mozilla.com.
talos-r3-leopard-016    IN  CNAME   talos-r3-leopard-16.build.mtv1.mozilla.com.
talos-r3-leopard-017    IN  CNAME   talos-r3-leopard-17.build.mtv1.mozilla.com.
talos-r3-leopard-019    IN  CNAME   talos-r3-leopard-19.build.mtv1.mozilla.com.
talos-r3-w7-002         IN  CNAME   talos-r3-w7-2.build.mtv1.mozilla.com.
talos-r3-w7-003         IN  CNAME   talos-r3-w7-3.build.mtv1.mozilla.com.
talos-r3-w7-004         IN  CNAME   talos-r3-w7-4.build.mtv1.mozilla.com.
talos-r3-w7-006         IN  CNAME   talos-r3-w7-6.build.mtv1.mozilla.com.
talos-r3-w7-007         IN  CNAME   talos-r3-w7-7.build.mtv1.mozilla.com.
talos-r3-w7-008         IN  CNAME   talos-r3-w7-8.build.mtv1.mozilla.com.
talos-r3-w7-009         IN  CNAME   talos-r3-w7-9.build.mtv1.mozilla.com.
talos-r3-w7-010         IN  CNAME   talos-r3-w7-10.build.mtv1.mozilla.com.
talos-r3-w7-012         IN  CNAME   talos-r3-w7-12.build.mtv1.mozilla.com.
talos-r3-w7-013         IN  CNAME   talos-r3-w7-13.build.mtv1.mozilla.com.
talos-r3-fed-020        IN  CNAME   talos-r3-fed-20.build.mtv1.mozilla.com.
talos-r3-fed-009        IN  CNAME   talos-r3-fed-009.build.mtv1.mozilla.com.
talos-r3-fed-010        IN  CNAME   talos-r3-fed-10.build.mtv1.mozilla.com.
talos-r3-fed-018        IN  CNAME   talos-r3-fed-18.build.mtv1.mozilla.com.
talos-r3-fed-019        IN  CNAME   talos-r3-fed-19.build.mtv1.mozilla.com.
talos-r3-leopard-027    IN  CNAME   talos-r3-leopard-27.build.mtv1.mozilla.com.
talos-r3-leopard-028    IN  CNAME   talos-r3-leopard-28.build.mtv1.mozilla.com.
talos-r3-leopard-029    IN  CNAME   talos-r3-leopard-29.build.mtv1.mozilla.com.
talos-r3-leopard-030    IN  CNAME   talos-r3-leopard-30.build.mtv1.mozilla.com.
talos-r3-leopard-031    IN  CNAME   talos-r3-leopard-31.build.mtv1.mozilla.com.
talos-r3-leopard-032    IN  CNAME   talos-r3-leopard-32.build.mtv1.mozilla.com.
talos-r3-leopard-033    IN  CNAME   talos-r3-leopard-33.build.mtv1.mozilla.com.
talos-r3-leopard-034    IN  CNAME   talos-r3-leopard-34.build.mtv1.mozilla.com.
talos-r3-leopard-035    IN  CNAME   talos-r3-leopard-35.build.mtv1.mozilla.com.
talos-r3-leopard-036    IN  CNAME   talos-r3-leopard-36.build.mtv1.mozilla.com.
talos-r3-leopard-037    IN  CNAME   talos-r3-leopard-37.build.mtv1.mozilla.com.
talos-r3-leopard-038    IN  CNAME   talos-r3-leopard-38.build.mtv1.mozilla.com.
talos-r3-leopard-039    IN  CNAME   talos-r3-leopard-39.build.mtv1.mozilla.com.
talos-r3-leopard-040    IN  CNAME   talos-r3-leopard-40.build.mtv1.mozilla.com.
talos-r3-fed-029        IN  CNAME   talos-r3-fed-29.build.mtv1.mozilla.com.
talos-r3-fed-030        IN  CNAME   talos-r3-fed-30.build.mtv1.mozilla.com.
talos-r3-fed-034        IN  CNAME   talos-r3-fed-34.build.mtv1.mozilla.com.
talos-r3-fed-035        IN  CNAME   talos-r3-fed-35.build.mtv1.mozilla.com.
talos-r3-fed-036        IN  CNAME   talos-r3-fed-36.build.mtv1.mozilla.com.
talos-r3-fed-037        IN  CNAME   talos-r3-fed-37.build.mtv1.mozilla.com.
talos-r3-fed-038        IN  CNAME   talos-r3-fed-38.build.mtv1.mozilla.com.
talos-r3-snow-013       IN  CNAME   talos-r3-snow-13.build.mtv1.mozilla.com.
talos-r3-snow-014       IN  CNAME   talos-r3-snow-14.build.mtv1.mozilla.com.
talos-r3-snow-015       IN  CNAME   talos-r3-snow-15.build.mtv1.mozilla.com.
talos-r3-snow-016       IN  CNAME   talos-r3-snow-16.build.mtv1.mozilla.com.
talos-r3-snow-018       IN  CNAME   talos-r3-snow-18.build.mtv1.mozilla.com.
talos-r3-snow-020       IN  CNAME   talos-r3-snow-20.build.mtv1.mozilla.com.
talos-r3-leopard-ref    IN  CNAME   talos-r3-leopard-ref.build.mtv1.mozilla.com.
talos-r3-snow-ref       IN  CNAME   talos-r3-snow-ref.build.mtv1.mozilla.com. 

How shall we batch these?
The rev2 hosts are offline, ignore them.
Proposing two batches:

Batch 1:
talos-r3-fed-009,010,018,019,020,029 (6)
talos-r3-leopard-013,014,016,017,019,027,028,029,030,031 (10)
talos-r3-snow-013,014,015 (3)
talos-r3-w7-001,002,003,004,006,007 (6)

Batch 2:
talos-r3-fed-030,034,035,036,037,038,ref (7)
talos-r3-fed64-ref (1)
talos-r3-leopard-032,033,034,035,036,037,038,039,040,ref (10)
talos-r3-snow-016,018,020,ref (4)
talos-r3-w7-008,009,010,012,013,ref (6)

I'll have updated zone files ready to commit ahead of the move, and will physically tag the machines and power cords with color codes so we don't yank the wrong cables.
Doing a quick survey, all of the machines that I found were on the 2nd floor server room.

Machines I can't find: All five -ref machines, talos-r3-fed-010

Machines I found and wasn't expecting: talos-r3-w7-14, leopard-003

Any pointers?
Some notes, for lack of a better place:

fed-010 is live in scl1
w7-14 and leopard-003 are live in mtv1

Should I fix up DNS for the current state?

The ref machines are on the 3rd floor, along with two machines that don't have hostname labels, but by the asset tags are in inventory w7-ref and w764-ref (machines labelled as such are sitting next to them)

Not sure how to unravel those yet.

Given that we have a relatively small number of machines for each OS, could we do all of this in one batch?
See Also: → 612288
(In reply to comment #5)
> Some notes, for lack of a better place:
> 
> fed-010 is live in scl1
> w7-14 and leopard-003 are live in mtv1
> 
> Should I fix up DNS for the current state?

Yes, please.

> 
> The ref machines are on the 3rd floor, along with two machines that don't have
> hostname labels, but by the asset tags are in inventory w7-ref and w764-ref
> (machines labelled as such are sitting next to them)
> 
> Not sure how to unravel those yet.

Lets leave them alone for now, i wonder if there were two reference images created to test a change.  who knows

> Given that we have a relatively small number of machines for each OS, could we
> do all of this in one batch?

If we need to do this before I can fix the slaves left over from Friday, I'd like to keep it at two batches.
(In reply to comment #6)
> (In reply to comment #5)
> > Some notes, for lack of a better place:
> > 
> > fed-010 is live in scl1
> > w7-14 and leopard-003 are live in mtv1
> > 
> > Should I fix up DNS for the current state?
> 
> Yes, please.

Done, fed-010, w7-14 and leopard-003 should now be accessible from build.m.o hostnames.

> If we need to do this before I can fix the slaves left over from Friday, I'd
> like to keep it at two batches.

I'm unaware of any time pressures from my side. If waiting a couple of days for the changes from last week to get ironed out lets me to the move in one batch, that's a win for me. (Power in the 2nd floor server room is a *mess*... if we can power that all down at once, we can avoid a lot of possibility for error)
(In reply to comment #7)
> Done, fed-010, w7-14 and leopard-003 should now be accessible from build.m.o
> hostnames.

Thanks :D

> I'm unaware of any time pressures from my side. If waiting a couple of days for
> the changes from last week to get ironed out lets me to the move in one batch,
> that's a win for me. (Power in the 2nd floor server room is a *mess*... if we
> can power that all down at once, we can avoid a lot of possibility for error)

Yah, I agree.  I hope to get the machines ironed out today.  Would thursday work for you?
Sounds great, let's plan for Thursday.
talos-r3-leopard-013 is pointing to talos-r3-leopard-013.build.mtv1 when it should be pointing to leopard-13.build.mtv1

This is temporary and depending on how many slaves are having this problem, might not be worth fixing before the move
(In reply to comment #10)
> talos-r3-leopard-013 is pointing to talos-r3-leopard-013.build.mtv1 when it
> should be pointing to leopard-13.build.mtv1

to be more specific, it should be pointing to 

talos-r3-leopard-13.build.mtv1.mozilla.com

> This is temporary and depending on how many slaves are having this problem,
> might not be worth fixing since the plan is to move them tomorrow
leopard-[0]13 was one of the machines that was allegedly moved, but I found on the 2nd floor.

Fixed. 

Inconsistent numbering makes me sad.
Ref machines are fine, btw...transcription error on my part. Sounds like we can move those at any time, so I'm going to ignore them for now.
(In reply to comment #12)
> leopard-[0]13 was one of the machines that was allegedly moved, but I found on
> the 2nd floor.

Um, ignore the above. Still fixed, though.
Revised move list: (NB: These are post-move CNAMES, /not/ what's currently in DNS)


+talos-r3-fed-009        IN  CNAME   talos-r3-fed-009.build.scl1.mozilla.com.
+talos-r3-fed-018        IN  CNAME   talos-r3-fed-018.build.scl1.mozilla.com.
+talos-r3-fed-019        IN  CNAME   talos-r3-fed-019.build.scl1.mozilla.com.
+talos-r3-fed-020        IN  CNAME   talos-r3-fed-020.build.scl1.mozilla.com.
+talos-r3-fed-029        IN  CNAME   talos-r3-fed-029.build.scl1.mozilla.com.
+talos-r3-fed-030        IN  CNAME   talos-r3-fed-030.build.scl1.mozilla.com.
+talos-r3-fed-034        IN  CNAME   talos-r3-fed-034.build.scl1.mozilla.com.
+talos-r3-fed-035        IN  CNAME   talos-r3-fed-035.build.scl1.mozilla.com.
+talos-r3-fed-036        IN  CNAME   talos-r3-fed-036.build.scl1.mozilla.com.
+talos-r3-fed-037        IN  CNAME   talos-r3-fed-037.build.scl1.mozilla.com.
+talos-r3-fed-038        IN  CNAME   talos-r3-fed-038.build.scl1.mozilla.com.
+talos-r3-leopard-003    IN  CNAME   talos-r3-leopard-003.build.scl1.mozilla.com.
+talos-r3-leopard-013    IN  CNAME   talos-r3-leopard-013.build.scl1.mozilla.com.
+talos-r3-leopard-014    IN  CNAME   talos-r3-leopard-014.build.scl1.mozilla.com.
+talos-r3-leopard-016    IN  CNAME   talos-r3-leopard-016.build.scl1.mozilla.com.
+talos-r3-leopard-017    IN  CNAME   talos-r3-leopard-017.build.scl1.mozilla.com.
+talos-r3-leopard-019    IN  CNAME   talos-r3-leopard-019.build.scl1.mozilla.com.
+talos-r3-leopard-027    IN  CNAME   talos-r3-leopard-027.build.scl1.mozilla.com.
+talos-r3-leopard-028    IN  CNAME   talos-r3-leopard-028.build.scl1.mozilla.com.
+talos-r3-leopard-029    IN  CNAME   talos-r3-leopard-029.build.scl1.mozilla.com.
+talos-r3-leopard-030    IN  CNAME   talos-r3-leopard-030.build.scl1.mozilla.com.
+talos-r3-leopard-031    IN  CNAME   talos-r3-leopard-031.build.scl1.mozilla.com.
+talos-r3-leopard-032    IN  CNAME   talos-r3-leopard-032.build.scl1.mozilla.com.
+talos-r3-leopard-033    IN  CNAME   talos-r3-leopard-033.build.scl1.mozilla.com.
+talos-r3-leopard-034    IN  CNAME   talos-r3-leopard-034.build.scl1.mozilla.com.
+talos-r3-leopard-035    IN  CNAME   talos-r3-leopard-035.build.scl1.mozilla.com.
+talos-r3-leopard-036    IN  CNAME   talos-r3-leopard-036.build.scl1.mozilla.com.
+talos-r3-leopard-037    IN  CNAME   talos-r3-leopard-037.build.scl1.mozilla.com.
+talos-r3-leopard-038    IN  CNAME   talos-r3-leopard-038.build.scl1.mozilla.com.
+talos-r3-leopard-039    IN  CNAME   talos-r3-leopard-039.build.scl1.mozilla.com.
+talos-r3-leopard-040    IN  CNAME   talos-r3-leopard-040.build.scl1.mozilla.com.
+talos-r3-snow-013       IN  CNAME   talos-r3-snow-013.build.scl1.mozilla.com.
+talos-r3-snow-014       IN  CNAME   talos-r3-snow-014.build.scl1.mozilla.com.
+talos-r3-snow-015       IN  CNAME   talos-r3-snow-015.build.scl1.mozilla.com.
+talos-r3-snow-016       IN  CNAME   talos-r3-snow-016.build.scl1.mozilla.com.
+talos-r3-snow-018       IN  CNAME   talos-r3-snow-018.build.scl1.mozilla.com.
+talos-r3-snow-020       IN  CNAME   talos-r3-snow-020.build.scl1.mozilla.com.
+talos-r3-w7-001         IN  CNAME   talos-r3-w7-001.build.scl1.mozilla.com.
+talos-r3-w7-002         IN  CNAME   talos-r3-w7-002.build.scl1.mozilla.com.
+talos-r3-w7-003         IN  CNAME   talos-r3-w7-003.build.scl1.mozilla.com.
+talos-r3-w7-004         IN  CNAME   talos-r3-w7-004.build.scl1.mozilla.com.
+talos-r3-w7-006         IN  CNAME   talos-r3-w7-006.build.scl1.mozilla.com.
+talos-r3-w7-007         IN  CNAME   talos-r3-w7-007.build.scl1.mozilla.com.
+talos-r3-w7-008         IN  CNAME   talos-r3-w7-008.build.scl1.mozilla.com.
+talos-r3-w7-009         IN  CNAME   talos-r3-w7-009.build.scl1.mozilla.com.
+talos-r3-w7-010         IN  CNAME   talos-r3-w7-010.build.scl1.mozilla.com.
+talos-r3-w7-012         IN  CNAME   talos-r3-w7-012.build.scl1.mozilla.com.
+talos-r3-w7-013         IN  CNAME   talos-r3-w7-013.build.scl1.mozilla.com.
+talos-r3-w7-014         IN  CNAME   talos-r3-w7-014.build.scl1.mozilla.com.
Ok, it looks like the majority of slaves that are not reporting to buildbot are either can't be pinged or are at mountain view.  Because of this, I think that the best choice is to move all remaining minis in mountain view to scl.  


I will start shutting these slaves down once I get confirmation that we are going ahead with the move tomorrow morning.
Sorry, didn't see this until this morning, and I don't think well have time to get this done right before I need to be back in MV this afternoon. Let's plan for tomorrow (Friday) morning to move these to scl1.
Assignee: server-ops → zandr
talos-r3-fed-009 -- not pingable
talos-r3-fed-018 -- not pingable
talos-r3-fed-019 -- not pingable
talos-r3-fed-020 -- not pingable
talos-r3-fed-029 -- off
talos-r3-fed-030 -- not pingable
talos-r3-fed-034 -- off
talos-r3-fed-035 -- not pingable
talos-r3-fed-036 -- not pingable
talos-r3-fed-037 -- off
talos-r3-fed-038 -- off

talos-r3-leopard-003 -- turned off, weird state
talos-r3-leopard-013 -- renamed buildbot.tac, still needs puppet work
talos-r3-leopard-014 -- renamed buildbot.tac, still needs puppet work
talos-r3-leopard-016 -- not pingable
talos-r3-leopard-017 -- renamed buildbot.tac, still needs puppet work
talos-r3-leopard-019 -- renamed buildbot.tac, still needs puppet work
talos-r3-leopard-027 -- renamed buildbot.tac, still needs puppet work
talos-r3-leopard-028 -- renamed buildbot.tac, still needs puppet work
talos-r3-leopard-029 -- renamed buildbot.tac, still needs puppet work
talos-r3-leopard-030 -- renamed buildbot.tac, still needs puppet work
talos-r3-leopard-031 -- renamed buildbot.tac, still needs puppet work
talos-r3-leopard-032 -- renamed buildbot.tac, still needs puppet work
talos-r3-leopard-033 -- renamed buildbot.tac, still needs puppet work
talos-r3-leopard-034 -- renamed buildbot.tac, still needs puppet work
talos-r3-leopard-035 -- renamed buildbot.tac, still needs puppet work
talos-r3-leopard-036 -- renamed buildbot.tac, still needs puppet work
talos-r3-leopard-037 -- renamed buildbot.tac, still needs puppet work
talos-r3-leopard-038 -- renamed buildbot.tac, still needs puppet work
talos-r3-leopard-039 -- renamed buildbot.tac, still needs puppet work
talos-r3-leopard-040 -- renamed buildbot.tac, still needs puppet work

talos-r3-snow-013 -- renamed buildbot.tac, still needs puppet work
talos-r3-snow-014 -- not pingable
talos-r3-snow-015 -- renamed buildbot.tac, still needs puppet work
talos-r3-snow-016 -- renamed buildbot.tac, still needs puppet work
talos-r3-snow-018 -- renamed buildbot.tac, still needs puppet work
talos-r3-snow-020 -- renamed buildbot.tac, still needs puppet work

talos-r3-w7-001 -- buildbot was already off
talos-r3-w7-002 -- buildbot was already off
talos-r3-w7-003 -- buildbot was already off
talos-r3-w7-004 -- buildbot was already off
talos-r3-w7-006 -- buildbot was already off
talos-r3-w7-007 -- buildbot was already off
talos-r3-w7-008 -- buildbot was already off
talos-r3-w7-009 -- buildbot was already off
talos-r3-w7-010 -- buildbot was already off
talos-r3-w7-012 -- buildbot was already off
talos-r3-w7-013 -- buildbot was already off
talos-r3-w7-014 -- buildbot was already off
(In reply to comment #18)
> talos-r3-snow-013 -- renamed buildbot.tac, still needs puppet work
> talos-r3-snow-014 -- not pingable
> talos-r3-snow-015 -- renamed buildbot.tac, still needs puppet work
> talos-r3-snow-016 -- renamed buildbot.tac, still needs puppet work
> talos-r3-snow-018 -- renamed buildbot.tac, still needs puppet work
> talos-r3-snow-020 -- renamed buildbot.tac, still needs puppet work

accidentally turned these off before making the required puppet changes.  Need to boot them to clear the ssl keys and change the puppet master.
(In reply to comment #18)
> talos-r3-fed-009 -- not pingable
> talos-r3-fed-018 -- not pingable
> talos-r3-fed-019 -- not pingable
> talos-r3-fed-020 -- not pingable
> talos-r3-fed-029 -- off
> talos-r3-fed-030 -- not pingable
> talos-r3-fed-034 -- off
> talos-r3-fed-035 -- not pingable
> talos-r3-fed-036 -- not pingable
> talos-r3-fed-037 -- off
> talos-r3-fed-038 -- off
> 
> talos-r3-leopard-003 -- turned off, weird state
> talos-r3-leopard-013 -- renamed buildbot.tac, still needs puppet work
> talos-r3-leopard-014 -- renamed buildbot.tac, still needs puppet work
> talos-r3-leopard-016 -- not pingable
> talos-r3-leopard-017 -- renamed buildbot.tac, still needs puppet work
> talos-r3-leopard-019 -- renamed buildbot.tac, still needs puppet work
> talos-r3-leopard-027 -- renamed buildbot.tac, still needs puppet work
> talos-r3-leopard-028 -- renamed buildbot.tac, still needs puppet work
> talos-r3-leopard-029 -- renamed buildbot.tac, still needs puppet work
> talos-r3-leopard-030 -- renamed buildbot.tac, still needs puppet work
> talos-r3-leopard-031 -- renamed buildbot.tac, still needs puppet work
> talos-r3-leopard-032 -- renamed buildbot.tac, still needs puppet work
> talos-r3-leopard-033 -- renamed buildbot.tac, still needs puppet work
> talos-r3-leopard-034 -- renamed buildbot.tac, still needs puppet work
> talos-r3-leopard-035 -- renamed buildbot.tac, still needs puppet work
> talos-r3-leopard-036 -- renamed buildbot.tac, still needs puppet work
> talos-r3-leopard-037 -- renamed buildbot.tac, still needs puppet work
> talos-r3-leopard-038 -- renamed buildbot.tac, still needs puppet work
> talos-r3-leopard-039 -- renamed buildbot.tac, still needs puppet work
> talos-r3-leopard-040 -- renamed buildbot.tac, still needs puppet work
> 
> talos-r3-snow-013 -- renamed buildbot.tac, still needs puppet work
> talos-r3-snow-014 -- not pingable
> talos-r3-snow-015 -- renamed buildbot.tac, still needs puppet work
> talos-r3-snow-016 -- renamed buildbot.tac, still needs puppet work
> talos-r3-snow-018 -- renamed buildbot.tac, still needs puppet work
> talos-r3-snow-020 -- renamed buildbot.tac, still needs puppet work
> 
> talos-r3-w7-001 -- buildbot was already off
> talos-r3-w7-002 -- buildbot was already off
> talos-r3-w7-003 -- buildbot was already off
> talos-r3-w7-004 -- buildbot was already off
> talos-r3-w7-006 -- buildbot was already off
> talos-r3-w7-007 -- buildbot was already off
> talos-r3-w7-008 -- buildbot was already off
> talos-r3-w7-009 -- buildbot was already off
> talos-r3-w7-010 -- buildbot was already off
> talos-r3-w7-012 -- buildbot was already off
> talos-r3-w7-013 -- buildbot was already off
> talos-r3-w7-014 -- buildbot was already off

all slaves either off or not pingable.  After a reboot (thanks Zandr!), I was able to prepare talos-r3-snow-014.

The fedora slaves that weren't pingable are going to be brought up *without* network to ensure that they boot and to change the required puppet settings before attempting to connect to the scl puppet.

Lets take these machines apart.
Just pushed the DNS changes so the build.mozilla.org CNAMEs now point at scl1 A records.
> > talos-r3-leopard-003 -- turned off, weird state

Don't know what the state of this staging slave is

> > talos-r3-leopard-013 -- renamed buildbot.tac, still needs puppet work
> > talos-r3-leopard-014 -- renamed buildbot.tac, still needs puppet work
> > talos-r3-leopard-016 -- not pingable

Probably needs reimaging

> > talos-r3-leopard-017 -- renamed buildbot.tac, still needs puppet work
> > talos-r3-leopard-019 -- renamed buildbot.tac, still needs puppet work
> > talos-r3-leopard-027 -- renamed buildbot.tac, still needs puppet work
> > talos-r3-leopard-028 -- renamed buildbot.tac, still needs puppet work
> > talos-r3-leopard-029 -- renamed buildbot.tac, still needs puppet work
> > talos-r3-leopard-030 -- renamed buildbot.tac, still needs puppet work
> > talos-r3-leopard-031 -- renamed buildbot.tac, still needs puppet work
> > talos-r3-leopard-032 -- renamed buildbot.tac, still needs puppet work
> > talos-r3-leopard-033 -- renamed buildbot.tac, still needs puppet work
> > talos-r3-leopard-034 -- renamed buildbot.tac, still needs puppet work
> > talos-r3-leopard-035 -- renamed buildbot.tac, still needs puppet work
> > talos-r3-leopard-036 -- renamed buildbot.tac, still needs puppet work
> > talos-r3-leopard-037 -- renamed buildbot.tac, still needs puppet work
> > talos-r3-leopard-038 -- renamed buildbot.tac, still needs puppet work
> > talos-r3-leopard-039 -- renamed buildbot.tac, still needs puppet work
> > talos-r3-leopard-040 -- renamed buildbot.tac, still needs puppet work


All other leopard slaves back online and running jobs
leopard 36,37,38,40 turned off to avoid overloading pdu
> talos-r3-snow-013 -- renamed buildbot.tac, still needs puppet work
> talos-r3-snow-014 -- not pingable
> talos-r3-snow-015 -- renamed buildbot.tac, still needs puppet work
> talos-r3-snow-016 -- renamed buildbot.tac, still needs puppet work
> talos-r3-snow-018 -- renamed buildbot.tac, still needs puppet work
> talos-r3-snow-020 -- renamed buildbot.tac, still needs puppet work

All online and accounted for
(In reply to comment #23)
> leopard 36,37,38,40 turned off to avoid overloading pdu

these were on the wrong pdu, instead we took down some of the offline fedora64 slaves.
fedora 29, 34, 37,38 are all connected to the buildbot master
Depends on: 613719
Declaring victory here. There's some cleanup to be done at a future downtime, but the machines are moved with power and net.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.