Closed Bug 611441 Opened 15 years ago Closed 15 years ago

Move remaining Castro minis to SCL

Categories

(mozilla.org Graveyard :: Server Operations, task)

All
Other
task
Not set
minor

Tracking

(Not tracked)

RESOLVED INCOMPLETE

People

(Reporter: jlaz, Assigned: jlaz)

References

Details

(Whiteboard: [buildduty])

These are the remaining slaves to move from Castro: talos-r3-fed-001 IN CNAME talos-r3-fed-1.build.mtv1.mozilla.com. talos-r3-fed-002 IN CNAME talos-r3-fed-2.build.mtv1.mozilla.com. talos-r3-fed-004 IN CNAME talos-r3-fed-4.build.mtv1.mozilla.com. talos-r3-fed-005 IN CNAME talos-r3-fed-5.build.mtv1.mozilla.com. talos-r3-fed-006 IN CNAME talos-r3-fed-6.build.mtv1.mozilla.com. talos-r3-fed-007 IN CNAME talos-r3-fed-7.build.mtv1.mozilla.com. talos-r3-fed-008 IN CNAME talos-r3-fed-8.build.mtv1.mozilla.com. talos-r3-fed-009 IN CNAME talos-r3-fed-9.build.mtv1.mozilla.com. talos-r3-fed-010 IN CNAME talos-r3-fed-10.build.mtv1.mozilla.com. talos-r3-fed-011 IN CNAME talos-r3-fed-11.build.mtv1.mozilla.com. talos-r3-fed-013 IN CNAME talos-r3-fed-13.build.mtv1.mozilla.com. talos-r3-fed-017 IN CNAME talos-r3-fed-17.build.mtv1.mozilla.com. talos-r3-fed-018 IN CNAME talos-r3-fed-18.build.mtv1.mozilla.com. talos-r3-fed-019 IN CNAME talos-r3-fed-19.build.mtv1.mozilla.com. talos-r3-fed-020 IN CNAME talos-r3-fed-20.build.mtv1.mozilla.com. talos-r3-fed-029 IN CNAME talos-r3-fed-29.build.mtv1.mozilla.com. talos-r3-fed-030 IN CNAME talos-r3-fed-30.build.mtv1.mozilla.com. talos-r3-fed-034 IN CNAME talos-r3-fed-34.build.mtv1.mozilla.com. talos-r3-fed-035 IN CNAME talos-r3-fed-35.build.mtv1.mozilla.com. talos-r3-fed-036 IN CNAME talos-r3-fed-36.build.mtv1.mozilla.com. talos-r3-fed-037 IN CNAME talos-r3-fed-37.build.mtv1.mozilla.com. talos-r3-fed-038 IN CNAME talos-r3-fed-38.build.mtv1.mozilla.com. talos-r3-fed-039 IN CNAME talos-r3-fed-39.build.mtv1.mozilla.com. talos-r3-fed-ref IN CNAME talos-r3-fed-ref.build.mtv1.mozilla.com. talos-r3-fed64-017 IN CNAME talos-r3-fed64-17.build.mtv1.mozilla.com. talos-r3-fed64-018 IN CNAME talos-r3-fed64-18.build.mtv1.mozilla.com. talos-r3-fed64-019 IN CNAME talos-r3-fed64-19.build.mtv1.mozilla.com. talos-r3-fed64-020 IN CNAME talos-r3-fed64-20.build.mtv1.mozilla.com. talos-r3-fed64-ref IN CNAME talos-r3-fed64-ref.build.mtv1.mozilla.com. talos-r3-leopard-001 IN CNAME talos-r3-leopard-1.build.mtv1.mozilla.com. talos-r3-leopard-002 IN CNAME talos-r3-leopard-2.build.mtv1.mozilla.com. talos-r3-leopard-003 IN CNAME talos-r3-leopard-3.build.mtv1.mozilla.com. talos-r3-leopard-005 IN CNAME talos-r3-leopard-5.build.mtv1.mozilla.com. talos-r3-leopard-006 IN CNAME talos-r3-leopard-6.build.mtv1.mozilla.com. talos-r3-leopard-007 IN CNAME talos-r3-leopard-7.build.mtv1.mozilla.com. talos-r3-leopard-008 IN CNAME talos-r3-leopard-8.build.mtv1.mozilla.com. talos-r3-leopard-009 IN CNAME talos-r3-leopard-9.build.mtv1.mozilla.com. talos-r3-leopard-013 IN CNAME talos-r3-leopard-13.build.mtv1.mozilla.com. talos-r3-leopard-014 IN CNAME talos-r3-leopard-14.build.mtv1.mozilla.com. talos-r3-leopard-015 IN CNAME talos-r3-leopard-15.build.mtv1.mozilla.com. talos-r3-leopard-016 IN CNAME talos-r3-leopard-16.build.mtv1.mozilla.com. talos-r3-leopard-017 IN CNAME talos-r3-leopard-17.build.mtv1.mozilla.com. talos-r3-leopard-018 IN CNAME talos-r3-leopard-18.build.mtv1.mozilla.com. talos-r3-leopard-019 IN CNAME talos-r3-leopard-19.build.mtv1.mozilla.com. talos-r3-leopard-027 IN CNAME talos-r3-leopard-27.build.mtv1.mozilla.com. talos-r3-leopard-028 IN CNAME talos-r3-leopard-28.build.mtv1.mozilla.com. talos-r3-leopard-029 IN CNAME talos-r3-leopard-29.build.mtv1.mozilla.com. talos-r3-leopard-030 IN CNAME talos-r3-leopard-30.build.mtv1.mozilla.com. talos-r3-leopard-031 IN CNAME talos-r3-leopard-31.build.mtv1.mozilla.com. talos-r3-leopard-032 IN CNAME talos-r3-leopard-32.build.mtv1.mozilla.com. talos-r3-leopard-033 IN CNAME talos-r3-leopard-33.build.mtv1.mozilla.com. talos-r3-leopard-034 IN CNAME talos-r3-leopard-34.build.mtv1.mozilla.com. talos-r3-leopard-035 IN CNAME talos-r3-leopard-35.build.mtv1.mozilla.com. talos-r3-leopard-036 IN CNAME talos-r3-leopard-36.build.mtv1.mozilla.com. talos-r3-leopard-037 IN CNAME talos-r3-leopard-37.build.mtv1.mozilla.com. talos-r3-leopard-038 IN CNAME talos-r3-leopard-38.build.mtv1.mozilla.com. talos-r3-leopard-039 IN CNAME talos-r3-leopard-39.build.mtv1.mozilla.com. talos-r3-leopard-040 IN CNAME talos-r3-leopard-40.build.mtv1.mozilla.com. talos-r3-leopard-ref IN CNAME talos-r3-leopard-ref.build.mtv1.mozilla.com. talos-r3-snow-013 IN CNAME talos-r3-snow-13.build.mtv1.mozilla.com. talos-r3-snow-014 IN CNAME talos-r3-snow-14.build.mtv1.mozilla.com. talos-r3-snow-015 IN CNAME talos-r3-snow-15.build.mtv1.mozilla.com. talos-r3-snow-016 IN CNAME talos-r3-snow-16.build.mtv1.mozilla.com. talos-r3-snow-017 IN CNAME talos-r3-snow-17.build.mtv1.mozilla.com. talos-r3-snow-018 IN CNAME talos-r3-snow-18.build.mtv1.mozilla.com. talos-r3-snow-019 IN CNAME talos-r3-snow-19.build.mtv1.mozilla.com. talos-r3-snow-020 IN CNAME talos-r3-snow-20.build.mtv1.mozilla.com. talos-r3-snow-ref IN CNAME talos-r3-snow-ref.build.mtv1.mozilla.com. talos-r3-w7-001 IN CNAME talos-r3-w7-1.build.mtv1.mozilla.com. talos-r3-w7-002 IN CNAME talos-r3-w7-2.build.mtv1.mozilla.com. talos-r3-w7-003 IN CNAME talos-r3-w7-3.build.mtv1.mozilla.com. talos-r3-w7-004 IN CNAME talos-r3-w7-4.build.mtv1.mozilla.com. talos-r3-w7-006 IN CNAME talos-r3-w7-6.build.mtv1.mozilla.com. talos-r3-w7-007 IN CNAME talos-r3-w7-7.build.mtv1.mozilla.com. talos-r3-w7-008 IN CNAME talos-r3-w7-8.build.mtv1.mozilla.com. talos-r3-w7-009 IN CNAME talos-r3-w7-9.build.mtv1.mozilla.com. talos-r3-w7-010 IN CNAME talos-r3-w7-10.build.mtv1.mozilla.com. talos-r3-w7-012 IN CNAME talos-r3-w7-12.build.mtv1.mozilla.com. talos-r3-w7-013 IN CNAME talos-r3-w7-13.build.mtv1.mozilla.com. talos-r3-w7-014 IN CNAME talos-r3-w7-14.build.mtv1.mozilla.com. talos-r3-w7-015 IN CNAME talos-r3-w7-15.build.mtv1.mozilla.com. talos-r3-w7-016 IN CNAME talos-r3-w7-16.build.mtv1.mozilla.com. talos-r3-w7-017 IN CNAME talos-r3-w7-17.build.mtv1.mozilla.com. talos-r3-w7-018 IN CNAME talos-r3-w7-18.build.mtv1.mozilla.com. talos-r3-w7-ref IN CNAME talos-r3-w7-ref.build.mtv1.mozilla.com. talos-r3-xp-001 IN CNAME talos-r3-xp-1.build.mtv1.mozilla.com. talos-r3-xp-002 IN CNAME talos-r3-xp-2.build.mtv1.mozilla.com. talos-r3-xp-003 IN CNAME talos-r3-xp-3.build.mtv1.mozilla.com. talos-r3-xp-005 IN CNAME talos-r3-xp-5.build.mtv1.mozilla.com. talos-r3-xp-006 IN CNAME talos-r3-xp-6.build.mtv1.mozilla.com. talos-r3-xp-007 IN CNAME talos-r3-xp-7.build.mtv1.mozilla.com. talos-r3-xp-008 IN CNAME talos-r3-xp-8.build.mtv1.mozilla.com. talos-r3-xp-009 IN CNAME talos-r3-xp-9.build.mtv1.mozilla.com. talos-r3-xp-013 IN CNAME talos-r3-xp-13.build.mtv1.mozilla.com. talos-r3-xp-014 IN CNAME talos-r3-xp-14.build.mtv1.mozilla.com. talos-r3-xp-018 IN CNAME talos-r3-xp-18.build.mtv1.mozilla.com. talos-r3-xp-019 IN CNAME talos-r3-xp-19.build.mtv1.mozilla.com. talos-r3-xp-020 IN CNAME talos-r3-xp-20.build.mtv1.mozilla.com. talos-r3-xp-032 IN CNAME talos-r3-xp-32.build.mtv1.mozilla.com. talos-r3-xp-033 IN CNAME talos-r3-xp-33.build.mtv1.mozilla.com. talos-r3-xp-034 IN CNAME talos-r3-xp-34.build.mtv1.mozilla.com. talos-r3-xp-035 IN CNAME talos-r3-xp-35.build.mtv1.mozilla.com. talos-r3-xp-036 IN CNAME talos-r3-xp-36.build.mtv1.mozilla.com. talos-r3-xp-037 IN CNAME talos-r3-xp-37.build.mtv1.mozilla.com. talos-r3-xp-038 IN CNAME talos-r3-xp-38.build.mtv1.mozilla.com. talos-r3-xp-039 IN CNAME talos-r3-xp-39.build.mtv1.mozilla.com. talos-r3-xp-040 IN CNAME talos-r3-xp-40.build.mtv1.mozilla.com.
Group: infra
I am proposing 4 trips to the colo. This would not require us to have to close the tree since we are only removing a MAX of 8 slaves per OS. I have also made the subsets in such a way that less minis have to be moved per trip (moving fed64 and snow on the last two trips). The long pole in here is leopard as there are 29 slaves. If we wanted we could move 10 slaves at a time for leopard so we could manage to do 3 trips. I am ignoring ref machines on this list as they can go anytime you guys want without our explicit intervention. IT please let me know if this works and/or if needs modifications. I will have the first set disconnected for 9AM PDT and ready to be departed if I see someone assigned to the bug and has ACKed the proposal. TRIP 1 ------ talos-r3-fed-00{1-2,4-9} - 8 slaves talos-r3-leopard-0{01-03,05-09} - 8 slaves talos-r3-w7-0{01-04,06-09} - 8 slaves talos-r3-xp-0{01-03,05-09} - 8 slaves TOTAL: 32 TRIP 2 ------ talos-r3-fed-0{10,11,13,17-20,29} - 8 slaves talos-r3-leopard-0{13-19,27} - 8 slaves talos-r3-w7-0{10,12-18} - 8 slaves talos-r3-xp-0{13-14,18-20,32-34} - 8 slaves TOTAL: 32 TRIP 3 ------ talos-r3-fed-0{30,34-39} - 8 slaves talos-r3-leopard-0{28-35} - 8 slaves talos-r3-xp-0{35-40} - 6 slaves TOTAL: 22 TRIP 4 ------ talos-r3-leopard-0{36-40} - 5 slaves talos-r3-snow-0{13-20} - 8 slaves talos-r3-fed64-017-20 - 4 slaves TOTAL: 17
Whiteboard: [buildduty]
Armen, could we possibly do this one hour earlier and start at 8 AM (or what is the earliest hour we can start? )
Assignee: server-ops → jlazaro
Flags: colo-trip+
I am going to start preparing these. ping me once you are online.
Revised 2nd batch: talos-r3-xp-013 talos-r3-xp-019 talos-r3-xp-020 talos-r3-leopard-003 talos-r3-leopard-013 talos-r3-leopard-014 talos-r3-leopard-016 talos-r3-leopard-017 talos-r3-leopard-018 talos-r3-leopard-019 talos-r3-fed-0{10,11,13,17-20,29} - 8 slaves talos-r3-w7-0{10,12-18} - 8 slaves
(In reply to comment #4) Out of this batch: > talos-r3-fed-0{10,11,13,17-20,29} - 8 slaves only the following are still pingable: * talos-r3-fed-0{17,29} They probably got moved with the first batch.
moz2-darwin10-slave (40-50) need to be taken offline to free up a switch
(In reply to comment #1) > TRIP 1 DONE: talos-r3-fed-00{1-2,4-8} talos-r3-xp-00{1-3,5-9} NOT DONE: talos-r3-leopard-00{1-3,5-9} talos-r3-w7-00{1-4,6-9} OFFLINE: talos-r3-fed-009
Done: talos-r3-fed-010
We are very very behind schedule and IT is nowhere to be found. This is the latest status update: - trip 1 - leopard and w7 slaves are not reachable - DNS problems - trip 2, trip3 & trip4 still @MV What is the plan for the rest of the day? - let's only finish batch2 (except leopard slaves) - let's debug the DNS issues for the leopard and win7 machines from trip#1 - let's do batches 3 & 4 next week in one shot Detailed summary ################ TRIP 1 DONE: talos-r3-fed-00{1-2,4-8} @ SCL talos-r3-xp-00{1-3,5-9} @ SCL CANNOT BE REACHED: talos-r3-leopard-00{1-3,5-9} @ SCL talos-r3-w7-00{1-4,6-9} @ SCL SIDE NOTE: There were more slaves taken to SCL: - We *believe* that these slaves were taken to SCL: - talos-r3-fed-0{10,11,13,18-20} - I don't know if more slaves were taken TRIP 2 @ MV - We are leaving behind talos-r3-leopard-0{13-19,27} because we still have leopard slaves down from first batch - We are still (if IT comes back) moving the following slaves: talos-r3-w7-0{10,12-18} - 8 slaves talos-r3-xp-0{13-14,18-20,32-34} - 8 slaves TRIP 3 & 4 @ MV - nothing has changed
Depends on: 611830
To make it more clear I will expand on the leopard slaves. We need to fix bug 611830 before we can move more Leopard and/or Win7 slaves. The following set from batch#2 has been moved to a switch (thanks jhford!!) that will be left behind at MV *and* I have put these slaves back into the production pool since we were falling behind for Leopard jobs. > talos-r3-leopard-0{13-19,27} jhford has volunteered to coordinate this with IT after I am gone.
Blocks: 611846
Host "talos-r3-fed-001" is at SCL Host "talos-r3-fed-002" is at SCL Host "talos-r3-fed-004" is at SCL Host "talos-r3-fed-005" is at SCL Host "talos-r3-fed-006" is at SCL Host "talos-r3-fed-007" is at SCL Host "talos-r3-fed-008" is at SCL Host "talos-r3-fed-009" is down Host "talos-r3-fed-010" is at SCL Host "talos-r3-fed-011" is down Host "talos-r3-fed-013" is down Host "talos-r3-fed-017" is down Host "talos-r3-fed-018" is not up at either SCL or MTV Host "talos-r3-fed-019" is down Host "talos-r3-fed-020" is down Host "talos-r3-fed-029" is not up at either SCL or MTV Host "talos-r3-fed-030" is not up at either SCL or MTV Host "talos-r3-fed-034" is not up at either SCL or MTV Host "talos-r3-fed-035" is down Host "talos-r3-fed-036" is not up at either SCL or MTV Host "talos-r3-fed-037" is not up at either SCL or MTV Host "talos-r3-fed-038" is not up at either SCL or MTV Host "talos-r3-fed-039" is down Host "talos-r3-fed64-017" is not up at either SCL or MTV Host "talos-r3-fed64-018" is not up at either SCL or MTV Host "talos-r3-fed64-019" is not up at either SCL or MTV Host "talos-r3-fed64-020" is down Host "talos-r3-fed64-ref" is at MTV Host "talos-r3-fed-ref" is down Host "talos-r3-leopard-001" is down Host "talos-r3-leopard-002" is down Host "talos-r3-leopard-003" is down Host "talos-r3-leopard-005" is down Host "talos-r3-leopard-006" is down Host "talos-r3-leopard-007" is down Host "talos-r3-leopard-008" is down Host "talos-r3-leopard-009" is down Host "talos-r3-leopard-013" is down Host "talos-r3-leopard-014" is not up at either SCL or MTV Host "talos-r3-leopard-015" is down Host "talos-r3-leopard-016" is not up at either SCL or MTV Host "talos-r3-leopard-017" is not up at either SCL or MTV Host "talos-r3-leopard-018" is down Host "talos-r3-leopard-019" is not up at either SCL or MTV Host "talos-r3-leopard-027" is not up at either SCL or MTV Host "talos-r3-leopard-028" is not up at either SCL or MTV Host "talos-r3-leopard-029" is not up at either SCL or MTV Host "talos-r3-leopard-030" is not up at either SCL or MTV Host "talos-r3-leopard-031" is not up at either SCL or MTV Host "talos-r3-leopard-032" is not up at either SCL or MTV Host "talos-r3-leopard-033" is not up at either SCL or MTV Host "talos-r3-leopard-034" is not up at either SCL or MTV Host "talos-r3-leopard-035" is not up at either SCL or MTV Host "talos-r3-leopard-036" is not up at either SCL or MTV Host "talos-r3-leopard-037" is not up at either SCL or MTV Host "talos-r3-leopard-038" is not up at either SCL or MTV Host "talos-r3-leopard-039" is down Host "talos-r3-leopard-040" is not up at either SCL or MTV Host "talos-r3-leopard-ref" is at MTV Host "talos-r3-snow-013" is not up at either SCL or MTV Host "talos-r3-snow-014" is not up at either SCL or MTV Host "talos-r3-snow-015" is not up at either SCL or MTV Host "talos-r3-snow-016" is not up at either SCL or MTV Host "talos-r3-snow-017" is not up at either SCL or MTV Host "talos-r3-snow-018" is not up at either SCL or MTV Host "talos-r3-snow-019" is not up at either SCL or MTV Host "talos-r3-snow-020" is not up at either SCL or MTV Host "talos-r3-snow-ref" is at MTV Host "talos-r3-w7-001" is down Host "talos-r3-w7-002" is down Host "talos-r3-w7-003" is down Host "talos-r3-w7-004" is down Host "talos-r3-w7-006" is down Host "talos-r3-w7-007" is down Host "talos-r3-w7-008" is down Host "talos-r3-w7-009" is down Host "talos-r3-w7-010" is not up at either SCL or MTV Host "talos-r3-w7-012" is not up at either SCL or MTV Host "talos-r3-w7-013" is not up at either SCL or MTV Host "talos-r3-w7-014" is not up at either SCL or MTV Host "talos-r3-w7-015" is down Host "talos-r3-w7-016" is down Host "talos-r3-w7-017" is at SCL Host "talos-r3-w7-018" is down Host "talos-r3-w7-ref" is at MTV Host "talos-r3-xp-001" is at SCL Host "talos-r3-xp-002" is at SCL Host "talos-r3-xp-003" is at SCL Host "talos-r3-xp-005" is at SCL Host "talos-r3-xp-006" is at SCL Host "talos-r3-xp-007" is at SCL Host "talos-r3-xp-008" is at SCL Host "talos-r3-xp-009" is at SCL Host "talos-r3-xp-013" is down Host "talos-r3-xp-014" is at SCL Host "talos-r3-xp-018" is at SCL Host "talos-r3-xp-019" is down Host "talos-r3-xp-020" is down Host "talos-r3-xp-032" is not up at either SCL or MTV Host "talos-r3-xp-033" is not up at either SCL or MTV Host "talos-r3-xp-034" is not up at either SCL or MTV Host "talos-r3-xp-035" is not up at either SCL or MTV Host "talos-r3-xp-036" is not up at either SCL or MTV Host "talos-r3-xp-037" is not up at either SCL or MTV Host "talos-r3-xp-038" is not up at either SCL or MTV Host "talos-r3-xp-039" is not up at either SCL or MTV Host "talos-r3-xp-040" is not up at either SCL or MTV What's weird is that there are some hosts which I can ping using the cname but not using build.*.mozilla.com addresses.
it seems that the weirdness is because i am pinging the zero padded addresses intead of its real non-zero padded mtv1.mozilla.com
Derek is going to bring the following slaves with him to scl fed64-020 leopard-018 xp-013 xp-019 xp-020
jlaz and I are bringing the following slaves and some w764 slaves fed-39 fed64-17 fed64-18 fed64-19 snow-17 snow-19 xp-32 xp-33 xp-34 xp-35 xp-36 xp-37 xp-38 xp-39 xp-40
xp-013 and 018 have busted opsi setups i think
talos-r3-xp-013 - online talos-r3-xp-018 - online talos-r3-xp-019 - online talos-r3-xp-020 - online - old password talos-r3-xp-035 - opsi talos-r3-xp-034 - online talos-r3-xp-032 - online talos-r3-xp-033 - online talos-r3-xp-036 - online talos-r3-xp-037 - online talos-r3-xp-038 - online talos-r3-xp-039 - online talos-r3-xp-040 - online xp-035 seems to have an opsi screen up. When i saw that screen on 013 and 018, they eventually came back onto the master after a little bit. I have double checked and all the values are correct. We also moved all windows 7 64 bit slaves to internap this trip.
If someone where to summarize, what's left? And who can drive moving the rest next week? (thanks to all involved today too!)
for my part (the non-XP systems), status is reassigned: fed64-020 fed64-017 fed64-018 fed64-019 snow-019 leopard-018 fed-039 bad: snow-017 - not in DNS, and when it was in DNS, it didn't accept old or new cltbld passwords (and thus DNS might not have given me the right address) *However*, all of the slaves marked "reassigned" above are not starting the buildslave process because they are not communicating with their puppetmaster. I'm told this can be fixed, but not on a Friday night.
Depends on: 612174
This bug is confused. I am going to track cleaning up fallout from this in bug 612288 and bug 612452 to track the remaining work for the mini move
Status: NEW → RESOLVED
Closed: 15 years ago
Resolution: --- → INCOMPLETE
See Also: → 612288
See Also: 612288
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.