1189476 - decommission foopies, mozpool, panda buildbot servers once pandas are no longer in use

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Kim Moir [:kmoir] ET]

Should be able to relocate some of this hardware to the linux talos test pool

Comment 1

[Kim Moir [:kmoir] ET]

foopies 102-104,39-56 can be decommissioned as described in bug 1193002

Comment 2

[Amy Rich [:arr] [:arich]]

It's going to take some figuring to correlate these to the panda chassis/relays and racks/mobile imaging servers since they weren't decommed according to the physical layout of the systems. I'll do that and update this bug with the things we can actually decomm.

Comment 3

[Amy Rich [:arr] [:arich]]

I'm going to open a dcops bug to decommission things by chassis/panda-relay instead of by specific board, since we aren't going to pull individual boards. Here's what I've come up with that most closely matches what you've specified. 

panda-relay-002.p10.releng.scl3.mozilla.com (22-32)
panda-relay-003.p10.releng.scl3.mozilla.com (34-43)
panda-relay-004.p10.releng.scl3.mozilla.com (46-55)

We can't decomm the rest of p10, so that leaves a rack less than half full.

panda-relay-007.p1.releng.scl3.mozilla.com (82-92, 610)
panda-relay-008.p1.releng.scl3.mozilla.com (93-103, 611)
panda-relay-009.p1.releng.scl3.mozilla.com (104-114, 612)
panda-relay-010.p1.releng.scl3.mozilla.com (115-125)
panda-relay-011.p1.releng.scl3.mozilla.com (127-136, 613-614)
panda-relay-012.p1.releng.scl3.mozilla.com (137-147)
panda-relay-013.p1.releng.scl3.mozilla.com (148-158, 615)
panda-relay-014.p1.releng.scl3.mozilla.com (159-169)

That's all of p1, but we can't decomm mobile-imaging-001 since it's the primary mozpool server that syncs with the DB. That has to go last. So we'll have a rack empty except for this one chassis (set of 4 machines, 3 foopies and the mobile imaging server).

panda-relay-015.p2.releng.scl3.mozilla.com (170-180)
panda-relay-016.p2.releng.scl3.mozilla.com (181-191, 616)
panda-relay-017.p2.releng.scl3.mozilla.com (192-202)
panda-relay-018.p2.releng.scl3.mozilla.com (203-212)
panda-relay-019.p2.releng.scl3.mozilla.com (214-224,617)
panda-relay-020.p2.releng.scl3.mozilla.com (225-235, 618)
panda-relay-021.p2.releng.scl3.mozilla.com (236-246)
panda-relay-022.p2.releng.scl3.mozilla.com (33, 247-255)

That's all of p2, so we can decommission mobile-imaging-002.p2.releng.scl3.mozilla.com and everything in that rack.

panda-relay-023.p3.releng.scl3.mozilla.com (258-268)
panda-relay-024.p3.releng.scl3.mozilla.com (269-279)
panda-relay-025.p3.releng.scl3.mozilla.com (280-290)
panda-relay-026.p3.releng.scl3.mozilla.com (291-301, 620)

Can we decomm panda-0620 so we can decomm all of panda-relay-026, please?

This leaves us with part of p3, part of p10, and all of p4, p5 and p6. p7, p8, and p9 had already been decommed.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

We could just change the primary

Comment 5

[Kim Moir [:kmoir] ET]

I've updated my patch to decomm panda-620 in bug 193002 and disabled it in slavealloc

Comment 6

[Amy Rich [:arr] [:arich]]

Attachment: panda-decomm.diff

Remove decommed pandas and infrastructure from nagios.

Comment 7

[Amy Rich [:arr] [:arich]]

Attachment: panda-decomm.diff

This adds in pandas that have been retasked to replace dead pandas.

Comment 8

[Amy Rich [:arr] [:arich]]

I had one typo and had to rmeove the mozpool relay check for mobile-imaging-001. Original checkin is revision 109696; fixes checked in in revision 109702.

Kim: let me know when I can tell dcops to decomm stuff.

Comment 9

[Kim Moir [:kmoir] ET]

I think the mozpool server change need to land ahead first (bug 1218571) before we decomm stuff

Comment 10

[Amy Rich [:arr] [:arich]]

I removed mobile-imaging-001 in revision 109808.
I'll open up a bug for dcops to do the actual physical decomm now.

Comment 11

[Amy Rich [:arr] [:arich]]

The remaining hosts in p3 and p10 removed from nagios in svn revision 109982.

Comment 12

[Kim Moir [:kmoir] ET]

I think the following foopies can be decommed too 
foopy102,103,53, 54, 55, 56

Comment 13

[Kim Moir [:kmoir] ET]

Please decommission all remaining panda racks, foopies 60-80 and mozpool servers.  No remaining panda jobs are running.

Comment 16

[Kim Moir [:kmoir] ET]

I also removed entries for bm89, and bm100-102 today in puppet etc and disabled them in slavealloc so these machines could be decommissioned