1056145 - put excess pandas in storage

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Amy Rich [:arr] [:arich]]

This bug is to track the storage of select pandachassis which are no longer being used. Which chassis we select will depend on where the most broken boards are, but the intended result is that we put three racks worth of chassis in storage and redistribute the working boards as needed to replace broken production boards.

Comment 1

[Amy Rich [:arr] [:arich]]

We'd like to put the panda chassis from the following racks in storage in scl3:

https://inventory.mozilla.org/en-US/systems/racks/?rack=372
https://inventory.mozilla.org/en-US/systems/racks/?rack=373
https://inventory.mozilla.org/en-US/systems/racks/?rack=374

We'll be retasking the iX machines and using some of the boards to back fill for broken pands in other chassis in bug 1056143.

The steps that need to happen (ignoring those pandaboards that are being used for backfill) are:

1) comment out all pandas in nagios
2) remove the location and IP information from the pandas in inventory(?)
3) physically pull the chassis, scavenging from those specified in bug 1056143 to backfill, and put them in storage.

What else am I missing?

Comment 2

[Amy Rich [:arr] [:arich]]

Attachment: remove-p7-9.patch

I also needed to comment out the foopies, mobile-imaging servers, and panda-relays as well. The foopies and mobile-imaging servers will be repurposed and should not be moved into storage. I'll open a separate bug to handle those.

Comment 3

[Amy Rich [:arr] [:arich]]

2a) remove all of the k/v store information for imaging server for each board.

Comment 4

[Amy Rich [:arr] [:arich]]

2a complete using raw mysql:

use inventory_mozilla_org
delete from key_value where value = 'mobile-imaging-007.p7.releng.scl3.mozilla.com';
delete from key_value where value = 'mobile-imaging-008.p8.releng.scl3.mozilla.com';
delete from key_value where value = 'mobile-imaging-009.p9.releng.scl3.mozilla.com';

I need to talk to uberj to find out the best way to zero out the IP info like we would do with a decommission (without actually decommissioning them, since we'd like to tag them as spares instead).

Ah, yes, which brings up:

4) mark the remaining good pandas and panda-relays as spares

Comment 5

[Amy Rich [:arr] [:arich]]

All remaining good pandas, panda-relays, and panda-chassis marked as spare.

Comment 6

[Amy Rich [:arr] [:arich]]

uberj had suggested using invtool decommission with the --decommission-system-status set to spare, but that also clears:

oob_ip, switch_ports_ and oob_switch_port (okay for this purpose)
operating_system and allocation (not okay, we want to keep these)

Comment 7

[Jacques Uber [:uberj]]

I've gone through and moved and cleared the following hosts:

panda-relay-062.p7.releng.scl3.mozilla.com
panda-0688.p7.releng.scl3.mozilla.com
panda-0689.p7.releng.scl3.mozilla.com
panda-0690.p7.releng.scl3.mozilla.com
panda-0691.p7.releng.scl3.mozilla.com
panda-0692.p7.releng.scl3.mozilla.com
panda-0693.p7.releng.scl3.mozilla.com
panda-0694.p7.releng.scl3.mozilla.com
panda-0695.p7.releng.scl3.mozilla.com
panda-0696.p7.releng.scl3.mozilla.com
panda-0697.p7.releng.scl3.mozilla.com
panda-relay-061.p7.releng.scl3.mozilla.com
panda-0677.p7.releng.scl3.mozilla.com
panda-0678.p7.releng.scl3.mozilla.com
panda-0679.p7.releng.scl3.mozilla.com
panda-0680.p7.releng.scl3.mozilla.com
panda-0682.p7.releng.scl3.mozilla.com
panda-0683.p7.releng.scl3.mozilla.com
panda-0684.p7.releng.scl3.mozilla.com
panda-0685.p7.releng.scl3.mozilla.com
panda-0686.p7.releng.scl3.mozilla.com
panda-0687.p7.releng.scl3.mozilla.com
panda-relay-060.p7.releng.scl3.mozilla.com
panda-0666.p7.releng.scl3.mozilla.com
panda-0667.p7.releng.scl3.mozilla.com
panda-0668.p7.releng.scl3.mozilla.com
panda-0669.p7.releng.scl3.mozilla.com
panda-0670.p7.releng.scl3.mozilla.com
panda-0671.p7.releng.scl3.mozilla.com
panda-0672.p7.releng.scl3.mozilla.com
panda-0673.p7.releng.scl3.mozilla.com
panda-0675.p7.releng.scl3.mozilla.com
panda-0676.p7.releng.scl3.mozilla.com
panda-relay-059.p7.releng.scl3.mozilla.com
panda-0654.p7.releng.scl3.mozilla.com
panda-0655.p7.releng.scl3.mozilla.com
panda-0656.p7.releng.scl3.mozilla.com
panda-0657.p7.releng.scl3.mozilla.com
panda-0658.p7.releng.scl3.mozilla.com
panda-0659.p7.releng.scl3.mozilla.com
panda-0660.p7.releng.scl3.mozilla.com
panda-0661.p7.releng.scl3.mozilla.com
panda-0663.p7.releng.scl3.mozilla.com
panda-relay-058.p7.releng.scl3.mozilla.com
panda-0644.p7.releng.scl3.mozilla.com
panda-0645.p7.releng.scl3.mozilla.com
panda-0646.p7.releng.scl3.mozilla.com
panda-0648.p7.releng.scl3.mozilla.com
panda-0649.p7.releng.scl3.mozilla.com
panda-0650.p7.releng.scl3.mozilla.com
panda-0651.p7.releng.scl3.mozilla.com
panda-0652.p7.releng.scl3.mozilla.com
panda-0653.p7.releng.scl3.mozilla.com
panda-relay-057.p7.releng.scl3.mozilla.com
panda-0636.p7.releng.scl3.mozilla.com
panda-0637.p7.releng.scl3.mozilla.com
panda-0639.p7.releng.scl3.mozilla.com
panda-0640.p7.releng.scl3.mozilla.com
panda-0641.p7.releng.scl3.mozilla.com
panda-0642.p7.releng.scl3.mozilla.com
panda-relay-056.p7.releng.scl3.mozilla.com
panda-relay-055.p7.releng.scl3.mozilla.com
panda-relay-070.p8.releng.scl3.mozilla.com
panda-0776.p8.releng.scl3.mozilla.com
panda-0777.p8.releng.scl3.mozilla.com
panda-0779.p8.releng.scl3.mozilla.com
panda-0780.p8.releng.scl3.mozilla.com
panda-0782.p8.releng.scl3.mozilla.com
panda-0783.p8.releng.scl3.mozilla.com
panda-0784.p8.releng.scl3.mozilla.com
panda-0785.p8.releng.scl3.mozilla.com
panda-0885.p8.releng.scl3.mozilla.com
panda-relay-069.p8.releng.scl3.mozilla.com
panda-0765.p8.releng.scl3.mozilla.com
panda-0766.p8.releng.scl3.mozilla.com
panda-0767.p8.releng.scl3.mozilla.com
panda-0768.p8.releng.scl3.mozilla.com
panda-0769.p8.releng.scl3.mozilla.com
panda-0770.p8.releng.scl3.mozilla.com
panda-0771.p8.releng.scl3.mozilla.com
panda-0772.p8.releng.scl3.mozilla.com
panda-0773.p8.releng.scl3.mozilla.com
panda-0774.p8.releng.scl3.mozilla.com
panda-0775.p8.releng.scl3.mozilla.com
panda-relay-068.p8.releng.scl3.mozilla.com
panda-0754.p8.releng.scl3.mozilla.com
panda-0755.p8.releng.scl3.mozilla.com
panda-0756.p8.releng.scl3.mozilla.com
panda-0757.p8.releng.scl3.mozilla.com
panda-0758.p8.releng.scl3.mozilla.com
panda-0759.p8.releng.scl3.mozilla.com
panda-0760.p8.releng.scl3.mozilla.com
panda-0761.p8.releng.scl3.mozilla.com
panda-0762.p8.releng.scl3.mozilla.com
panda-0763.p8.releng.scl3.mozilla.com
panda-0764.p8.releng.scl3.mozilla.com
panda-relay-067.p8.releng.scl3.mozilla.com
panda-0743.p8.releng.scl3.mozilla.com
panda-0744.p8.releng.scl3.mozilla.com
panda-0745.p8.releng.scl3.mozilla.com
panda-0746.p8.releng.scl3.mozilla.com
panda-0748.p8.releng.scl3.mozilla.com
panda-0750.p8.releng.scl3.mozilla.com
panda-0751.p8.releng.scl3.mozilla.com
panda-0752.p8.releng.scl3.mozilla.com
panda-0753.p8.releng.scl3.mozilla.com
panda-relay-066.p8.releng.scl3.mozilla.com
panda-0732.p8.releng.scl3.mozilla.com
panda-0733.p8.releng.scl3.mozilla.com
panda-0735.p8.releng.scl3.mozilla.com
panda-0737.p8.releng.scl3.mozilla.com
panda-0738.p8.releng.scl3.mozilla.com
panda-0739.p8.releng.scl3.mozilla.com
panda-0740.p8.releng.scl3.mozilla.com
panda-0741.p8.releng.scl3.mozilla.com
panda-0742.p8.releng.scl3.mozilla.com
panda-relay-065.p8.releng.scl3.mozilla.com
panda-0721.p8.releng.scl3.mozilla.com
panda-0722.p8.releng.scl3.mozilla.com
panda-0723.p8.releng.scl3.mozilla.com
panda-0724.p8.releng.scl3.mozilla.com
panda-0725.p8.releng.scl3.mozilla.com
panda-0727.p8.releng.scl3.mozilla.com
panda-0728.p8.releng.scl3.mozilla.com
panda-0729.p8.releng.scl3.mozilla.com
panda-relay-064.p8.releng.scl3.mozilla.com
panda-0709.p8.releng.scl3.mozilla.com
panda-0710.p8.releng.scl3.mozilla.com
panda-0711.p8.releng.scl3.mozilla.com
panda-0712.p8.releng.scl3.mozilla.com
panda-0713.p8.releng.scl3.mozilla.com
panda-0714.p8.releng.scl3.mozilla.com
panda-0715.p8.releng.scl3.mozilla.com
panda-0716.p8.releng.scl3.mozilla.com
panda-0717.p8.releng.scl3.mozilla.com
panda-0718.p8.releng.scl3.mozilla.com
panda-0719.p8.releng.scl3.mozilla.com
panda-relay-063.p8.releng.scl3.mozilla.com
panda-0698.p8.releng.scl3.mozilla.com
panda-0699.p8.releng.scl3.mozilla.com
panda-0700.p8.releng.scl3.mozilla.com
panda-0701.p8.releng.scl3.mozilla.com
panda-0702.p8.releng.scl3.mozilla.com
panda-0703.p8.releng.scl3.mozilla.com
panda-0704.p8.releng.scl3.mozilla.com
panda-0705.p8.releng.scl3.mozilla.com
panda-0706.p8.releng.scl3.mozilla.com
panda-0707.p8.releng.scl3.mozilla.com
panda-0708.p8.releng.scl3.mozilla.com
panda-relay-078.p9.releng.scl3.mozilla.com
panda-0864.p9.releng.scl3.mozilla.com
panda-0865.p9.releng.scl3.mozilla.com
panda-0866.p9.releng.scl3.mozilla.com
panda-0867.p9.releng.scl3.mozilla.com
panda-0868.p9.releng.scl3.mozilla.com
panda-0869.p9.releng.scl3.mozilla.com
panda-0870.p9.releng.scl3.mozilla.com
panda-0871.p9.releng.scl3.mozilla.com
panda-0872.p9.releng.scl3.mozilla.com
panda-0873.p9.releng.scl3.mozilla.com
panda-0886.p9.releng.scl3.mozilla.com
panda-relay-077.p9.releng.scl3.mozilla.com
panda-0853.p9.releng.scl3.mozilla.com
panda-0854.p9.releng.scl3.mozilla.com
panda-0855.p9.releng.scl3.mozilla.com
panda-0856.p9.releng.scl3.mozilla.com
panda-0857.p9.releng.scl3.mozilla.com
panda-0858.p9.releng.scl3.mozilla.com
panda-0859.p9.releng.scl3.mozilla.com
panda-0860.p9.releng.scl3.mozilla.com
panda-0861.p9.releng.scl3.mozilla.com
panda-0862.p9.releng.scl3.mozilla.com
panda-0863.p9.releng.scl3.mozilla.com
panda-relay-076.p9.releng.scl3.mozilla.com
panda-0842.p9.releng.scl3.mozilla.com
panda-0843.p9.releng.scl3.mozilla.com
panda-0844.p9.releng.scl3.mozilla.com
panda-0845.p9.releng.scl3.mozilla.com
panda-0846.p9.releng.scl3.mozilla.com
panda-0847.p9.releng.scl3.mozilla.com
panda-0849.p9.releng.scl3.mozilla.com
panda-0850.p9.releng.scl3.mozilla.com
panda-0851.p9.releng.scl3.mozilla.com
panda-0852.p9.releng.scl3.mozilla.com
panda-relay-075.p9.releng.scl3.mozilla.com
panda-0831.p9.releng.scl3.mozilla.com
panda-0833.p9.releng.scl3.mozilla.com
panda-0836.p9.releng.scl3.mozilla.com
panda-0837.p9.releng.scl3.mozilla.com
panda-0838.p9.releng.scl3.mozilla.com
panda-0839.p9.releng.scl3.mozilla.com
panda-0840.p9.releng.scl3.mozilla.com
panda-0841.p9.releng.scl3.mozilla.com
panda-relay-074.p9.releng.scl3.mozilla.com
panda-0820.p9.releng.scl3.mozilla.com
panda-0821.p9.releng.scl3.mozilla.com
panda-0822.p9.releng.scl3.mozilla.com
panda-0823.p9.releng.scl3.mozilla.com
panda-0824.p9.releng.scl3.mozilla.com
panda-0825.p9.releng.scl3.mozilla.com
panda-0826.p9.releng.scl3.mozilla.com
panda-0827.p9.releng.scl3.mozilla.com
panda-0828.p9.releng.scl3.mozilla.com
panda-0829.p9.releng.scl3.mozilla.com
panda-0830.p9.releng.scl3.mozilla.com
panda-relay-073.p9.releng.scl3.mozilla.com
panda-0809.p9.releng.scl3.mozilla.com
panda-0810.p9.releng.scl3.mozilla.com
panda-0811.p9.releng.scl3.mozilla.com
panda-0812.p9.releng.scl3.mozilla.com
panda-0813.p9.releng.scl3.mozilla.com
panda-0814.p9.releng.scl3.mozilla.com
panda-0815.p9.releng.scl3.mozilla.com
panda-0816.p9.releng.scl3.mozilla.com
panda-0817.p9.releng.scl3.mozilla.com
panda-0818.p9.releng.scl3.mozilla.com
panda-relay-072.p9.releng.scl3.mozilla.com
panda-0798.p9.releng.scl3.mozilla.com
panda-0799.p9.releng.scl3.mozilla.com
panda-0800.p9.releng.scl3.mozilla.com
panda-0801.p9.releng.scl3.mozilla.com
panda-0802.p9.releng.scl3.mozilla.com
panda-0804.p9.releng.scl3.mozilla.com
panda-0805.p9.releng.scl3.mozilla.com
panda-0806.p9.releng.scl3.mozilla.com
panda-0808.p9.releng.scl3.mozilla.com
panda-relay-071.p9.releng.scl3.mozilla.com
panda-0786.p9.releng.scl3.mozilla.com
panda-0787.p9.releng.scl3.mozilla.com
panda-0788.p9.releng.scl3.mozilla.com
panda-0789.p9.releng.scl3.mozilla.com
panda-0790.p9.releng.scl3.mozilla.com
panda-0791.p9.releng.scl3.mozilla.com
panda-0792.p9.releng.scl3.mozilla.com
panda-0794.p9.releng.scl3.mozilla.com
panda-0795.p9.releng.scl3.mozilla.com
panda-0796.p9.releng.scl3.mozilla.com