829113 - replace open power supplies in panda chassis

Status: RESOLVED FIXED

(Infrastructure & Operations :: DCOps, task)

Description

[Amy Rich [:arr] [:arich]]

Because of the urgency to get the panda chassis up and running, some were built with open power supplies.  This is dangerous for anyone working on the chassis and the open PSUs should be replaced with the closed PSUs now that we have them available from the manufacturer. 

The first step is to have DCOps identify which chassis have the open power supplies in them.

Comment 1

[Amy Rich [:arr] [:arich]]

Per email from Sal, DCops has received the 29 closed power supplies from DJGrey.

Comment 2

[Van Le [:van]]

The following 29 panda-chassis needs to have it's power supply replaced.

panda-chassis:
[18,19,20,21,22,23,24,26,28,30,32,33,34,35,36,37,38,42,43,44,48,49,50,51,53,54,56,57,58]

Comment 3

[Jake Watkins [:dividehex]]

Are these new Power Supplies in SCL1 already or do they need to be delivered there?

Comment 4

[Salvador Espinoza [:sal]]

power supplies are in the new cage, rack 201-1 I believe

Comment 5

[Jake Watkins [:dividehex]]

I did find them here at SCL1.  And FTR, I will be the person doing the actual swap and wiring.  I would like DCOPS to assist me in getting the chassis opened and closed. (thanks in advance)  I'll schedule a date with you guys (dcops) offline.

Comment 6

[Jake Watkins [:dividehex]]

This is a proposed take down schedule for replacing the open power supplies in the panda chassis listed below.  The work will require each chassis (pandas and relayboards) to be turned off while the powersupply is replaced.  Work will be done over a period of 3 days and divided in to 3 batches so downtime is less impactful to queue times.

Proposed batches
BATCH 1 - Tues - 02/19
chassis - 18    201-2 14    0203-0213            foopy43-foopy52
chassis - 19    201-2 18    0214-0224            foopy43-foopy52
chassis - 20    201-2 22    0225-0235            foopy43-foopy52
chassis - 21    201-2 27    0236-0246            foopy43-foopy52
chassis - 22    201-2 31    0033,0247-0257   foopy43-foopy52 (plus unknown foopy)
chassis - 23    201-3 1      0258-0268            foopy53-foopy59
chassis - 24    201-3 5      0269-0279            foopy53-foopy59
chassis - 48    201-6 5      0533-0543    Android    foopy 74-75
chassis - 49    201-6 10    0544-0554    Android    foopy75 -76
chassis - 50    201-6 14    0555-0565    Android    foopy76-77

BATCH 2 - Wed - 02/20
chassis - 26    201-3 14    0291-0301            foopy53-foopy59
chassis - 28    201-3 22    0313-0323            foopy53-foopy59
chassis - 30    201-3 31    0335-0345            foopy53-foopy59
chassis - 32    201-4 5      0357-0367            foopy60-foopy66
chassis - 33    201-4 10    0368-0378            foopy60-foopy66
chassis - 34    201-4 14    0379-0389            foopy60-foopy66
chassis - 35    201-4 18    0390-0400            foopy60-foopy66
chassis - 51    201-6 18    0057,0069,0566-0569,0571-0573,0575,0576    Android  foopy77-78
chassis - 53    201-6 27    0588-0598    Android   foopy 78-79
chassis - 54    201-6 31    0599-0609    Android   foopy 79-80

BATCH 3 - Thurs - 2/21
chassis - 36    201-4 22    0401-0411            foopy60-foopy66
chassis - 37    201-4 27    0412-0422            foopy60-foopy66
chassis - 38    201-4 31    0423-0433            foopy60-foopy66
chassis - 42    201-5 14    0045,0467,0469-0477    foopy69-foopy72 (plus unknown foopy)
chassis - 43    201-5 18    0478-0488            foopy69-foopy72
chassis - 44    201-5 22    0489-0499            foopy69-foopy72
chassis - 56    201-7 5       0621-0631    Android  foopy 80-81
chassis - 57    201-7 10    0632-0642    Android   foopy 81-82
chassis - 58    201-7 14    0643-0653    Android   foopy 82-83

Comment 7

[Jake Watkins [:dividehex]]

This has been coordinated with :kmoir, :armenzg and :van.

B2G pandas will have *NO* impact since none of chassis being used by b2g are effected.  Android pandas in production will be shutdown but queue times shouldn't be impacted significantly (per :kmoir).  :kmoir will disable the affected pandas on the morning of each batch so chassis power downs can take place at 9am PST.

:van will be joining me on these dates to assist. (thanks, Van!)

Comment 8

[Jake Watkins [:dividehex]]

Batch 1 has been completed

Comment 9

[Jake Watkins [:dividehex]]

Batch 2 has been completed

Comment 10

[Jake Watkins [:dividehex]]

Batch 3 has been completed

Comment 11

[Jake Watkins [:dividehex]]

All power supplies have been swapped at this point. \o/