Closed Bug 860028 Opened 11 years ago Closed 11 years ago

replace 5v supply wire and adj power supply output in panda chassis in scl1

Categories

(Infrastructure & Operations :: RelOps: General, task)

Product:
Infrastructure & Operations
Component:
RelOps: General
Platform:
ARM
Android
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dividehex, Assigned: dividehex)

References

Details

(Whiteboard: [2013Q2]) 

(Assuming tests running in rack 201-4 (sc11) continue to show positive results..)

The current panda chassis in scl1 are suffering from random reboots while in the middle of running tests.  All indications point to this being caused by an UVLO (under voltage lock out) IC being triggered when the voltage drops during a surge in load on the panda boards.  Replace the supply wire( from 18awg to 12awg) between the power supply and the fuse bus will lower the resistance.  Adjusting the power supply output to 5.5 volts (no load) will compensate for the rest.
Just to tack on to the work load here and as a reminder to myself, I'll need to remove the test power supplies in chassis 27,28,29.
Whiteboard: [2013Q2]
Batch 1 is completed

Rack 201-10:
    Chassis 63 - panda(0698-0708)
    Chassis 64 - panda(0709-0719)
    Chassis 65 - panda(0720-0730)
    Chassis 66 - panda(0731-0741)
    Chassis 67 - panda(0742-0752)
    Chassis 68 - panda(0753-0763)
    Chassis 69 - panda(0764-0774)
    Chassis 70 - panda(0775-0780, 0885, 0782-0785)
Rack 201-11:
    Chassis 71 - panda(0786-0792, 0886, 0794-0796)
    Chassis 72 - panda(0797-0807)
    Chassis 73 - panda(0808-0818)
    Chassis 74 - panda(0819-0829)
    Chassis 75 - panda(0830-0840)
    Chassis 76 - panda(0841-0851)
    Chassis 77 - panda(0852-0862)
    Chassis 78 - panda(0863-0873)
Todays batch will be:
Rack 201-1:
    Chassis 7 - panda(0082-0092)
    Chassis 8 - panda(0093-0103)
    Chassis 9 - panda(0104-0114)
    Chassis 10 - panda(0115-0125)
    Chassis 11 - panda(0126-0136)
    Chassis 12 - panda(0137-0147)
    Chassis 13 - panda(0148-0158)
    Chassis 14 - panda(0159-0169)
Rack 201-2:
    Chassis 15 - panda(0170-0180)
    Chassis 16 - panda(0181-0191)
    Chassis 17 - panda(0192-0202)
    Chassis 18 - panda(0203-0213)
    Chassis 19 - panda(0214-0224)
    Chassis 20 - panda(0225-0235)
    Chassis 21 - panda(0236-0346)
    Chassis 22 - panda(0247-0249,0022,0251-0257)

These have all been locked_out in mozpool

Relays have been downtimed:
panda-relay-007.p1.releng.scl1.mozilla.com	04-17-2013 08:18:58	jwatkins@mozilla.com	bug 860028	04-17-2013 08:16:34	04-17-2013 18:16:34	Flexible	0d 1h 0m 0s	6479
panda-relay-008.p1.releng.scl1.mozilla.com	04-17-2013 08:19:53	jwatkins@mozilla.com	bug 860028	04-17-2013 08:19:35	04-17-2013 18:19:35	Flexible	0d 1h 0m 0s	6480
panda-relay-009.p1.releng.scl1.mozilla.com	04-17-2013 08:20:25	jwatkins@mozilla.com	bug 860028	04-17-2013 08:20:10	04-17-2013 18:20:10	Flexible	0d 1h 0m 0s	6481
panda-relay-010.p1.releng.scl1.mozilla.com	04-17-2013 08:21:31	jwatkins@mozilla.com	bug 860028	04-17-2013 08:21:18	04-17-2013 18:21:18	Flexible	0d 1h 0m 0s	6482
panda-relay-011.p1.releng.scl1.mozilla.com	04-17-2013 08:22:37	jwatkins@mozilla.com	bug 860028	04-17-2013 08:22:13	04-17-2013 18:22:13	Flexible	0d 1h 0m 0s	6483
panda-relay-012.p1.releng.scl1.mozilla.com	04-17-2013 08:23:11	jwatkins@mozilla.com	bug 860028	04-17-2013 08:22:59	04-17-2013 18:22:59	Flexible	0d 1h 0m 0s	6484
panda-relay-013.p1.releng.scl1.mozilla.com	04-17-2013 08:23:41	jwatkins@mozilla.com	bug 860028	04-17-2013 08:23:28	04-17-2013 18:23:28	Flexible	0d 1h 0m 0s	6485
panda-relay-014.p1.releng.scl1.mozilla.com	04-17-2013 08:24:03	jwatkins@mozilla.com	bug 860028	04-17-2013 08:23:52	04-17-2013 18:23:52	Flexible	0d 1h 0m 0s	6486
panda-relay-015.p2.releng.scl1.mozilla.com	04-17-2013 08:24:32	jwatkins@mozilla.com	bug 860028	04-17-2013 08:24:16	04-17-2013 18:24:16	Flexible	0d 1h 0m 0s	6487
panda-relay-016.p2.releng.scl1.mozilla.com	04-17-2013 08:25:05	jwatkins@mozilla.com	bug 860028	04-17-2013 08:24:49	04-17-2013 18:24:49	Flexible	0d 1h 0m 0s	6488
panda-relay-017.p2.releng.scl1.mozilla.com	04-17-2013 08:25:35	jwatkins@mozilla.com	bug 860028	04-17-2013 08:25:21	04-17-2013 18:25:21	Flexible	0d 1h 0m 0s	6489
Batch 2 is complete:
Rack 201-1:
    Chassis 7 - panda(0082-0092)
    Chassis 8 - panda(0093-0103)
    Chassis 9 - panda(0104-0114)
    Chassis 10 - panda(0115-0125)
    Chassis 11 - panda(0126-0136)
    Chassis 12 - panda(0137-0147)
    Chassis 13 - panda(0148-0158)
    Chassis 14 - panda(0159-0169)
Rack 201-2:
    Chassis 15 - panda(0170-0180)
    Chassis 16 - panda(0181-0191)
    Chassis 17 - panda(0192-0202)
    Chassis 18 - panda(0203-0213)
    Chassis 19 - panda(0214-0224)
    Chassis 20 - panda(0225-0235)
    Chassis 21 - panda(0236-0246)
    Chassis 22 - panda(0247-0249,0022,0251-0257)
Batch 3 is complete:
Rack 201-6: 
    Chassis 47 - panda(0522-0532)
    Chassis 48 - panda(0533-0543)
    Chassis 49 - panda(0544-0554)
    Chassis 50 - panda(0555-0565)
    Chassis 51 - panda(0566-0569,0057,0571-0573,0069,0575,0576)
    Chassis 52 - panda(0577-0587)
    Chassis 53 - panda(0588-0598)
    Chassis 54 - panda(0599-0609)
Rack 201-7: 
    Chassis 55 - panda(0610-0620)
    Chassis 56 - panda(0621-0631)
    Chassis 57 - panda(0632-0642)
    Chassis 58 - panda(0643-0653)
    Chassis 59 - panda(0654-0661,0081,0663,0664)
    Chassis 60 - panda(0665-0675)
    Chassis 61 - panda(0676-0686)
    Chassis 62 - panda(0687-0697)
Batch 4 is complete:
Rack 201-3:
    Chassis 23 - panda(0258-0268)
    Chassis 24 - panda(0269-0279)
    Chassis 25 - panda(0280-0290)
    Chassis 26 - panda(0291-0301)
    Chassis 27 - panda(0302-0312) - remove test PSU
    Chassis 28 - panda(0313-0323) - remove test PSU
    Chassis 29 - panda(0324-0334) - remove test PSU
    Chassis 30 - panda(0335-0345)
Rack 201-5:
    Chassis 39 - panda(0434-0444)
    Chassis 40 - panda(0445-0455)
    Chassis 41 - panda(0456-0466)
    Chassis 42 - panda(0467-0477)
    Chassis 43 - panda(0478-0488)
    Chassis 44 - panda(0489-0499)
    Chassis 45 - panda(0500-0510)
    Chassis 46 - panda(0511-0521)
Rack 201-12
    Chassis 79 - panda(0874-0884
    Chassis 2 - panda(0022-0032)
    Chassis 3 - panda(0034-0044)
    Chassis 4 - panda(0046-0056)
    Chassis 5 - panda(0058-0068)
    Chassis 6 - panda(0070-0080)

And we didn't forget the prototype chassis!

This completes the wire upgrades and power supply adjustments
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Component: Server Operations: RelEng → RelOps
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.