hp-node0[3-4].phx1 power supply failure

RESOLVED FIXED

Status

Infrastructure & Operations
DCOps
RESOLVED FIXED
6 years ago
4 years ago

People

(Reporter: ericz, Assigned: rbryce)

Tracking

Details

(Whiteboard: phx1 | HP Case 4640909246)

(Reporter)

Description

6 years ago
Power supply failed on hp-node04.phx1:

[root@hp-node04.phx1 ~]# ipmitool sel list
  18 | 09/20/2010 | 13:16:01 | Power Unit #0xb9 | Fully Redundant
  30 | 09/20/2010 | 13:17:13 | Power Supply #0xb7 | Failure detected | Asserted
  48 | 09/20/2010 | 13:17:13 | Power Unit #0xb9 | Redundancy Lost
  60 | 09/20/2010 | 13:20:01 | Power Unit #0xb9 | Fully Redundant
  78 | 06/30/2012 | 01:35:24 | Power Supply #0xb8 | Failure detected | Asserted
  90 | 06/30/2012 | 01:35:24 | Power Unit #0xb9 | Redundancy Lost
  a8 | 07/01/2012 | 01:47:01 | Power Supply #0xb8 | Failure detected | Asserted
(Assignee)

Comment 1

6 years ago
Case ID: 	4640909246
Submitted: 	2012-07-05 10:12:05 PDT
Problem description :
	
1 Bad PSU 

ipmitool sel list
  18 | 09/20/2010 | 13:16:01 | Power Unit #0xb9 | Fully Redundant
  30 | 09/20/2010 | 13:17:13 | Power Supply #0xb7 | Failure
detected | Asserted
  48 | 09/20/2010 | 13:17:13 | Power Unit #0xb9 | Redundancy Lost
  60 | 09/20/2010 | 13:20:01 | Power Unit #0xb9 | Fully Redundant
  78 | 06/30/2012 | 01:35:24 | Power Supply #0xb8 | Failure
detected | Asserted
  90 | 06/30/2012 | 01:35:24 | Power Unit #0xb9 | Redundancy Lost
  a8 | 07/01/2012 | 01:47:01 | Power Supply #0xb8 | Failure
detected | Asserted
Whiteboard: HP Case 4640909246
(Assignee)

Updated

6 years ago
Duplicate of this bug: 769982
(Assignee)

Updated

6 years ago
Summary: hp-node04.phx1 power supply failure → hp-node0[3-4].phx1 power supply failure
(Assignee)

Comment 3

6 years ago
HP sent the PSUs to MTV instead of PHX. I am forwarding the packages to PHX1 datacenter.  

These 2 nodes are in one chassis.  Im not exactly sure which PSU to replace.  Once someone is onsite, I can work with whomever to identify the correct PSU to replace.  HP sent 2 PSU, although I think we only need 1.
(Assignee)

Updated

6 years ago
Component: Server Operations → Server Operations: DCOps
QA Contact: phong → dmoore
Whiteboard: HP Case 4640909246 → HP Case 4640909246 || In Transit to PHX
(Assignee)

Comment 4

6 years ago
Fedex Tracking # 5342 6663 1290

Updated

6 years ago
Whiteboard: HP Case 4640909246 || In Transit to PHX → phx1 | HP Case 4640909246
PSUs received at phx1

We can schedule this for 7/24 between 16:00 and 19:00, or 7/25 between 09:00 and 11:00.
Next window will be on 8/1
Was this completed?
Next window is on 8/6. We need an SRE to be available to assist, per Comment 3.
(Assignee)

Updated

6 years ago
Assignee: server-ops → rbryce.bugs
(Assignee)

Comment 9

6 years ago
Sal switched out the bad PSU just now.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.