637907 - HP Array Controllers need firmware update

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Ben Kero [:bkero]]

An hp array nagios check timed out.  When it recovered, it returned with this status:

15:16 <@nagios> surf:RAID is OK: RAID OK:  FIRMWARE UPGRADE REQUIRED: A firmware update is recommended for this controller to prevent rare potential data write errors on a RAID 1 or RAID 1+0 volume in a scenario of concurrent 
                background surface analysis and I/O write operations. Please refer to Customer Advisory c01587778 which can be found at hp.com. Smart Array 6400 in Slot 2 array A logicaldrive 1 (1.9 TB, RAID 1+0, OK) FIRMWARE UPGRADE 
                REQUIRED: A firmware update is
15:16 <@nagios> roller to prevent rare potential data write


It is unclear how long the update has been out, but if there is a potential for data write loss, it's probably a worthwhile upgrade.

Doing this would probably need some downtimes that would likely close the tree.

Comment 1

[Corey Shields [:cshields]]

pulling zandr in, we can schedule this for a post-fx4 downtime window

Comment 2

[Phong Tran [:phong]]

passing this over to RelEng to coordinate a downtime.

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

1) What systems would need to be taken down for this firmware upgrade?

2) how long will this upgrade take?

Comment 4

[Corey Shields [:cshields]]

(In reply to comment #3)
> 1) What systems would need to be taken down for this firmware upgrade?

surf

> 2) how long will this upgrade take?

realistically probably 15 mins from shutdown to back up, but would need an hour window to account for turbulence.

Comment 5

[Amy Rich [:arr] [:arich]]

Okay, Dumitru, we're scheduled for 9:00 eastern (6:00 pacific) to do the HP array firmware upgrade.  I'll be making DNS/DHCP changes at the same time, but I will be reachable on irc if there's an issue.  If you could drop into #build and give us a heads up before you start (and when you're done and things are back onlint), that would be great.

Comment 6

[Dumitru Gherman [:dumitru]]

The two RAID controllers on surf have now the newest firmware.