1094087 - LSI Raid on backup1.private.par1.mozilla.com is WARNING: WARNING: 0:0:RAID-6:16 drives:1.818TB:Optimal Drives:16 online(23 Errors)

Status: RESOLVED FIXED

(Infrastructure & Operations :: MOC: Problems, task)

Description

[MOC Nagios API]

Automated alert report from nagios1.private.scl3.mozilla.com:

Hostname: backup1.private.par1.mozilla.com
Service:  LSI Raid
State:    WARNING
Output:   WARNING: 0:0:RAID-6:16 drives:1.818TB:Optimal Drives:16 online(23 Errors)

Runbook:  http://m.allizom.org/LSI+Raid

Comment 1

[MOC Nagios API]

Automated alert acknowledgement: (ashish)rebuilding drive bug 1094087

Comment 2

[Ashish Vijayaram [:ashish]]

Took drive offline and initiated a rebuild:

> Rebuild Progress on Device at Enclosure 17, Slot 1 Completed 0% in 2 Minutes.

Once rebuild completes, run the following command to put the drive online again:

> # cd /opt/MegaRAID/MegaCli
> # ./MegaCli64 -PDOnline -PhysDrv [17:1] -a0

Comment 3

[Guillaume Canavaggio [:guiom]]

Asish, the server is beeping all the time. Since there is nothing on it I just took it offline. Please let me know when we can work together on  it. Thanks!

Comment 4

[Ryan Watson [:w0ts0n]]

The machine has been downtimed until later in the day (when PAR staff are not in the office) the rebuild was interrupted so needs to be done again.

Comment 5

[Ryan Watson [:w0ts0n]]

Rick can you take a look at this on your shift, the PAR staff should be out and I've asked Guiom to switch it on.

Comment 6

[Adrian J Fernandez [:Aj]]

Hello Guillaume,

Can you turn on the server again?
The server will continue to beep until the raid finishes rebuilding, we could perhaps silence the alarm if needed.

Comment 7

[Adrian J Fernandez [:Aj]]

The server is back online;
$ uptime
 17:51:40 up 56 min,  1 user,  load average: 0.00, 0.01, 0.00

cd /opt/MegaRAID/MegaCli
sudo ./MegaCli64 -AdpSetProp AlarmSilence -a0 # this silences the beeping

# Rebuild status:
$ sudo ./MegaCli64 -PDRbld -ShowProg -PhysDrv [17:1] -a0
                                     
Rebuild Progress on Device at Enclosure 17, Slot 1 Completed 91% in 311 Minutes.

Exit Code: 0x00


Seems the rebuilding might be stuck so will check in again (~1-2hrs) for any progress.
May need to "remove" the drive and restart rebuild (possible that the turning off of the server interrupted the rebuilding and left it in a irrecoverable state).

Comment 8

[MOC Nagios API]

Automated alert acknowledgement: (rbryce)WIP

Comment 9

[MOC Nagios API]

Automated alert recovery:

Hostname: backup1.private.par1.mozilla.com
Service:  LSI Raid
State:    OK
Output:   OK: 0:0:RAID-6:16 drives:1.818TB:Optimal Drives:16 online

Comment 11

[Ashish Vijayaram [:ashish]]

This recurred and a repeat of rebuilding didn't help clear out the errors. I've filed Bug 1096415 for replacing the drive.

Comment 12

[Guillaume Canavaggio [:guiom]]

Ashish, I was hoping you could help with the bleeping, the Paris folks are complaining that they cannot use the community space close-by.

Comment 13

[Adrian J Fernandez [:Aj]]

Alarm was silenced.
[12:22] <      guiom> | Aj awesome the bleeping stopped.

Comment 14

[Adrian J Fernandez [:Aj]]

The following (which was executed), should permanently disable ALL alarms (beeping):
$ sudo ./MegaCli64 -AdpSetProp AlarmDsbl -aALL
                                     
Adapter 0: Set alarm to Disabled success.

Exit Code: 0x00