1236924 - node18.peach.metrics.scl3.mozilla.com:HP Log is WARNING: WARNING 0017: Internal Storage Enclosure Device Failure (Bay 10, Box 1, Port 1I, Slot 0

Status: RESOLVED FIXED

(Infrastructure & Operations :: DCOps, task)

Description

[Marlena]

6:17 AM <@nagios-scl3> Tue 06:17:12 PST [5163] node18.peach.metrics.scl3.mozilla.com:HP Log is WARNING: WARNING 0017: Internal Storage Enclosure Device Failure (Bay 10, Box 1, Port 1I, Slot 0) (http://m.mozilla.org/HP+Log)

Comment 1

[Marlena]

6:23 AM <@nagios-scl3> Tue 06:23:52 PST [5166] node18.peach.metrics.scl3.mozilla.com:HP RAID is WARNING: RAID WARNING - Component Failure:  Smart Array P420i in Slot 0 (Embedded) array A logicaldrive 1 (2.7 TB, RAID 0, OK) array B logicaldrive 2 (2.7 TB, RAID 0, OK) array C logicaldrive 3 (2.7 TB, RAID 0, OK) array D logicaldrive 4 (2.7 TB, RAID 0, OK) array E logicaldrive 5 (2.7 TB, RAID 0, OK) array F logicaldrive 6 (2.7 TB, RAID 0, OK) array G logica


==============


0017 Caution        12:49  01/05/2016 12:49  01/05/2016 0001
LOG: Internal Storage Enclosure Device Failure (Bay 10, Box 1, Port 1I, Slot 0)

[[e-mail]3 ~]# hplog -v

Comment 2

[Marlena]

Cleared log and further monitoring 


[e-mail]3 ~]# hpasmcli -s "clear iml;"

IML Log successfully cleared.

[[e-mail]3 ~]# hplog -v

ID   Severity       Initial Time      Update Time       Count
-------------------------------------------------------------
0000 Information    14:27  01/05/2016 14:27  01/05/2016 0001
LOG: Maintenance note: IML cleared through hpasmcli

[[e-mail]3 ~]#

Comment 3

[Peter Radcliffe [:pir]]

   Physical Drives
      physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SATA, 3 TB, OK)
      physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SATA, 3 TB, OK)
      physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SATA, 3 TB, OK)
      physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SATA, 3 TB, OK)
      physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SATA, 3 TB, OK)
      physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SATA, 3 TB, OK)
      physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SATA, 3 TB, OK)
      physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SATA, 3 TB, OK)
      physicaldrive 1I:1:9 (port 1I:box 1:bay 9, SATA, 3 TB, OK)
      physicaldrive 1I:1:10 (port 1I:box 1:bay 10, SATA, 3 TB, Predictive Failure)
      physicaldrive 1I:1:11 (port 1I:box 1:bay 11, SATA, 3 TB, OK)
      physicaldrive 1I:1:12 (port 1I:box 1:bay 12, SATA, 3 TB, OK)

As usual with peach machines this will need to be pulled from service my Pythian, the disk unmounted, new disk swapped in, formatted, remounted and then put back into service.

Pythian can you drain this host, please?

Comment 4

[Van Le [:van]]

opened HP case id - 4765919814 for RMA.

Comment 5

[u550631]

CM work has been completed.
Device is ready for replacement, please proceed and let us know when done:

[[e-mail]3 ~]# hpacucli controller slot=0 ld 10 show

Smart Array P420i in Slot 0 (Embedded)

   array J

      Logical Drive: 10
         Size: 2.7 TB
         Fault Tolerance: 0
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 65535
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         Caching:  Enabled
         Unique Identifier: 600508B1001CBBABAED9396F39C82A1A
         Disk Name: /dev/sdj          Mount Points: None
         Logical Drive Label: A20D7D615001438025189B50  1E6E
         Drive Type: Data
         LD Acceleration Method: Controller Cache

Regards,

Nicolas Parducci
Pythian SRE team

Comment 6

[Peter Radcliffe [:pir]]

Nicolas - please be careful when updating bugs to not overwrite other people's changes. You wiped out :van's assignee, QA contact and whiteboard changes.

Comment 7

[Van Le [:van]]

drive came in and has been replaced and mounted. thanks for the mana page ryanc!

[[e-mail]3 dev]$ df -h | sort
cm_processes     32G     0   32G   0% /var/run/cloudera-scm-agent/process
/dev/md0        248M   76M  160M  32% /boot
/dev/md1         30G   16G   13G  55% /
/dev/sda4       2.7T  2.0T  573G  78% /data1
/dev/sdb4       2.7T  2.0T  565G  78% /data2
/dev/sdc1       2.7T  2.0T  631G  76% /data3
/dev/sdd1       2.7T  1.9T  680G  74% /data4
/dev/sde1       2.7T  2.0T  664G  75% /data5
/dev/sdf1       2.7T  2.0T  650G  76% /data6
/dev/sdg1       2.7T  2.0T  640G  76% /data7
/dev/sdh1       2.7T  1.9T  688G  74% /data8
/dev/sdi1       2.7T  2.0T  618G  77% /data9
/dev/sdj1       2.7T   73M  2.6T   1% /data10
/dev/sdk1       2.7T  2.0T  612G  77% /data11
/dev/sdl1       2.7T  2.0T  652G  76% /data12
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            32G     0   32G   0% /dev/shm

Comment 8

[u550631]

Node has been recomissioned into the cluster