node18.peach.metrics.scl3.mozilla.com:HP Log is WARNING: WARNING 0017: Internal Storage Enclosure Device Failure (Bay 10, Box 1, Port 1I, Slot 0

RESOLVED FIXED

Status

Infrastructure & Operations
DCOps
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: Marlena, Assigned: van)

Tracking

Details

(Whiteboard: HP case id- 4765919814)

(Reporter)

Description

2 years ago
6:17 AM <@nagios-scl3> Tue 06:17:12 PST [5163] node18.peach.metrics.scl3.mozilla.com:HP Log is WARNING: WARNING 0017: Internal Storage Enclosure Device Failure (Bay 10, Box 1, Port 1I, Slot 0) (http://m.mozilla.org/HP+Log)
(Reporter)

Comment 1

2 years ago
6:23 AM <@nagios-scl3> Tue 06:23:52 PST [5166] node18.peach.metrics.scl3.mozilla.com:HP RAID is WARNING: RAID WARNING - Component Failure:  Smart Array P420i in Slot 0 (Embedded) array A logicaldrive 1 (2.7 TB, RAID 0, OK) array B logicaldrive 2 (2.7 TB, RAID 0, OK) array C logicaldrive 3 (2.7 TB, RAID 0, OK) array D logicaldrive 4 (2.7 TB, RAID 0, OK) array E logicaldrive 5 (2.7 TB, RAID 0, OK) array F logicaldrive 6 (2.7 TB, RAID 0, OK) array G logica


==============


0017 Caution        12:49  01/05/2016 12:49  01/05/2016 0001
LOG: Internal Storage Enclosure Device Failure (Bay 10, Box 1, Port 1I, Slot 0)

[root@node18.peach.metrics.scl3 ~]# hplog -v
(Reporter)

Comment 2

2 years ago
Cleared log and further monitoring 


root@node18.peach.metrics.scl3 ~]# hpasmcli -s "clear iml;"

IML Log successfully cleared.

[root@node18.peach.metrics.scl3 ~]# hplog -v

ID   Severity       Initial Time      Update Time       Count
-------------------------------------------------------------
0000 Information    14:27  01/05/2016 14:27  01/05/2016 0001
LOG: Maintenance note: IML cleared through hpasmcli

[root@node18.peach.metrics.scl3 ~]#
   Physical Drives
      physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SATA, 3 TB, OK)
      physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SATA, 3 TB, OK)
      physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SATA, 3 TB, OK)
      physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SATA, 3 TB, OK)
      physicaldrive 1I:1:5 (port 1I:box 1:bay 5, SATA, 3 TB, OK)
      physicaldrive 1I:1:6 (port 1I:box 1:bay 6, SATA, 3 TB, OK)
      physicaldrive 1I:1:7 (port 1I:box 1:bay 7, SATA, 3 TB, OK)
      physicaldrive 1I:1:8 (port 1I:box 1:bay 8, SATA, 3 TB, OK)
      physicaldrive 1I:1:9 (port 1I:box 1:bay 9, SATA, 3 TB, OK)
      physicaldrive 1I:1:10 (port 1I:box 1:bay 10, SATA, 3 TB, Predictive Failure)
      physicaldrive 1I:1:11 (port 1I:box 1:bay 11, SATA, 3 TB, OK)
      physicaldrive 1I:1:12 (port 1I:box 1:bay 12, SATA, 3 TB, OK)

As usual with peach machines this will need to be pulled from service my Pythian, the disk unmounted, new disk swapped in, formatted, remounted and then put back into service.

Pythian can you drain this host, please?
(Assignee)

Comment 4

2 years ago
opened HP case id - 4765919814 for RMA.
Assignee: server-ops-dcops → vle
QA Contact: jbarnell
Whiteboard: HP case id- 4765919814

Comment 5

2 years ago
CM work has been completed.
Device is ready for replacement, please proceed and let us know when done:

[root@node18.peach.metrics.scl3 ~]# hpacucli controller slot=0 ld 10 show

Smart Array P420i in Slot 0 (Embedded)

   array J

      Logical Drive: 10
         Size: 2.7 TB
         Fault Tolerance: 0
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 65535
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         Caching:  Enabled
         Unique Identifier: 600508B1001CBBABAED9396F39C82A1A
         Disk Name: /dev/sdj          Mount Points: None
         Logical Drive Label: A20D7D615001438025189B50  1E6E
         Drive Type: Data
         LD Acceleration Method: Controller Cache

Regards,

Nicolas Parducci
Pythian SRE team
Assignee: vle → server-ops-dcops
QA Contact: jbarnell
Whiteboard: HP case id- 4765919814
Nicolas - please be careful when updating bugs to not overwrite other people's changes. You wiped out :van's assignee, QA contact and whiteboard changes.
Assignee: server-ops-dcops → vle
QA Contact: jbarnell
Whiteboard: HP case id- 4765919814
(Assignee)

Comment 7

2 years ago
drive came in and has been replaced and mounted. thanks for the mana page ryanc!

[vle@node18.peach.metrics.scl3 dev]$ df -h | sort
cm_processes     32G     0   32G   0% /var/run/cloudera-scm-agent/process
/dev/md0        248M   76M  160M  32% /boot
/dev/md1         30G   16G   13G  55% /
/dev/sda4       2.7T  2.0T  573G  78% /data1
/dev/sdb4       2.7T  2.0T  565G  78% /data2
/dev/sdc1       2.7T  2.0T  631G  76% /data3
/dev/sdd1       2.7T  1.9T  680G  74% /data4
/dev/sde1       2.7T  2.0T  664G  75% /data5
/dev/sdf1       2.7T  2.0T  650G  76% /data6
/dev/sdg1       2.7T  2.0T  640G  76% /data7
/dev/sdh1       2.7T  1.9T  688G  74% /data8
/dev/sdi1       2.7T  2.0T  618G  77% /data9
/dev/sdj1       2.7T   73M  2.6T   1% /data10
/dev/sdk1       2.7T  2.0T  612G  77% /data11
/dev/sdl1       2.7T  2.0T  652G  76% /data12
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            32G     0   32G   0% /dev/shm
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED

Comment 8

2 years ago
Node has been recomissioned into the cluster
You need to log in before you can comment on or make changes to this bug.