1240526 - cache module failed on windows machine tableau3.metrics.scl3.mozilla.com

Status: RESOLVED FIXED

(Infrastructure & Operations :: DCOps, task)

Description

[Marlena]

7:19 AM <@nagios-scl3> Mon 07:19:39 PST [5057] boa-d1.r302-2.console.scl3.mozilla.com:HP Blade Chassis is WARNING: Blade 11 (TABLEAU3-SCL3, ProLiant BL460c Gen9) status is Degraded (http://m.mozilla.org/HP+Blade+Chassis)

Comment 1

[Marlena]

Cannot access via ssh with id or root.  More details will be added once accessible

MacBook-Pro-38:secrets marlenalankford-28535$ ssh [e-mail]
ssh: connect to host boa-d1.r302-2.console.scl3.mozilla.com port 22: Operation timed out

MacBook-Pro-38:secrets marlenalankford-28535$ ssh [e-mail]
ssh: connect to host boa-d1.r302-2.console.scl3.mozilla.com port 22: Operation timed out
MacBook-Pro-38:secrets marlenalankford-28535$

Comment 2

[Peter Radcliffe [:pir]]

Connect as the mozillaadmin user from the appropriate admin machine.

[[e-mail]3 ~]$ ssh [e-mail]
The authenticity of host 'boa-d1.r302-2.console.scl3.mozilla.com (10.22.2.206can't be established.
DSA key fingerprint is 67:47:9b:65:42:34:a6:17:24:d7:21:a4:71:91:ef:6a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'boa-d1.r302-2.console.scl3.mozilla.com,10.22.2.20(DSA) to the list of known hosts.

-----------------------------------------------------------------------------
WARNING: This is a private system.  Do not attempt to login unless you are an
authorized user.  Any authorized or unauthorized access and use may be moni-
tored and can result in criminal or civil prosecution under applicable law.
-----------------------------------------------------------------------------

[[e-mail]3 ~]$ ssh [e-mail]

-----------------------------------------------------------------------------
WARNING: This is a private system.  Do not attempt to login unless you are an
authorized user.  Any authorized or unauthorized access and use may be moni-
tored and can result in criminal or civil prosecution under applicable law.
-----------------------------------------------------------------------------

Firmware Version: 4.22
Built: 06/13/2014 @ 07:29
OA Bay Number:  1 
OA Role:        Active 
[e-mail]'s password: 






HP BladeSystem Onboard Administrator
(C) Copyright 2006-2014 Hewlett-Packard Development Company, L.P.


Type 'HELP' to display a list of valid commands.
Type 'HELP <command>' to display detailed information about a specific comman
Type 'HELP HELP' to display more detailed information about the help system.


boa102-02d1>

Comment 3

[Marlena]

[[e-mail]3 ~]$ ssh [e-mail]

-----------------------------------------------------------------------------
WARNING: This is a private system.  Do not attempt to login unless you are an
authorized user.  Any authorized or unauthorized access and use may be moni-
tored and can result in criminal or civil prosecution under applicable law.
-----------------------------------------------------------------------------

Firmware Version: 4.22
Built: 06/13/2014 @ 07:29
OA Bay Number:  1 
OA Role:       	Active 
[e-mail]'s password: 


HP BladeSystem Onboard Administrator
(C) Copyright 2006-2014 Hewlett-Packard Development Company, L.P.


Type 'HELP' to display a list of valid commands.
Type 'HELP <command>' to display detailed information about a specific command.
Type 'HELP HELP' to display more detailed information about the help system.


No entry for terminal type "xterm-256color";
using dumb terminal settings.
boa102-02d1> show server names

Bay Server Name                                       Serial Number   Status   Power   UID Partner
--- ------------------------------------------------- --------------- -------- ------- --- -------
  1 [Absent]                                          
  2 hgweb8.dmz.scl3.mozilla.com                       MXQ20209JC      OK       On      Off 
  3 elasticsearch2.webapp.scl3.mozilla.com            MXQ2260JMQ      OK       Off     Off 
  4 elasticsearch4.webapp.scl3.mozilla.com            MXQ22615DF      OK       Off     Off 
  5 backup5.db.scl3.mozilla.com                       MXQ30101JN      OK       On      Off 6
  6 Storage Blade                                     TWT2430075      OK       On      On  5
  7 pgdb2.paas.scl3.mozilla.com                       MXQ25005Y0      OK       Off     Off 
  8 [Absent]                                          
  9 [Absent]                                          
 10 zlb02.nms.mozilla.org                             MXQ03806Z3      OK       On      Off 
 11 TABLEAU3-SCL3                                     2M2502025F      Degraded On      Off    ************
 12 [Absent]                                          
 13 [Absent]                                          
 14 [Absent]                                          
 15 [Absent]                                          
 16 [Absent]                                          
Totals: 8 server blades installed, 5 powered on.

boa102-02d1>

Comment 4

[Peter Radcliffe [:pir]]

From ilo:

Controller on System Board

    Controller Status	 OK
    Serial Number	PDZVU0HLM7S1N2
    Model	HP Smart Array P244br Controller
    Firmware Version	1.34

    Cache Module Status	 Failed
    Cache Module Serial Number	PDZVU0HLM7S1N2
    Cache Module Memory	1048576 KB

    Encryption Status	 Not Enabled
    Encryption ASIC Status	 OK
    Encryption Critical Security Parameter NVRAM Status	 OK

Comment 5

[Vinh Hua [:vinh]]

HP Case ID is 4653284646

HP is shipping replacement parts.

Comment 6

[Vinh Hua [:vinh]]

2 parts will be shipped.


Part Number: 815984-001      SPS-BATT PACK ENHANCED MegaCell 12W

UPS Tracking number: 1ZA7Y0140132148469 

For Next Business Day orders, the order can be tracked at:
http://wwwapps.ups.com/etracking/tracking.cgi?TypeOfInquiryNumber=T&InquiryNumber1=1ZA7Y0140132148469

For Same Day orders, the order can be tracked at: 
https://www.upspostsaleslogistics.com/cfw/trackOrder.do?trackNumber=1ZA7Y0140132148469



Part Number: 749800-001      SPS-BD AROC P244br PCIe CNTRL

UPS Tracking number: 1ZA7Y0140132148478 

For Next Business Day orders, the order can be tracked at:
http://wwwapps.ups.com/etracking/tracking.cgi?TypeOfInquiryNumber=T&InquiryNumber1=1ZA7Y0140132148478

For Same Day orders, the order can be tracked at: 
https://www.upspostsaleslogistics.com/cfw/trackOrder.do?trackNumber=1ZA7Y0140132148478

Comment 7

[Van Le [:van]]

i believe the pcie controller they sent is bad/DOA after tinkering with it for 30 minutes. i was unable to get the host to boot with it and it kept bypassing the smart array options. i used the original controller but swapped out the  cache battery. :vinh informed me it's no longer alerting but please reopen and we'll revisit if issues persist.