1484258 - (T-W1064-MS-217) [MDC1] T-W1064-MS-217 problem tracking

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Bogdan Crisan [:bcrisan] (EEST - GMT + 3)]

The workers was in a bad state and has been re-imaged, needs verification.

Comment 1

[Roland Mutter Michael (:rmutter)]

Looks okey now: https://tools.taskcluster.net/groups/On0nKVuGShuM2_w5q5ZKeQ/tasks/FPXmx5JQSbuMCm1OlKAyqg/runs/0

Comment 2

[:dhouse]

I have powered off this machine until Mark can check it out.


Moonshot Chassis ILO reported this machine as overheating.
```
EVENT 710: (19-Sep-2018 14:21): Automatic Operating System Shutdown Initiated Due to Overheat Condition

Integrated Management Log Severity: CRITICAL
Integrated Management Log Class: OS
Integrated Management Log Location: Cartridge 37

iLO CM URL: https://moon-chassis-5.inband.releng.mdc1.mozilla.com
moon-chassis-5, 1.58 Jun 01 2018 | Unknown
```


Maybe it is stuck during boot. I see no logs in papertrail for this
cartridge for 1 day.
https://papertrailapp.com/groups/1141234?filter=T-W1064-MS-217
https://papertrailapp.com/systems/1730896111/events?focus=978743890456776709

Last task in taskcluster was yesterday:
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-win10-64-hw/workers/mdc1/T-W1064-MS-217

cartridge serial #CN67200BAG
10.49.40.143 / f4:03:43:df:39:c1

Temperatures:
```
hpiLO->show cartridge temperature history c37
[...]
    Temperature Sensor 2:
      Location:  02-CPU
      Status:  Failed
      Reading:            127 Celsius
      Temperature History:        #  1    2    3    4    5    6    7    8    9   10
                                  #---  ---  ---  ---  ---  ---  ---  ---  ---  ---
        Last Minute (2 seconds):   143  143  143  143  143  143  143  143  143  143
                                   143  143  143  143  143  143  143
[...]
# powered off
    Temperature Sensor 2:
      Location:  02-CPU
      Status:  Ok
      Reading:            N/A
      Temperature History:        #  1    2    3    4    5    6    7    8    9   10
                                  #---  ---  ---  ---  ---  ---  ---  ---  ---  ---
        Last Minute (2 seconds):                                                   
        Last Hour (1 minute)   :   143  143  143  143  100   19                    
        Last Day (1 hour)      :    39   40   88  143  139  143  143  142  142  143
                                   143  143  143  143  142  142  143  143  143  143
        Last Month (1 day)     :                                                   
        Last Year (1 month)    :        
```

Comment 3

[:dhouse]

I verified with Mark we can ask for an RMA on this machine (created bug 1492927).

Comment 4

[Mark Cornmesser [:markco]]

Closing this bug. It appears to be back in place: https://papertrailapp.com/groups/1141234/events?q=ms-217

Comment 5

[Adrian Pop]

Re-opening bug.
I have tried to connect on the machine via ssh, iLo integrated console but I couldn't.
By pinging the machine, I'vereceived the following :
fping T-W1064-MS-217.wintest.releng.mdc1.mozilla.com
T-W1064-MS-217.wintest.releng.mdc1.mozilla.com is unreachable

On papertrai, I have found this 
Nov 11 03:47:01 rejh1.srv.releng.mdc1.mozilla.com sshd: error: connect_to t-w1064-ms-217.wintest.releng.mdc1.mozilla.com port 22: failed. 

I've checked also from the web but there I've found our that the cartridge is not set. Can you please take a look ?

Comment 6

[Adrian Pop]

re-open bug. the machine seems to be offline

Comment 7

[Attila Craciun [:arny]]

VAN, please reseat this: MDC1, chassis-5, C37 - is not present in the chassis.

Comment 8

[Van Le [:van]]

REQ0255913  opened to reseat cartridge.

Comment 9

[Attila Craciun [:arny]]

This is back in action, running jobs.