1516332 - [MDC1] T-W1064-MS-090 boot-up problems

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Hardware, task)

Description

[Zsolt Fay [:zfay]]

Worker has been experiencing problems for the past 4 days. From the logs i believe the worker hasn't recovered after a reboot:

https://papertrailapp.com/systems/1659271201/events?focus=1012444402418872360&selected=1012444402418872360

Re-imaging is not possible since during boot, the bios loading happens way faster than it should (makes you think some steps are skipped), and it doesn't enter netboot.

Comment 1

[Radu Iman[:riman]]

[MDC1]T-W1064-MS-022 and [MDC1]T-W1064-MS-508 have the same issue

Comment 2

[Zsolt Fay [:zfay]]

Mark please take a look at these 3 workers when you have the time

Comment 3

[Mark Cornmesser [:markco]]

I will take a look at these this week. Not clearing the NI so i have a reminder.

Comment 4

[Mark Cornmesser [:markco]]

ms-090 is now installing.

I had to ssh to the chassis and run:

reset cartridge power force c45

Comment 5

[Jake Watkins [:dividehex]]

Mark, did the reset fix this and can this be closed?

Comment 6

[Mark Cornmesser [:markco]]

I was hoping buildduty would address the other nodes here.

Comment 7

[Zsolt Fay [:zfay]]

Worker has been successfully completing jobs since the reset.
https://tools.taskcluster.net/provisioners/releng-hardware/worker-types/gecko-t-win10-64-hw/workers/mdc1/T-W1064-MS-090

Comment 8

[Attila Craciun [:arny]]

Fixed T-W1064-MS-508 with reset cartridge power force cX, reimaging now.