1039313 - Suspiciously high number of talos slaves are broken

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Simone Bruno [:simone]]

A high percentage of both talos-linux32-ix and talos-linux64-ix boxes are reported as broken in slave_health. This is causing an increasing jobs backlog (especially for 64 machines).

Comment 1

[Simone Bruno [:simone]]

slaverebooter is apparently failing to reboot these machines:

[6:03pm] coop: something is causing slaverebooter to hang before it gets to the talos-linux64-ix machines
[6:03pm] coop: since those are last in its list
[6:03pm] coop: i blame xp

Also see: https://bugzilla.mozilla.org/show_bug.cgi?id=971861#c7, which "may" be related

Comment 2

[Simone Bruno [:simone]]

:coop: now the situation seems improved: did you do anything (slaverebooter related steps, manual rebooting, ...)?

Comment 3

[Chris Cooper [:coop] (he/him)]

(In reply to Simone Bruno [:simone] from comment #2)
> :coop: now the situation seems improved: did you do anything (slaverebooter
> related steps, manual rebooting, ...)?

Jordan discovered that slaveapi needed the updated passwords for cltbld and Administrator. On top of that, I manually rebooted all the talos-linux* slaves that weren't still actively taking jobs.

Windows pending jobs are still high today, but I think Linux is fixed for now.