915457 - Triage tegras with no completed jobs within last 24 hours

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P1)

Description

[Justin Wood (:Callek)]

We currently have about 145 tegras that have not run/completed a job within the last 24 hours, and have a large (>1100) pending count for tegra-run jobs.

We need to triage this list, look for any systemic problems and nurse these back to life as soon as possible.

https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slavetype.html?class=test&type=tegra

Comment 1

[Justin Wood (:Callek)]

Coop recovered most of these this past evening... so I concentrated on other related work instead of this bug, so handing to him pending when I get up ;-)

Coop managed the recovery by:
(in a script loop, across foopies/devices):

* Kill any hung/idle buildbot procs
* force reboot device

As of a few hours ago we had all but 40 devices up!

Comment 2

[Chris Cooper [:coop] (he/him)]

Attachment: resurrect_tegras.py

Here's the script I ran last night to resurrect the hung tegras.

'tegra_list' contained a list of hung tegras, as reported by slave_health. I'll get this script added to braindump today.

I think we should turn back on kittenherder reboots for tegras in the short-term.

Comment 3

[Chris Cooper [:coop] (he/him)]

The tegra problem tracking bugs are a mess. I'll take some time today to try to resolve any open tegra bugs that shouldn't still be open, provided we don't hit another try-nado or similar apocalypse.

Comment 4

[Chris Cooper [:coop] (he/him)]

This is mostly cleaned up now. tegras that were in the buildduty queue have all been nudged to their next state, whether that's recovery or production.