620041 - bring back up remaining fed/fed64 minis at scl

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[[e-mail] (:bhearsum)]

After the downtime today a bunch of Fedora and Fedora 64-bit minis wouldn't come back online. I suspect most or all need a re-image:

talos-r3-fed-001.build.scl1.mozilla.com
talos-r3-fed-012.build.scl1.mozilla.com
talos-r3-fed-033.build.scl1.mozilla.com
talos-r3-fed-036.build.scl1.mozilla.com
talos-r3-fed-038.build.scl1.mozilla.com
talos-r3-fed-040.build.scl1.mozilla.com
talos-r3-fed-041.build.scl1.mozilla.com
talos-r3-fed-044.build.scl1.mozilla.com
talos-r3-fed64-001.build.scl1.mozilla.com
talos-r3-fed64-016.build.scl1.mozilla.com
talos-r3-fed64-017.build.scl1.mozilla.com
talos-r3-fed64-021.build.scl1.mozilla.com
talos-r3-fed64-027.build.scl1.mozilla.com
talos-r3-fed64-030.build.scl1.mozilla.com
talos-r3-fed64-043.build.scl1.mozilla.com
talos-r3-fed64-047.build.scl1.mozilla.com
talos-r3-fed64-053.build.scl1.mozilla.com
talos-r3-fed64-055.build.scl1.mozilla.com

This is enough that it makes a dent in our wait times, so marking this as "major".

Comment 1

[Zandr Milewski [:zandr]]

Did a quick survey and fixed a few:

talos-r3-fed-001.build.scl1.mozilla.com - OK
talos-r3-fed-012.build.scl1.mozilla.com - needs reimage (errors)
talos-r3-fed-033.build.scl1.mozilla.com - needs reimage (errors)
talos-r3-fed-036.build.scl1.mozilla.com - OK - reimaged
talos-r3-fed-038.build.scl1.mozilla.com - hangs
talos-r3-fed-040.build.scl1.mozilla.com - not in scl1
talos-r3-fed-041.build.scl1.mozilla.com - OK - reimaged
talos-r3-fed-044.build.scl1.mozilla.com - needs reimage (errors)
talos-r3-fed64-001.build.scl1.mozilla.com - needs reimage (hangs)
talos-r3-fed64-016.build.scl1.mozilla.com - needs reimage (hangs)
talos-r3-fed64-017.build.scl1.mozilla.com - responding
talos-r3-fed64-021.build.scl1.mozilla.com - needs reimage (hangs)
talos-r3-fed64-027.build.scl1.mozilla.com - needs reimage (hangs)
talos-r3-fed64-030.build.scl1.mozilla.com - not in scl1
talos-r3-fed64-043.build.scl1.mozilla.com - needs reimage (hangs)
talos-r3-fed64-047.build.scl1.mozilla.com - not in scl1
talos-r3-fed64-053.build.scl1.mozilla.com - needs reimage (hangs)
talos-r3-fed64-055.build.scl1.mozilla.com - not in scl1

Out of time for tonight, will pick this up next week.

Comment 2

[Zandr Milewski [:zandr]]

talos-r3-fed-012.build.scl1.mozilla.com - imaging OK
talos-r3-fed-033.build.scl1.mozilla.com - imaging OK
talos-r3-fed-038.build.scl1.mozilla.com - imaging OK
talos-r3-fed-040.build.scl1.mozilla.com - OK
talos-r3-fed-044.build.scl1.mozilla.com - imaging OK
talos-r3-fed64-001.build.scl1.mozilla.com - imaging OK
talos-r3-fed64-016.build.scl1.mozilla.com - imaging OK
talos-r3-fed64-021.build.scl1.mozilla.com - imaging OK
talos-r3-fed64-027.build.scl1.mozilla.com - imaging OK
talos-r3-fed64-030.build.scl1.mozilla.com - OK
talos-r3-fed64-043.build.scl1.mozilla.com - imaging OK
talos-r3-fed64-053.build.scl1.mozilla.com - imaging OK


Please use a new bug if you find any other machines in scl1 amiss. I think we're in good shape as of now.

/me leaves quickly, while that's still true.