The Win opt build here: https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=win&rev=dadb81f1db4f Opt + debug here: https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=win&rev=e79ca6ca1173 And all pushes newer than that. Inbound is closed anyway, but would remain closed for this once the other issues are fixed.
To be clear, everything in the jaccuzzi pool for inbound win builds is running an inbound win build. And we just had a round of non-unified and PGO start up.
RyanVM canceled all the broken builds (caused by a bad landing) so they don't retry,and I rebooted all the jacuzzi machines attached to these builders, all is now well.
What was the cause of this?
IIUC, the busted landing for bug 1046087 was leaving the slaves in a zombie state. Things went back to normal when we killed all pending/running Windows builds prior to the backout and Justin manually rebooted the slaves.