967083 - Jobs not being scheduled

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Armen [:armenzg]]

jlund|buildduty: https://tbpl.mozilla.org/?showall=1&tree=Fx-Team and look at the lack of any grey letters indicating either pending or running builds on either of the tip two pushes
<philor> jlund|buildduty: and https://secure.pub.build.mozilla.org/buildapi/self-serve/jobs noting the lack of any completed-at dates
<jlund|buildduty> hmm, ya last completed at was 8:59
<jlund|buildduty> might have to restart buildapi or the serve agent
<armenzg> it sounds like the cruncher restart could be involved
* armenzg logs in to cruncher
<philor> since I just closed every single tree over not scheduling
<aki> oh, guessing that's due to cruncher
<armenzg> I can't ssh into cruncher

Comment 1

[Armen [:armenzg]]

It could be related to the 1hr downtime
https://adam.pastebin.mozilla.org/4174290

Comment 2

[Armen [:armenzg]]

armenzg> cknowles, did cruncher come back up for you?
<armenzg> we might be having a tree closure issue
<cknowles> I was working on the console, and it was happy as a clam. 
<cknowles> After awhile I was able to get back in, but I appear to have been cut off again.

Comment 3

[Jordan Lund (:jlund)]

after yesterday's issues were resolved: mostly boiling down to downtime + network availability, jobs started scheduling again. :)