759214 - Jobs not being scheduled

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P1)

Description

[Phil Ringnalda (:philor)]

See, for instance, https://tbpl.mozilla.org/?tree=Mozilla-Inbound where the tip push hasn't scheduled builds, or https://tbpl.mozilla.org/?tree=Try where various builds have finished, but not scheduled the tests they should have.

Trees are closed.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Scheduling is working again now. More details to follow.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

There were two problems here

* it looked like builds were not started on inbound because the push to inbound was right on top of the reconfig, and the change got dropped on the floor (bug 675558 gets rid of an exception but changes can still be lost)

* tests were not being scheduled because that master got wedged trying to process all the changes since larch, jaegermonkey, and devtools were last enabled. More on that with an attachment ....

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: List of deleted schedulers

This is the schedulers I deleted while the test master was shutdown (a kill -9 was needed because the process was blocked on a futex). It was derived from using a hacked dump_master.py to get the list of active test schedulers, then looking in the buildbot_schedulers db to see which had processed a change less than 1300000.