528670 - 1.9.2 tinderboxes are not cycling

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

As pointed out by philor on IRC, according to http://tests.themasta.com/tinderboxpushlog/?tree=Firefox3.6 there are a bunch of builds on the 1.9.2 branch that should have finished many hours ago but didn't, and then there are a whole bunch of pushes since then that should have triggered builds but have not.  http://tinderbox.mozilla.org/Firefox3.6/ and http://hg.mozilla.org/releases/mozilla-1.9.2/pushloghtml seem to confirm this impression.

Comment 1

[Phil Ringnalda (:philor)]

While superficially looking to be in better shape, the trunk is looking pretty messed up too - Windows build completed pretty normally for a push at 11:30, and the only other completed build was for a push at 11:47, which started at 16:26 (though there are quite a few others that claim they are building, for up to 11 hours).

Comment 2

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Though it looks like m-c wasn't cycling for a period around the same time, and has now started again.  So maybe whatever was wrong is now fixed, and we're just missing builds for those pushes.

Comment 3

[Aki Sasaki (not active)]

production-master crashed at 15:05 and i brought it back up at 15:50.  any builds in progress or queued before 15:50 would be lost.

Comment 4

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Yeah, that fits with what happened.

I pushed a spelling fix to m-c and 1.9.2, so I think things should be ok once that cycles on 1.9.2.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: moz2-win32-slave13 twist.log

A number of slaves seem to be missing after the buildbot master restart. Here's a log from a win32 slave, where buildbot is still running but not connected to the master.

I'll go round and give them a boot.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Restarted buildbot on moz2-win32-slave01,07,13,14,17,23,24,27,29,30,33,35,38,39, 42. Probably everything that was running a job when the master died. Linux and mac up next.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Restarted buildbot on:
bm-xserve09,11,12,19
moz2-darwin9-slave02,05,06,07,09,11,13,14
moz2-linux-slave-07,08,11,12,14,16,19,20,22,23,25,27,28,41
moz2-linux64-slave04,06,08

Also found some pm02 slaves hung, see bug 523827 comment #22.