626283 - Zombie running builds in scheduler db

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

The MPT outage over the weekend appears to have left lots of builds in a running state in the db when in fact they have finished. Presumably the DB was not responding at that point and the transaction timed out or was dropped. Upstream bug for that ?

For example, 
 http://production-master03.build.mozilla.org:8010/builders/OS%20X%2010.6.2%20mozilla-central%20build/builds/2359
is 
 Start	Fri Jan 14 23:00:31 2011
 End	Fri Jan 14 23:09:20 2011
 Elapsed	8 mins, 48 secs
with a retry final state due to the network connection dropping.
In the db, there are two rows in the builds table (?!), both with builds.finish still NULL, and in buildrequests complete=0, results=NULL, complete_at=NULL.
[SELECT * from builds,buildrequests WHERE builds.brid=buildrequests.id AND builds.number=2359 AND buildrequests.buildername="OS X 10.6.2 mozilla-central build" ORDER BY builds.start_time LIMIT 10;]

The fix we need here is to clean up the db so that the list of running list is accurate.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Just one remaining zombie:

Branch	Revision	Builder name	Submitted at	Running since	Running for	Master
try	3e8c4b0f6e68	Rev3 Fedora 12 tryserver debug test mochitest-other	2011-01-14 14:18:28	2011-01-14 19:25:52	2 days, 5:06:49	buildbot-master2:8011

which I bet would go away if the master gets a restart like the others have had.

Comment 2

[Chris AtLee [:catlee]]

Looks ok now.