Zombie running builds in scheduler db

RESOLVED FIXED

Status

Release Engineering
General
RESOLVED FIXED
7 years ago
5 years ago

People

(Reporter: nthomas, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

7 years ago
The MPT outage over the weekend appears to have left lots of builds in a running state in the db when in fact they have finished. Presumably the DB was not responding at that point and the transaction timed out or was dropped. Upstream bug for that ?

For example, 
 http://production-master03.build.mozilla.org:8010/builders/OS%20X%2010.6.2%20mozilla-central%20build/builds/2359
is 
 Start	Fri Jan 14 23:00:31 2011
 End	Fri Jan 14 23:09:20 2011
 Elapsed	8 mins, 48 secs
with a retry final state due to the network connection dropping.
In the db, there are two rows in the builds table (?!), both with builds.finish still NULL, and in buildrequests complete=0, results=NULL, complete_at=NULL.
[SELECT * from builds,buildrequests WHERE builds.brid=buildrequests.id AND builds.number=2359 AND buildrequests.buildername="OS X 10.6.2 mozilla-central build" ORDER BY builds.start_time LIMIT 10;]

The fix we need here is to clean up the db so that the list of running list is accurate.
(Reporter)

Comment 1

7 years ago
Just one remaining zombie:

Branch	Revision	Builder name	Submitted at	Running since	Running for	Master
try	3e8c4b0f6e68	Rev3 Fedora 12 tryserver debug test mochitest-other	2011-01-14 14:18:28	2011-01-14 19:25:52	2 days, 5:06:49	buildbot-master2:8011

which I bet would go away if the master gets a restart like the others have had.
Looks ok now.
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
(Assignee)

Updated

5 years ago
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.