837453 - Please remove zombie running builds from db

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Phil Ringnalda (:philor)]

+++ This bug was initially created as a clone of Bug #825466 +++

I was going to wait 60 days, because annoying, but a combination of things makes that too long

* because we can't run Android tests on a rev which doesn't have the jar signing patch from bug 705807, every time someone pushes something to try which doesn't have that, I have to kill (and kill and kill and kill, since catching RETRY takes some doing, as the number of jobs on https://tbpl.mozilla.org/?tree=Try&rev=a6775caaf73e demonstrates) the Android tests on it, which requires seeing whether or not any are running by sorting https://secure.pub.build.mozilla.org/buildapi/running by Submitted at and looking for things that were submitted long ago but just started running

* shutting off mochitests on Ubuntu and shutting off gaia-central builds in rapid succession left us with not only the 5 mis-named zombie spidermonkey builds from three weeks ago, but also 25 Ubuntu mochitest zombies and 3 gaia-central zombies, so the entire first page of buildapi/running is now nothing but zombies

Comment 1

[Justin Wood (:Callek)]

Ok, this took some manual looking at https://secure.pub.build.mozilla.org/buildapi/running and its running jobs, to correlate what I needed to find.

I'll be attaching the SQL log of my actions, for future reference and perusal. (and incase I missed a step, or deleted stuff I shouldn't have) including flagging catlee for a f+ that I did the right thing.

(all submitted at dates in the SQL tables are in unix epoch format, I use http://www.epochconverter.com/ locally when I need to translate a small number at once)

As of this writing oldest "submitted at" job in the web BuildAPI is 2013-02-02 18:08:40

Comment 2

[Justin Wood (:Callek)]

Attachment: commands, as run

These commands were issued through a mysql connection from bm36 our host:

mysql -u <our user> -p -h <our host>.db.scl3.mozilla.com buildbot_schedulers

See cltbld history if you [another releng person] needs this in future (`history | grep buildbot_schedulers`) password is in passwords.py for the master

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 709442 [details]
commands, as run

from_unixtime(claimed_at) is your friend here. IIRC that's the field which gets updated every few minutes when the job is running, rather than the time it actually started (which is builds.start_time).

Comment 4

[Chris AtLee [:catlee]]

Comment on attachment 709442 [details]
commands, as run

looks ok to me.

I usually mark them with complete=1, complete_at=unix_timestamp(now()) instead of deleting them