If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Please remove zombie running builds from db



Release Engineering
5 years ago
5 months ago


(Reporter: philor, Unassigned)



Firefox Tracking Flags

(Not tracked)



(1 attachment)



5 years ago
+++ This bug was initially created as a clone of Bug #825466 +++

I was going to wait 60 days, because annoying, but a combination of things makes that too long

* because we can't run Android tests on a rev which doesn't have the jar signing patch from bug 705807, every time someone pushes something to try which doesn't have that, I have to kill (and kill and kill and kill, since catching RETRY takes some doing, as the number of jobs on https://tbpl.mozilla.org/?tree=Try&rev=a6775caaf73e demonstrates) the Android tests on it, which requires seeing whether or not any are running by sorting https://secure.pub.build.mozilla.org/buildapi/running by Submitted at and looking for things that were submitted long ago but just started running

* shutting off mochitests on Ubuntu and shutting off gaia-central builds in rapid succession left us with not only the 5 mis-named zombie spidermonkey builds from three weeks ago, but also 25 Ubuntu mochitest zombies and 3 gaia-central zombies, so the entire first page of buildapi/running is now nothing but zombies
Ok, this took some manual looking at https://secure.pub.build.mozilla.org/buildapi/running and its running jobs, to correlate what I needed to find.

I'll be attaching the SQL log of my actions, for future reference and perusal. (and incase I missed a step, or deleted stuff I shouldn't have) including flagging catlee for a f+ that I did the right thing.

(all submitted at dates in the SQL tables are in unix epoch format, I use http://www.epochconverter.com/ locally when I need to translate a small number at once)

As of this writing oldest "submitted at" job in the web BuildAPI is 2013-02-02 18:08:40
Last Resolved: 5 years ago
Resolution: --- → FIXED
Created attachment 709442 [details]
commands, as run

These commands were issued through a mysql connection from bm36 our host:

mysql -u <our user> -p -h <our host>.db.scl3.mozilla.com buildbot_schedulers

See cltbld history if you [another releng person] needs this in future (`history | grep buildbot_schedulers`) password is in passwords.py for the master
Attachment #709442 - Flags: feedback?(catlee)
Comment on attachment 709442 [details]
commands, as run

from_unixtime(claimed_at) is your friend here. IIRC that's the field which gets updated every few minutes when the job is running, rather than the time it actually started (which is builds.start_time).
Comment on attachment 709442 [details]
commands, as run

looks ok to me.

I usually mark them with complete=1, complete_at=unix_timestamp(now()) instead of deleting them
Attachment #709442 - Flags: feedback?(catlee) → feedback+


4 years ago
Product: mozilla.org → Release Engineering


5 months ago
Component: Tools → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.