+++ This bug was initially created as a clone of Bug #825466 +++ I was going to wait 60 days, because annoying, but a combination of things makes that too long * because we can't run Android tests on a rev which doesn't have the jar signing patch from bug 705807, every time someone pushes something to try which doesn't have that, I have to kill (and kill and kill and kill, since catching RETRY takes some doing, as the number of jobs on https://tbpl.mozilla.org/?tree=Try&rev=a6775caaf73e demonstrates) the Android tests on it, which requires seeing whether or not any are running by sorting https://secure.pub.build.mozilla.org/buildapi/running by Submitted at and looking for things that were submitted long ago but just started running * shutting off mochitests on Ubuntu and shutting off gaia-central builds in rapid succession left us with not only the 5 mis-named zombie spidermonkey builds from three weeks ago, but also 25 Ubuntu mochitest zombies and 3 gaia-central zombies, so the entire first page of buildapi/running is now nothing but zombies
Ok, this took some manual looking at https://secure.pub.build.mozilla.org/buildapi/running and its running jobs, to correlate what I needed to find. I'll be attaching the SQL log of my actions, for future reference and perusal. (and incase I missed a step, or deleted stuff I shouldn't have) including flagging catlee for a f+ that I did the right thing. (all submitted at dates in the SQL tables are in unix epoch format, I use http://www.epochconverter.com/ locally when I need to translate a small number at once) As of this writing oldest "submitted at" job in the web BuildAPI is 2013-02-02 18:08:40
Created attachment 709442 [details] commands, as run These commands were issued through a mysql connection from bm36 our host: mysql -u <our user> -p -h <our host>.db.scl3.mozilla.com buildbot_schedulers See cltbld history if you [another releng person] needs this in future (`history | grep buildbot_schedulers`) password is in passwords.py for the master
Comment on attachment 709442 [details] commands, as run from_unixtime(claimed_at) is your friend here. IIRC that's the field which gets updated every few minutes when the job is running, rather than the time it actually started (which is builds.start_time).
Comment on attachment 709442 [details] commands, as run looks ok to me. I usually mark them with complete=1, complete_at=unix_timestamp(now()) instead of deleting them