If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Please remove zombie running builds from db

RESOLVED FIXED

Status

Release Engineering
General
RESOLVED FIXED
5 years ago
5 months ago

People

(Reporter: philor, Unassigned)

Tracking

({buildapi})

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

5 years ago
+++ This bug was initially created as a clone of Bug #825466 +++

I was going to wait 60 days, because annoying, but a combination of things makes that too long

* because we can't run Android tests on a rev which doesn't have the jar signing patch from bug 705807, every time someone pushes something to try which doesn't have that, I have to kill (and kill and kill and kill, since catching RETRY takes some doing, as the number of jobs on https://tbpl.mozilla.org/?tree=Try&rev=a6775caaf73e demonstrates) the Android tests on it, which requires seeing whether or not any are running by sorting https://secure.pub.build.mozilla.org/buildapi/running by Submitted at and looking for things that were submitted long ago but just started running

* shutting off mochitests on Ubuntu and shutting off gaia-central builds in rapid succession left us with not only the 5 mis-named zombie spidermonkey builds from three weeks ago, but also 25 Ubuntu mochitest zombies and 3 gaia-central zombies, so the entire first page of buildapi/running is now nothing but zombies
Ok, this took some manual looking at https://secure.pub.build.mozilla.org/buildapi/running and its running jobs, to correlate what I needed to find.

I'll be attaching the SQL log of my actions, for future reference and perusal. (and incase I missed a step, or deleted stuff I shouldn't have) including flagging catlee for a f+ that I did the right thing.

(all submitted at dates in the SQL tables are in unix epoch format, I use http://www.epochconverter.com/ locally when I need to translate a small number at once)

As of this writing oldest "submitted at" job in the web BuildAPI is 2013-02-02 18:08:40
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Created attachment 709442 [details]
commands, as run

These commands were issued through a mysql connection from bm36 our host:

mysql -u <our user> -p -h <our host>.db.scl3.mozilla.com buildbot_schedulers

See cltbld history if you [another releng person] needs this in future (`history | grep buildbot_schedulers`) password is in passwords.py for the master
Attachment #709442 - Flags: feedback?(catlee)
Comment on attachment 709442 [details]
commands, as run

from_unixtime(claimed_at) is your friend here. IIRC that's the field which gets updated every few minutes when the job is running, rather than the time it actually started (which is builds.start_time).
Comment on attachment 709442 [details]
commands, as run

looks ok to me.

I usually mark them with complete=1, complete_at=unix_timestamp(now()) instead of deleting them
Attachment #709442 - Flags: feedback?(catlee) → feedback+
(Assignee)

Updated

4 years ago
Product: mozilla.org → Release Engineering
(Assignee)

Updated

5 months ago
Component: Tools → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.