1223042 - Old revisions of android tests were scheduled

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P1)

Description

[Chris AtLee [:catlee]]

Something happened this morning to make us schedule jobs from last week for 'Android 4.3 armv7 API 11+' jobs. We have about 20,000 extra jobs that shouldn't have been scheduled.

Here's the query I'm using to try and fix it:

update buildrequests set complete=1, results=2, complete_at=1447086265 where buildrequests.id in (select buildrequests.id from buildsets, sourcestamp_changes, changes where changes.changeid = sourcestamp_changes.changeid and sourcestamp_changes.sourcestampid = buildsets.sourcestampid and buildrequests.buildsetid = buildsets.id and buildrequests.complete = 0 and buildrequests.claimed_at =0 and buildername like 'Android 4.3 armv7 API 11+%' and when_timestamp < 1447081200);

Comment 1

[Chris AtLee [:catlee]]

that was taking ages, so I did this instead:

create temporary table ids select buildrequests.id from buildrequests, buildsets, sourcestamp_changes, changes w
here changes.changeid = sourcestamp_changes.changeid and sourcestamp_changes.sourcestampid = buildsets.sourcestampid an
d buildrequests.buildsetid = buildsets.id and buildrequests.complete = 0 and buildrequests.claimed_at =0 and buildernam
e like 'Android 4.3 armv7 API 11+%' and when_timestamp < 1447081200;

update buildrequests, ids set complete=1, results=2, complete_at=1447086265 where buildrequests.id=ids.id and co
mplete=0 and claimed_at=0;

Comment 2

[Chris AtLee [:catlee]]

Ok, cleaned up the mess....

I wonder if there's something we can do to prevent this from happening again. Perhaps refusing to schedule jobs for changes more than X hours old?

Comment 4

[Kim Moir [:kmoir] ET]

alin

puppet/modules/buildmaster/manifests/db_maintenance.pp is a good reference for the puppet config that creates the scripts that clean the dbs of old jobs

Comment 5

[Vlad Ciobancai [:vladC] [:vciobancai]]

Is there any database, where we can test the script ?

Comment 6

[Kim Moir [:kmoir] ET]

Well, we don't really want to test the script on the production databases.  I don't think there is a staging version of the buildbot scheduler db.  What you could do look sql that the changes to the script generates and see if the entries it wants to mark as complete are sane.

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

I think we can close this since we've gotten Android off of BB.
Please feel free to re-open if I'm wrong.