755012 - prune old buildbot schedulers

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Chris AtLee [:catlee]]

schedulers that are disabled for a "long time" still sit around in the buildbot database. If they're ever re-activated then they have a large backlog of changes to work through.

Ideally we should delete old buildbot schedulers from the db so this doesn't happen.

The symptoms of this are the "last_processed" in the scheduler state referring to a really old change.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

We talked about this today in triage. We could expire schedulers with last_processed data which refers to a change which is older than some threshold (a week, a month). It could be also be limited by scheduler class to avoid accidental damage. As the state column is json data it would make sense to do the cleanup as python script rather than bare SQL.

Comment 4

[Chris AtLee [:catlee]]

I took a stab at this today.

http://cruncher.build.mozilla.org/~catlee/old_schedulers.txt is the result of all schedulers with:
- last_build < cutoff time
or
- date of last_processed change < cutoff time

I chose 2 weeks as the cutoff.

I don't think any of these are active schedulers.

Comment 5

[Chris AtLee [:catlee]]

hmm...
to-delete (old build) mozilla-central pgo {u'last_build': 1338836404.9286871, u'last_buildid': u'20120604120004', u'last_processed': 787419}

isn't that still active? why isn't last_build or last_processed updated?

Comment 6

[Chris AtLee [:catlee]]

(In reply to Chris AtLee [:catlee] from comment #5)
> hmm...
> to-delete (old build) mozilla-central pgo {u'last_build':
> 1338836404.9286871, u'last_buildid': u'20120604120004', u'last_processed':
> 787419}
> 
> isn't that still active? why isn't last_build or last_processed updated?

Ah, we have two schedulers with that name, different classes:

mysql> select * from schedulers where name = 'mozilla-central pgo';                                                                                                                                                                                                             
+-------------+---------------------+------------------------------------------------+------------------------------------------------------------------------------------------------+
| schedulerid | name                | class_name                                     | state                                                                                          |
+-------------+---------------------+------------------------------------------------+------------------------------------------------------------------------------------------------+
|        3382 | mozilla-central pgo | buildbotcustom.scheduler.Nightly-props         | {"last_build": 1338836404.9286871, "last_buildid": "20120604120004", "last_processed": 787419} | 
|        4296 | mozilla-central pgo | buildbotcustom.scheduler.SpecificNightly-props | {"last_build": 1368104401.524662, "last_buildid": "20130508120241", "last_processed": 1363805} | 
+-------------+---------------------+------------------------------------------------+------------------------------------------------------------------------------------------------+

Comment 7

[Chris AtLee [:catlee]]

I deleted a bunch of old schedulers this morning. So far no fallout detected.

Comment 8

[Chris AtLee [:catlee]]

Attachment: cleanup_db-schedulers.diff

clean up schedulers we know how to

many schedulers don't have enough state in the db to determine if they can be deleted or not (e.g. nightly schedulers). you'd need to know which branches are active, or inspect buildbot state or something.

this patch will cleanup up the worst offenders at least, which are the regular test schedulers.

Comment 9

[Chris AtLee [:catlee]]

Attachment: cleanup.log

list of schedulers it would delete now

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8343151 [details] [diff] [review]
cleanup_db-schedulers.diff

lgtm, the log was helpful.

Comment 11

[Chris AtLee [:catlee]]

Attachment: pass schedulerdb info to cleanup_db

Comment 12

[Sheeri Cabral [:sheeri]]

Is this on both production *and* stage?

Comment 13

[Chris AtLee [:catlee]]

No, just for production.