Closed Bug 755012 Opened 10 years ago Closed 9 years ago

prune old buildbot schedulers

Categories

(Release Engineering :: General, defect, P2)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: catlee, Assigned: catlee)

References

Details

(Whiteboard: [buildbot][schedulers][cleanup])

Attachments

(3 files)

schedulers that are disabled for a "long time" still sit around in the buildbot database. If they're ever re-activated then they have a large backlog of changes to work through.

Ideally we should delete old buildbot schedulers from the db so this doesn't happen.

The symptoms of this are the "last_processed" in the scheduler state referring to a really old change.
Duplicate of this bug: 759222
We talked about this today in triage. We could expire schedulers with last_processed data which refers to a change which is older than some threshold (a week, a month). It could be also be limited by scheduler class to avoid accidental damage. As the state column is json data it would make sense to do the cleanup as python script rather than bare SQL.
Blocks: 617017
Priority: -- → P2
Duplicate of this bug: 786226
I took a stab at this today.

http://cruncher.build.mozilla.org/~catlee/old_schedulers.txt is the result of all schedulers with:
- last_build < cutoff time
or
- date of last_processed change < cutoff time

I chose 2 weeks as the cutoff.

I don't think any of these are active schedulers.
hmm...
to-delete (old build) mozilla-central pgo {u'last_build': 1338836404.9286871, u'last_buildid': u'20120604120004', u'last_processed': 787419}

isn't that still active? why isn't last_build or last_processed updated?
(In reply to Chris AtLee [:catlee] from comment #5)
> hmm...
> to-delete (old build) mozilla-central pgo {u'last_build':
> 1338836404.9286871, u'last_buildid': u'20120604120004', u'last_processed':
> 787419}
> 
> isn't that still active? why isn't last_build or last_processed updated?

Ah, we have two schedulers with that name, different classes:

mysql> select * from schedulers where name = 'mozilla-central pgo';                                                                                                                                                                                                             
+-------------+---------------------+------------------------------------------------+------------------------------------------------------------------------------------------------+
| schedulerid | name                | class_name                                     | state                                                                                          |
+-------------+---------------------+------------------------------------------------+------------------------------------------------------------------------------------------------+
|        3382 | mozilla-central pgo | buildbotcustom.scheduler.Nightly-props         | {"last_build": 1338836404.9286871, "last_buildid": "20120604120004", "last_processed": 787419} | 
|        4296 | mozilla-central pgo | buildbotcustom.scheduler.SpecificNightly-props | {"last_build": 1368104401.524662, "last_buildid": "20130508120241", "last_processed": 1363805} | 
+-------------+---------------------+------------------------------------------------+------------------------------------------------------------------------------------------------+
I deleted a bunch of old schedulers this morning. So far no fallout detected.
Product: mozilla.org → Release Engineering
clean up schedulers we know how to

many schedulers don't have enough state in the db to determine if they can be deleted or not (e.g. nightly schedulers). you'd need to know which branches are active, or inspect buildbot state or something.

this patch will cleanup up the worst offenders at least, which are the regular test schedulers.
Attachment #8343151 - Flags: review?(nthomas)
Attached file cleanup.log
list of schedulers it would delete now
Comment on attachment 8343151 [details] [diff] [review]
cleanup_db-schedulers.diff

lgtm, the log was helpful.
Attachment #8343151 - Flags: review?(nthomas) → review+
Attachment #8343151 - Flags: checked-in+
Attachment #8345891 - Flags: review?(dustin)
Attachment #8345891 - Flags: review?(dustin) → review+
Attachment #8345891 - Flags: checked-in+
Assignee: nobody → catlee
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Is this on both production *and* stage?
No, just for production.
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.