prune old buildbot schedulers

RESOLVED FIXED

Status

P2
normal
RESOLVED FIXED
7 years ago
5 months ago

People

(Reporter: catlee, Assigned: catlee)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [buildbot][schedulers][cleanup])

Attachments

(3 attachments)

(Assignee)

Description

7 years ago
schedulers that are disabled for a "long time" still sit around in the buildbot database. If they're ever re-activated then they have a large backlog of changes to work through.

Ideally we should delete old buildbot schedulers from the db so this doesn't happen.

The symptoms of this are the "last_processed" in the scheduler state referring to a really old change.
(Assignee)

Updated

7 years ago
Duplicate of this bug: 759222
We talked about this today in triage. We could expire schedulers with last_processed data which refers to a change which is older than some threshold (a week, a month). It could be also be limited by scheduler class to avoid accidental damage. As the state column is json data it would make sense to do the cleanup as python script rather than bare SQL.
Blocks: 617017
(Assignee)

Updated

7 years ago
Priority: -- → P2
(Assignee)

Updated

6 years ago
Duplicate of this bug: 786226
(Assignee)

Comment 4

6 years ago
I took a stab at this today.

http://cruncher.build.mozilla.org/~catlee/old_schedulers.txt is the result of all schedulers with:
- last_build < cutoff time
or
- date of last_processed change < cutoff time

I chose 2 weeks as the cutoff.

I don't think any of these are active schedulers.
(Assignee)

Comment 5

6 years ago
hmm...
to-delete (old build) mozilla-central pgo {u'last_build': 1338836404.9286871, u'last_buildid': u'20120604120004', u'last_processed': 787419}

isn't that still active? why isn't last_build or last_processed updated?
(Assignee)

Comment 6

6 years ago
(In reply to Chris AtLee [:catlee] from comment #5)
> hmm...
> to-delete (old build) mozilla-central pgo {u'last_build':
> 1338836404.9286871, u'last_buildid': u'20120604120004', u'last_processed':
> 787419}
> 
> isn't that still active? why isn't last_build or last_processed updated?

Ah, we have two schedulers with that name, different classes:

mysql> select * from schedulers where name = 'mozilla-central pgo';                                                                                                                                                                                                             
+-------------+---------------------+------------------------------------------------+------------------------------------------------------------------------------------------------+
| schedulerid | name                | class_name                                     | state                                                                                          |
+-------------+---------------------+------------------------------------------------+------------------------------------------------------------------------------------------------+
|        3382 | mozilla-central pgo | buildbotcustom.scheduler.Nightly-props         | {"last_build": 1338836404.9286871, "last_buildid": "20120604120004", "last_processed": 787419} | 
|        4296 | mozilla-central pgo | buildbotcustom.scheduler.SpecificNightly-props | {"last_build": 1368104401.524662, "last_buildid": "20130508120241", "last_processed": 1363805} | 
+-------------+---------------------+------------------------------------------------+------------------------------------------------------------------------------------------------+
(Assignee)

Comment 7

5 years ago
I deleted a bunch of old schedulers this morning. So far no fallout detected.
Product: mozilla.org → Release Engineering
(Assignee)

Comment 8

5 years ago
Created attachment 8343151 [details] [diff] [review]
cleanup_db-schedulers.diff

clean up schedulers we know how to

many schedulers don't have enough state in the db to determine if they can be deleted or not (e.g. nightly schedulers). you'd need to know which branches are active, or inspect buildbot state or something.

this patch will cleanup up the worst offenders at least, which are the regular test schedulers.
Attachment #8343151 - Flags: review?(nthomas)
(Assignee)

Comment 9

5 years ago
Created attachment 8343152 [details]
cleanup.log

list of schedulers it would delete now
Comment on attachment 8343151 [details] [diff] [review]
cleanup_db-schedulers.diff

lgtm, the log was helpful.
Attachment #8343151 - Flags: review?(nthomas) → review+
(Assignee)

Updated

5 years ago
Attachment #8343151 - Flags: checked-in+
(Assignee)

Comment 11

5 years ago
Created attachment 8345891 [details] [diff] [review]
pass schedulerdb info to cleanup_db
Attachment #8345891 - Flags: review?(dustin)
Attachment #8345891 - Flags: review?(dustin) → review+
(Assignee)

Updated

5 years ago
Attachment #8345891 - Flags: checked-in+
(Assignee)

Updated

5 years ago
Assignee: nobody → catlee
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Is this on both production *and* stage?
(Assignee)

Comment 13

5 years ago
No, just for production.
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.