1152747 - [mbdb] management command to implement build data retention policy

Status: RESOLVED FIXED

(Webtools Graveyard :: Elmo, defect)

Description

[Axel Hecht [:Pike]]

We need a way to get rid of "old" build data, so that we can safely mount the latest status.

I checked in what buildbot's standard buildHorizon and logHorizon do, and they're sadly bound by numbers. Which is bad for merge day, when we just have a lot of current data.

Thus we should have a retention policy based on dates, and I think we can safely do:

- throw files away after a day
- throw mbdb data away after a week

This doesn't touch data in l10nstats nor ES, so that stuff is all still long-term.

Caveat: we need to keep the last build around, so that BuilderStatus.determineNextBuildNumber still works.

This blocks better infrastructure setup, best represented by bug 1147816.

Taking, as I have WIP.

Comment 1

[Axel Hecht [:Pike]]

Attachment: make it so

Comment 2

[[github robot]]

Commit pushed to develop at https://github.com/mozilla/elmo

https://github.com/mozilla/elmo/commit/003fbec2619ea70e7a0e73e5698d24c6856433c2
bug 1152747, management command for build data retention policy, r=rhelmer

We keep 7 days of build data, and 1 day of log files.
Also keep the data for the latest build for each builder, as buildbot
uses that for startup.

Comment 3

[[github robot]]

Commit pushed to develop at https://github.com/mozilla/elmo

https://github.com/mozilla/elmo/commit/ef1f39e2cdc92c33383a2377addfe090f575170a
Following up, fixing build-retention --dry-run --backup=foo

The original command is in bug 1152747.

Comment 4

[Axel Hecht [:Pike]]

This is in prod, and now chucking along on the disk and the db.