1572154 - Reduce data retention of stage and dev to 1 month

Status: RESOLVED INVALID

(Tree Management :: Treeherder: Data Ingestion, enhancement, P2)

Description

[Armen [:armenzg]]

We don't believe there's value and we would run out of space in H2 at the current space.

We might want to refresh the DB from a snapshot before doing so.

igoldan, since you look at the code most recently, would you be able to give some direction? No need to tackle yourself if you're not up to it but any help would be appreciate it.

Comment 1

[Ionuț Goldan [:igoldan]]

(In reply to Armen [:armenzg] from comment #0)

igoldan, since you look at the code most recently, would you be able to give some direction? No need to tackle yourself if you're not up to it but any help would be appreciate it.

I can provide some guidance in terms of the cycle_data Django management script.
When expiring Treeherder data, you can set --days 31 .

But if you want to expire Perfherder data too, I'd recommend some extra precautions first.
At the moment, you cannot run something like newrelic-admin run-program ./manage.py cycle_data --days 31 --chunk-size 1000 from:perfherder, as I made sure one cannot simply set the script to expire data more recent than a year.

You should first add an extra --force flag here, at its subparser. Use it to condition setting a --day option less than 365 days.

After that, you can configure the Heroku Scheduler to run newrelic-admin run-program ./manage.py cycle_data --days 31 --chunk-size 1000 from:perfherder --force.

If you do these Heroku Scheduler changes on treeherder-prototype & treeherder-staging, I think it'll address the problem.