1117915 - Audit codebase for use of reports, reports_clean

Status: RESOLVED FIXED

(Socorro :: Database, task)

Description

[Selena Deckelmann :selenamarie :selena]

Goal is to see if we can truncate these tables to within 2 weeks or so.

Comment 1

[Robert Kaiser]

We agreed that data in the reports table will never be deleted, I think. Truncating it would be a very huge change of our retention policies and would reduce our ability to do analysis.

Comment 2

[Selena Deckelmann :selenamarie :selena]

(In reply to Robert Kaiser (:[e-mail]) from comment #1)
> We agreed that data in the reports table will never be deleted, I think.
> Truncating it would be a very huge change of our retention policies and
> would reduce our ability to do analysis.

Kairo -- did you mean the matview tables as opposed to our incoming raw crash data? the 'reports' table falls under our regular expiration policy which is currently 6 months: 

https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=5734601#crash-stats.mozilla.com%28Socorro%29-DataExpirationPolicy

We will move to a way of fetching this data from another source outside of our primary Postgres database, rather than eliminate access to all raw crash data.

The "Crashes/ADU" means "crashes per adu". I'm going to update that page to be more explicit about what we are maintaining and which data store it involves.

Comment 3

[Selena Deckelmann :selenamarie :selena]

Unblocking the Q1 goal as we're already under the targeted size.

Comment 4

[Robert Kaiser]

(In reply to Selena Deckelmann :selenamarie :selena from comment #2)
> the 'reports' table falls under our regular expiration policy
> which is currently 6 months: 


Then we should stay with those 6 months if in any way possible. We surely should have more than 2 weeks available via the APIs and in the UI, we need to be able to dig *at least* 6 months back into data of crashes on a regular basis.
In the DB, we may not care once we do not have anything outside of actual Socorro code accessing the DB directly, but that is still some time off - and even then, we definitely need to be able to dig at least up to a few months back in reasonable speed via APIs and the UI.

Comment 5

[Selena Deckelmann :selenamarie :selena]

This ticket can be closed as the tables have been investigated and I found enough space in other locations to not have to reduce reports any further this quarter.