314943 - Data retention policy and tools

Status: RESOLVED FIXED

(Webtools Graveyard :: Litmus, defect)

Description

[Chris Cooper [:coop] (he/him)]

We will need decide how much historical testing data we need to keep for Litmus, and the frequency with which we purge that data.

There are two components to be considered: database storage and on-disk log storage (if separate log storage is necessary).

The QA team has said that they rarely need to go back further that a month to compare results, but I see no reason to purge results older than that unless space becomes an issue. Even when space does become an issue, we will probably want to keep test run results associated with releases and perhaps purge only the random test runs/results.

We will need to monitor disk usage to determine what our daily/weekly/monthly storage requirements are and create a policy accordingly. This will also be influenced by whether we are end up running on the current server or integrating into the LVS cluster.

Comment 1

[Chris Cooper [:coop] (he/him)]

Since we're running on a shared db now, maybe this is a moot point? We should still decide when to purge out-of-date data though, or at least when to take it offline.

justdave: what sort of data backup is already in place on the database server? Can we get an idea of how much room we have to grow, i.e. current size of Litmus db vs. available size?

Comment 2

[Dave Miller [:justdave]]

Your live database is currently 23 MB.  We take daily backups, each backup (mysqldump file, gzip compressed) is about 1.3 MB.  Backups are kept for a year.

For comparison, the Bugzilla database is about 6 GB, and daily backups are about 2.5 GB compressed.  We haven't ever archived and deleted data out of Bugzilla, so that 6 GB contains all of the comments and attachments since 1997.

The shared DB server that you share has 335 GB total disk space, with 59 GB currently in use, leaving 259 GB free for future use.  The used disk space includes both the live databases and the last 3 days worth of backups.

I would say you have plenty of room to grow.

Comment 3

[Chris Cooper [:coop] (he/him)]

OK, I also take quasi-regular backups for my own sanity, but this is good to know.