526345 - Configure socorro to keep at least 7 days of raw dumps for processed reports

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Michael Morgan [:morgamic]]

Right now after a crash report is processed we keep the output but discard the raw dump.  Engineering has requested that we keep those for 7 days.

Questions:
1. How much disk capacity would it take to keep raw dumps along with the gz for processed reports?
2. How much space do we have, and how much could we realistically store
3. How much would it cost to get more space and store all raw dumps for all processed reports?

Damon, all - please add anything I'm missing.

Comment 1

[Justin Dolske [:Dolske]]

I'll just add that my reasoning behind this is that if we need to look at the raw dump in a debugger to gather additional information for a type of crash, we're out of luck unless the crash is ongoing. So, if we see a weird spike that goes away (eg, caused by a broken website/addon which is subsequently fixed), we're out of luck unless we detect and investigate it before the raw dumps for the event get purged.

The simple solution would be to keep all (processed) raw dumps for 2-3 weeks, or whatever's economical. I think aravind said we process ~20GB of dumps per day right now.

It might be nice to store them even longer, but that would probably require a more complex solution to only retain some representative sample of dumps.

Comment 2

[Aravind Gottipati [:aravind]]

We are currently keeping 7 days of crash dumps (mounted on cm-fs01).  There seems to be quite a bit of variation in the partition sizes.  The biggest right now is 37GB.  Lets say that goes up to 40GB per day (in the short term).  Using that, it works out to about 1.2TB for a month of raw crash dumps (that have been processed).  I think the latest estimate for how long we want to keep these crash dumps is currently at 2 years (the duration of a typical firefox release cycle).  So going by that, we'd need nearly 30 TB of disk space.

If we need to keep more than the current 1 week of dumps around, let me know and I will adjust my cleanup and storage scripts accordingly.

Comment 3

[matthew zeier [:mrz]]

Pulling Daniel in - does Hadoop fix any of this?

Comment 4

[Daniel Einspanjer [:dre] [:deinspanjer]]

Yes, the system we are designing is definitely being purpose built to solve this particular problem.  The only important thing to consider is that the goal for this quarter is strictly having a proof of concept solution, not one that would be ready for replacing production systems by the end of the month.

Comment 5

[matthew zeier [:mrz]]

Great.  So this looks done - we're keeping 7 days now and long-term the hadoop cluster will magically grow to handle everything we could ever need!