996029 - resolve file system inode problem

Status: RESOLVED FIXED

(Socorro :: Backend, task)

Description

[K Lars Lohn [:lars] [:klohn]]

the current file system storage in use by the collectors leave a trail of empty dense directory trees behind it.  Each day gets its own tree of (256^2 + K) directories that do not get cleaned up.  This eventually eats up all the inodes in the file system.

The solution:

Rather than store each day with a root directory of YYYYMMDD, store instead as DD.  this means that at the end of each month, the directories automatically recycle.  This is a fine solution for temporary storage of crashes, but is not appropriate for permanent storage.  Make sure that this change leaves us with options for both temporary and permanent storage.

There will be three classes out of this effort:

FSTemporaryStorage - the directories recycle on a monthly basis with roots in the form 'DD', can be traversed in an FIFO order using the 'new_crashes' method.  Useful as temporary storage between the collector and the crashmover.
 
FSDatedPermanentStorage - the directories work as they do now with roots in the form 'YYYYMMDD' and can be traversed in a FIFO order using the 'new_crashes' method.  This is an alias for the current filesystem storage called FSLegacyDatedRadixTreeStorage.  Kept for legacy use only. 

FSPermanentStorage - the directories work as they do now with roots in the form 'YYYYMMDD' minus the ability to do a FIFO traversal.  This storage method is useful for installations that want only file system storage system rather than using HBase.

Comment 1

[[github robot]]

Commit pushed to master at https://github.com/mozilla/socorro

https://github.com/mozilla/socorro/commit/61907b8326aaea53a0ab8f2d95ac39ae5ed9c984
Merge pull request #1992 from twobraids/inode-joy

Fixes Bug 996029 - added temporary file system class that recycles directories