This is the tale of crashmover getting slower and slower over a one week period.
This problem lives in the File System codebase circa 2008. In the file system storage, there is the name and date branches. Date branch provides an time based index of crashes while the name branch stores the actual crashes themselves. The date index is strange, though, it is a rare case of read once storage. The act of reading it, destroys it. This is the desired behavior because it guarantees that a crash will not be referenced twice - only new crashes that have never been seen before are discovered. Deep within the file system classes, there are two delete functions, one called 'quickDelete' and the default one called 'remove'. The selection of which one to use depends on if the date branch has an index for the crash. The 'quickDelete' is for when the crash has been seen and the date index for it was destroyed. The default 'remove' is thorough and does a needless exhaustive scan for a date index entry when a crash is deleted.
The multidump code used the default 'remove' function because it is written to be generic and therefore doesn't know the context. It takes the safe and thorough path. The old crashmover code knows that it is safe to use the quick delete and so can be very efficient.
the ultimate solution is to throw this codebase into the nearest active volcano and start using the the replacement written by our intern, rfw. But since that is not yet ready, then I will patch this ancient code with a healing hack and hope to never see it again...
Commit pushed to master at https://github.com/mozilla/socorro
Merge pull request #1073 from twobraids/Bug836986-fs-delete
Fixes Bug 836986 - call the right delete in file system storage
Needs to be in stage branch so we can push it.
This moved to stage yesterday, will be released today.
This is on stage but was not released due to instability in our production postgres. It will go out in the next release, though.