533613 - Socorro - simplifying opportunity

Status: RESOLVED WONTFIX

(Socorro :: General, task)

Description

[K Lars Lohn [:lars] [:klohn]]

simplifying opportunity.  Let's consider eliminating the
concept of successful (and failed) storage entirely.  Rather than moving the
data, we should consider just making the original standard storage directories
available as the raw file storage in the same way that we make the successful
storage directory available now.  After a week, the older date trees can be
deleted.

That would have some benefits:
1) simplification of monitor as we could possibly eliminate the cleanup tread
entirely
2) eliminate the overhead of the NFS traffic created by all this reading,
writing and shuffling files around.
3) make raw dumps available instantly for download after (or even before)
processing
4) make failed dumps available for forensics

Comment 1

[Aravind Gottipati [:aravind]]

I think this needs some code changes before we can test it in staging?

Comment 2

[matthew zeier [:mrz]]

Lars?

Comment 3

[Austin King [:ozten]]

(In reply to comment #2)
Let's move this to our next release and we need to spec out the code changes.

Comment 4

[K Lars Lohn [:lars] [:klohn]]

Yes, there needs to be some changes to processor to implement this.  The clean up thread needs to have the code that saves or deletes completed jobs needs to be disables.  The task of cleaning up the 'jobs' table needs to be preserved, though.

Because of recent surgery, I am on medical leave of absence. I will not be returning until sometime in February.  This bug either needs to be postponed until my return or it needs to be taken over by another person.

Comment 5

[K Lars Lohn [:lars] [:klohn]]

the ability construct coherent sentences is the first skill to go...

Comment 6

[Austin King [:ozten]]

Releasing bug back to the pool.

Comment 7

[K Lars Lohn [:lars] [:klohn]]

This is likely to be superseded by the hbase stuff...  I think we can drop it.