1241869 - Read crashes from 2 S3 buckets

Status: RESOLVED INVALID

(Socorro :: General, task)

Description

[Peter Bengtsson [:peterbe]]

We should always use a fallback crash storage for READING. One primary bucket name and one secondary. 

The places where we read crashes (raw or processed) are:

* processors
* crashsubmitter
* correlations
* crashdata

All of these places need to use the FallbackCrashStorage class [0]. 

Note that this class will be doing the fallback pattern for both READS and WRITES. We might need to think about letting it WRITE to the secondary. If we're trying to get away from using the secondary S3 bucket, and a write fails to the primary (because of a temporary network problem) we don't necessarily want to let it slip into the secondary S3 bucket. 

Writing a wrapper class on this one called FallbackReadCrashStorage should be relatively straight forward. 

[0] https://github.com/mozilla/socorro/blob/5d56b484473db866b9e98a3091bdec0ae80535bc/socorro/external/crashstorage_base.py#L602

Comment 1

[Peter Bengtsson [:peterbe]]

Lars, in irc you said the places where we read are processors, crashsubmitter and correlations. I added crashdata (e.g. report index). Can you think of any other places?
Also, what do you think about the idea of a FallbackReadCrashStorage sub-class whose save_* methods use those in FallbackCrashStorage's PARENT class?

Comment 2

[K Lars Lohn [:lars] [:klohn]]

crashsubmitter doesn't need to read from the old S3 as it only deals in new crashes.  however, it would harm nothing to give it the capability.

right now we've got app based prefixes like this:
    envconsul -prefix socorro/common -prefix socorro/processor ...

it might be useful to add another prefix called socorro/common_source. you can put all the config for the source fallback there and then give that to all the socorro apps that use it:
     envconsul -prefix socorro/common -prefix socorro/processor -prefix socorro/common_source   
          [processor|submitter|correlations|crashdata]
that way you only have to deal with this config in one place rather than in each app's config

As for the FallbackCrashStorage question, that would not work.  the parent class is the base class that effectively doesn't implement the save methods (silent do nothing, actually).  The right thing to do is to reimplement FallbackCrashStorage with the behavior that you want, or just make a new class entirely.

Comment 3

[Peter Bengtsson [:peterbe]]

Let's abandon ship! 
Instead we're going to work on https://bugzilla.mozilla.org/show_bug.cgi?id=1242024 which is order of magnitude simpler.