1240017 - Read from a LIST of S3 buckets

Status: RESOLVED WONTFIX

(Socorro :: General, task)

Description

[Peter Bengtsson [:peterbe]]

At some point we need to move all our raw crashes from one bucket to another (a new one that doesn't have dots in the name). During that transition time (the slow sync) we're going to have some raw crashes in the new bucket and some in the old since we're going to change the bucket that collector (or crashmover) writes to. 

So, when downloading a raw crash we should have the capability to read from a LIST of buckets instead. That way it'll try the first and not raise a CrashIDNotFound until it has tried all buckets.

Comment 1

[Peter Bengtsson [:peterbe]]

Lars, 
Does this make sense? Perhaps we need to do the same with reading processed crashes too. I'm in particular thinking about socorro.external.boto.crash_data

Comment 2

[Peter Bengtsson [:peterbe]]

There is already a crashstorage class we can use https://github.com/mozilla/socorro/blob/cb73ba18eef033f3d18eac151eef65354b45ba07/socorro/external/crashstorage_base.py#L602

Comment 3

[K Lars Lohn [:lars] [:klohn]]

the FallbackCrashStorage class doesn't implement the 'get*' methods as it was intended for save operations only.  To use it as you suggest, the "get*" methods will have to be added.

Comment 4

[Peter Bengtsson [:peterbe]]

(In reply to K Lars Lohn [:lars] [:klohn] from comment #3)
> the FallbackCrashStorage class doesn't implement the 'get*' methods as it
> was intended for save operations only.  To use it as you suggest, the "get*"
> methods will have to be added.

What's this then?
https://github.com/mozilla/socorro/blob/cb73ba18eef033f3d18eac151eef65354b45ba07/socorro/external/crashstorage_base.py#L697-L753

Comment 5

[K Lars Lohn [:lars] [:klohn]]

well, i'm obviously working from faulty memory.

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

At some point, we're going to switch S3 buckets, but all the good plans I've seen so far deal with the complexity via infrastructure and not via code changes like this one.

Given that, I'm going to WONTFIX this. If it turns out we really do need to make big changes like this, then we can re-open.