1336265 - switch antenna -stage to use socorro -stage s3 bucket [antenna]

Status: RESOLVED FIXED

(Socorro :: Antenna, task)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

In order to tie Antenna -stage to Socorro -stage, we need them both to use the same S3 bucket.

This bug covers making the necessary changes for that.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Making this an infra bug.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

I actually don't know whether we should:

a. Change Antenna -stage to use the existing Socorro -stage s3 bucket, OR

b. Set up some kind of forwarding or copying or something from the Antenna -stage s3 bucket to the Socorro -stage s3 bucket, OR

c. Some third option?

Whatever solution we pick, the Socorro -stage system needs to be able to find raw crashes without us reconfiguring it. Are there other requirements?

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Lonnen, Miles and I talked about this at length.

We have two buckets:

* s3 bucket in the Antenna environment AWS account (hereafter referred to as the Antenna bucket)
* s3 bucket in the Socorro AWS account (hereafter referred to as the Socorro bucket)

We decided to use the Antenna bucket. Ultimately, we want to get out of the Socorro AWS account, so this moves us in that direction.

We want to minimize code changes to Socorro required because code changes will probably be messy.

Options:

* If we could copy everything from the Socorro bucket to the Antenna
  bucket and then switch everything over to the Antenna bucket, that’d
  be cool.

* If AWS can do some kind of “your object is in another bucket” magic
  passthrough redirect thing such that we can save to Antenna s3
  bucket and the processor and webapp don’t require configuration
  or code changes, that’d be cool. 

* If we could copy objects from the Antenna S3 bucket to the Socorro
  S3 bucket, then that’d be cool. Could do this with an AWS Lambda
  thing. Then Pigeon would sit in the Socorro infra and listen to the
  Socorro S3 bucket.

We decided to talk to an AWS solutions architect to see which of those is possible and how and all that.

Next steps:

1. Miles will talk to AWS solutions architect and find out whether:

   Is there a good way to copy everything from Socorro bucket to
   Antenna bucket?

   If not, can we magically redirect GET requests from Antenna bucket
   to Socorro bucket?

2. Wait for answer. Figure out what we need to implement then.


Assigning this to Miles for the next steps.

Comment 4

[Miles Crabill [:miles]]

I conferred with my team and have received a couple opinions back. If we have the option of waiting for data retention time (6mo) then we gain a lot of flexibility.

Below is an excerpt of our perceived options from what I sent to AWS support:

1. copy the complete contents of the (many terabyte) original bucket to the new bucket, use the new bucket as the source of truth going forward

2. have some sort of passthrough wherein references to keys in the old bucket that do not exist are automatically resolved to the new bucket (similar to how 404s can work in S3 websites, but without S3 website mode)

3. save everything going forward to both buckets, wait for our data retention policy to time-out, switch over to the new bucket

4. point the new service at the old bucket, continue using the old bucket as the source of truth

I think that a combination of 3+4 could be a viable option as well if we have the time to spare:
  - enable S3 replication [1] from the current Socorro bucket to the new bucket
  - point Antenna to save crashes to the old bucket in Socorro infra
  - wait 6-9 months, now all of our data is in the new location
  - switch over Socorro apps to new bucket
    - NB: by this point the contents of both buckets should be completely mirrored, so swapping buckets should have few / no repercussions outside of terrible race conditions related to mirroring

NB: using this plan we can still migrate the Socorro apps in their current state from mozilla-webeng infra to cloudops infra so long as we keep them pointing at the old bucket.

[1] http://docs.aws.amazon.com/AmazonS3/latest/dev/crr-walkthrough-2.html

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

+1 to doing 3 + 4. I'll update the Antenna "to -prod" plan accordingly.

For 3, we can start that anytime. We should probably spin that off into a separate bug. Want me to write up an infra bug for it?

For 4, can you (between loadtests) reconfigure Antenna -stage to point to the Socorro S3 bucket?

Comment 6

[Miles Crabill [:miles]]

If we reconfigure antenna to point to the socorro bucket now socorro stage end up processing a bunch of antenna loadtest crashes, right?
This is a very minor configuration change that we can do at any time, so I think it can wait.

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

I think we can change Antenna -stage to point to the Socorro -stage bucket--that'll be fine and won't affect processing in Socorro -stage at all. I think this problem comes up only if we've also got Pigeon running.

But Pigeon isn't running now, right?

Comment 8

[Will Kahn-Greene [:willkg] ET needinfo? me]

We've got Antenna using the webeng bucket for Socorro -stage and tied into Socorro -stage. So we're all set here and I'm marking as FIXED.

Comment 9

[Will Kahn-Greene [:willkg] ET needinfo? me]

Switching Antenna bugs to Antenna component.