1075700 - encrypt/decrypt crash blobs in primary store

Status: RESOLVED WONTFIX

(Socorro :: Backend, task)

Description

[Lonnen :lonnen]

As an added layer of protection, we should encrypt and decrypt crash blobs before putting them on disk.

This could be accomplished with a metacrashstorage class.

It is prudent to implement this for s3, but I'm not sure if its worth using with hbase.

Comment 1

[Daniel Maher [:phrawzty]]

What does this look like in the context of the larger system ?  In particular, where is the key (or equivalent, depending on what encryption mechanism is used) stored ?  I'm guessing it'd be somewhere on our internal infra, meaning that we'd encrypt the blobs here, then transfer them over to Amazon for storage.

Elucidation ?

Comment 2

[Daniel Maher [:phrawzty]]

06:25:45 <@lars> It would be implemented as a meta-crashstore in the same manner as the benchmarking crashstore.
06:27:57 <@lars> it accepts the same api as the other crashstores, but holds a reference to a subsidiary crashstore.  On receiving a crash via the save_* method, it does whatever the encryption step ends up being, then passes the result on to the subsidiary crashstore
06:28:46 <@lars> implemented like that, it can be add via configuration to _any_ other crashstore that we've got.

Comment 3

[Lonnen :lonnen]

bcrypt in metacrashstorage class with a secret in the config will work fine. The collector and processor and mware will all need to be able to use it, since all three read or write crashes.

We want to encrypt raw+processed crashes not just before they get written down, but also before they are transmitted between data centers.

Comment 4

[Lonnen :lonnen]

First -- I was wrong about bcrypt and kind of hand wavy about the whole thing. We can use ssl for upload and download from s3. I think we can downgrade this from 'need' to 'nice to have' iff boto validates SSL.

We should look into s3 access control and maybe we have to run a proxy in AWS until the rest of the services move, just so the primary store isn't open to the world.

Comment 5

[Robert Helmer [:rhelmer]]

(In reply to Chris Lonnen :lonnen from comment #4)
> First -- I was wrong about bcrypt and kind of hand wavy about the whole
> thing. We can use ssl for upload and download from s3. I think we can
> downgrade this from 'need' to 'nice to have' iff boto validates SSL.
> 
> We should look into s3 access control and maybe we have to run a proxy in
> AWS until the rest of the services move, just so the primary store isn't
> open to the world.

We should be able to lock the bucket down using ACLs, to restrict read access to certain AWS IDs.

There are also "policies" that would allow us to restrict to IP, which might be a reasonable additional measure (that way the AWS ID+key leaking wouldn't lead to disclosure unless the IP was also controlled):

http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html

Comment 6

[Robert Helmer [:rhelmer]]

Might be nice to do still, per bug 1096510 we have verified that over-the-wire encryption is good. So, leaving open but not blocking.

Comment 7

[Robert Helmer [:rhelmer]]

Not currently working on this but I suggest looking into Amazon's KMS instead of doing a custom implementation.

Comment 8

[Will Kahn-Greene [:willkg] ET needinfo? me]

This hasn't been touched in 6 years. I understand the use case and it would be nice to have, but the buckets we store crash data in are pretty restricted so I'm not sure the value here is worth the effort.

I'm going to close it as WONTFIX. We can revisit if a compelling reason comes up.