1311522 - Configure Telemetry S3 crash storage on prod

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Peter Bengtsson [:peterbe]]

+++ This bug was initially created as a clone of Bug #1295559 +++

The bucket is called: org-mozilla-telemetry-crashes
(https://console.aws.amazon.com/s3/home?region=us-west-2#&bucket=org-mozilla-telemetry-crashes&prefix=)

The crashstorage class is: TelemetryBotoS3CrashStorage
https://github.com/mozilla/socorro/blob/18211427819111827a50c5e1912cdc1baff94c15/socorro/external/boto/crashstorage.py#L275

Comment 1

[[DEACTIVATED] Adrian Gaudebert]

[centos@prod-admin-i-ecb76837 ~]$ consulate kv get socorro/processor/destination.storage_classes
socorro.external.postgresql.crashstorage.PostgreSQLCrashStorage, socorro.external.es.crashstorage.ESCrashStorageRedactedJsonDump, socorro.external.boto.crashstorage.BotoS3CrashStorage, socorro.external.statsd.crashstorage.StatsdCrashStorage


# Set the new storage4 keys. 
$ consulate kv set socorro/processor/destination.storage4.crashstorage_class 'socorro.external.statsd.statsd_base.StatsdBenchmarkingWrapper'
$ consulate kv set socorro/processor/destination.storage4.wrapped_object_class 'socorro.external.boto.crashstorage.TelemetryBotoS3CrashStorage'
$ consulate kv set socorro/processor/destination.storage4.statsd_prefix 'processor.telemetry'
$ consulate kv set socorro/processor/destination.storage4.bucket_name 'org-mozilla-telemetry-crashes'
$ consulate kv set socorro/processor/destination.storage4.active_list 'save_raw_and_processed'

# Enable the new destination.
$ consulate kv set socorro/processor/destination.storage_classes 'socorro.external.postgresql.crashstorage.PostgreSQLCrashStorage, socorro.external.boto.crashstorage.BotoS3CrashStorage, socorro.external.es.crashstorage.ESCrashStorageRedactedJsonDump, socorro.external.statsd.crashstorage.StatsdCrashStorage, socorro.external.boto.crashstorage.TelemetryBotoS3CrashStorage'

Note that in that list, I have change the order of classes. It doesn't matter, since these values are actually not being used. The only that matters with that key is the number of destinations, everything else will be decided inside the storageX namespace. I changed the order simply to reflect the actual order of those storages (storage1 is S3 and storage2 is ES, as opposed to what is currently defined in that key). 

Peter, I don't feel confident doing that when I'm the lone worker and it's Friday afternoon for me, so I'm leaving it to you to apply these changes. All you have to do is apply them in this order. If anything goes wrong, you just have to run this to roll back:

$ consulate kv set socorro/processor/destination.storage_classes 'socorro.external.postgresql.crashstorage.PostgreSQLCrashStorage, socorro.external.boto.crashstorage.BotoS3CrashStorage, socorro.external.es.crashstorage.ESCrashStorageRedactedJsonDump, socorro.external.statsd.crashstorage.StatsdCrashStorage'

And then re-run the last command to re-enable it again.

Comment 2

[Peter Bengtsson [:peterbe]]

I too get nervous about these changes. Let's do them on Tuesday afternoon (your Wednesday morning). 
In fact, I'll make them and if things break by the time you're online we can fix it. On Tuesdays I have a bit more head room if I have to stay beyond working hours.

Comment 3

[Peter Bengtsson [:peterbe]]

It didn't work :(


Oct 26 20:52:44 prod-processor-i-2be58c33 bash: 2016-10-26 20:52:44,773 CRITICAL - processor -  - Thread-8 - Unrecoverable socorro.external.boto.connection_context transaction error
Oct 26 20:52:44 prod-processor-i-2be58c33 bash: Traceback (most recent call last):
Oct 26 20:52:44 prod-processor-i-2be58c33 bash: File "/data/socorro/socorro-virtualenv/lib/python2.7/site-packages/socorro-master-py2.7.egg/socorro/database/transaction_executor.py", line 114, in __call__
Oct 26 20:52:44 prod-processor-i-2be58c33 bash: result = function(connection, *args, **kwargs)
Oct 26 20:52:44 prod-processor-i-2be58c33 bash: File "/data/socorro/socorro-virtualenv/lib/python2.7/site-packages/socorro-master-py2.7.egg/socorro/external/boto/crashstorage.py", line 390, in _do_save_processed
Oct 26 20:52:44 prod-processor-i-2be58c33 bash: processed_crash_as_string
Oct 26 20:52:44 prod-processor-i-2be58c33 bash: File "/data/socorro/socorro-virtualenv/lib/python2.7/site-packages/socorro-master-py2.7.egg/socorro/external/boto/connection_context.py", line 246, in submit
Oct 26 20:52:44 prod-processor-i-2be58c33 bash: bucket = self._get_or_create_bucket(conn, self.config.bucket_name)
Oct 26 20:52:44 prod-processor-i-2be58c33 bash: File "/data/socorro/socorro-virtualenv/lib/python2.7/site-packages/socorro-master-py2.7.egg/socorro/external/boto/connection_context.py", line 391, in _get_or_create_bucket
Oct 26 20:52:44 prod-processor-i-2be58c33 bash: location=self._region,
Oct 26 20:52:44 prod-processor-i-2be58c33 bash: File "/data/socorro/socorro-virtualenv/lib/python2.7/site-packages/boto/s3/connection.py", line 590, in create_bucket
Oct 26 20:52:44 prod-processor-i-2be58c33 bash: response.status, response.reason, body)
Oct 26 20:52:44 prod-processor-i-2be58c33 bash: S3ResponseError: S3ResponseError: 403 Forbidden
Oct 26 20:52:44 prod-processor-i-2be58c33 bash: <?xml version="1.0" encoding="UTF-8"?>
Oct 26 20:52:44 prod-processor-i-2be58c33 bash: <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>06BEC1681826FB8A</RequestId><HostId>EhYalrfV80EITQ4fJsm+qfYgREDACTED</HostId></Error>

Comment 4

[Peter Bengtsson [:peterbe]]

JP stepped in and corrected the permissions and now it's filling up nicely.