1432884 - [ops infra socorro] Crontabber uses the wrong S3 endpoint / region

Status: RESOLVED FIXED

(Socorro :: General, task, P1)

Description

[Miles Crabill [:miles]]

The crontabber in -new-prod doesn't use the correct S3 endpoint for the telemetry bucket.

It has a connection context: socorro.external.boto.connection_context.S3ConnectionContext whose endpoint is s3.amazonaws.com, which only works for the us-east-1 region. [0]

We don't have this issue with the processor. Maybe the connection context is being created without a region?

[0] https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Grabbing this one. I like Miles' theory. I'll look into that first.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

The crontabber job in question is upload_crash_report_json_schema. The error I see locally is this one:

crontabber_1     | 2018-02-13 19:01:25,967 DEBUG - crontabber -  - MainThread - error when running <class 'socorro.cron.jobs.upload_crash_report_json_schema.UploadCrashReportJSONSchemaCronApp'> on None
crontabber_1     | Traceback (most recent call last):
crontabber_1     |   File "/usr/local/lib/python2.7/site-packages/crontabber/app.py", line 1053, in _run_one
crontabber_1     |     for last_success in self._run_job(job_class, config, info):
crontabber_1     |   File "/usr/local/lib/python2.7/site-packages/crontabber/base.py", line 189, in main
crontabber_1     |     function()
crontabber_1     |   File "/usr/local/lib/python2.7/site-packages/crontabber/base.py", line 259, in _run_proxy
crontabber_1     |     return self.run(*args, **kwargs)
crontabber_1     |   File "/app/socorro/cron/jobs/upload_crash_report_json_schema.py", line 74, in run
crontabber_1     |     bucket = connection_context._get_bucket(connection, bucket_name)
crontabber_1     |   File "/app/socorro/external/boto/connection_context.py", line 220, in _get_bucket
crontabber_1     |     self._bucket_cache[bucket_name] = conn.get_bucket(bucket_name)
crontabber_1     |   File "/usr/local/lib/python2.7/site-packages/boto/s3/connection.py", line 509, in get_bucket
crontabber_1     |     return self.head_bucket(bucket_name, headers=headers)
crontabber_1     |   File "/usr/local/lib/python2.7/site-packages/boto/s3/connection.py", line 553, in head_bucket
crontabber_1     |     raise err
crontabber_1     | S3ResponseError: S3ResponseError: 404 Not Found

However, I think that's because "make dockersetup" isn't creating the bucket.

I'll fix that first, then look into regions.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

I don't see us setting the resource.boto.resource_class in -prod so we use the defaults.

In the local dev environment, we set it to HostPortS3ConnectionContext because we're using a localstack s3.

The processor works in all environments. It uses BotoS3CrashStorage. The default resource_class for that is set in code here:

https://github.com/mozilla-services/socorro/blob/9030ee13662c80cabfa552e24b7b5b82a29c4398/socorro/external/boto/crashstorage.py#L290

However, the crontabber job uses a different default set here:

https://github.com/mozilla-services/socorro/blob/9030ee13662c80cabfa552e24b7b5b82a29c4398/socorro/cron/jobs/upload_crash_report_json_schema.py#L33

I'm going to change that default to socorro.external.boto.connection_context.RegionalS3ConnectionContext. Pretty sure that should fix it.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 4338: fixes bug 1432884 - upload_crash_report_json_schema s3 fix

Comment 5

[Miles Crabill [:miles]]

Yay, this is what I was thinking when I looked at this.

We're setting resource.boto.region which is correct to override region for the RegionalS3ConnectionContext, correct?

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

Yes. And the reason it's being ignored right now is because we're using the wrong connection context. Once this lands, it should work out fine ... or it'll still be broken, but for a different reason.

Comment 7

[[github robot]]

Commits pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/9d82d4a2e24db6aae365521acc89c2f74a44ea0e
bug 1432884 - create telemetry bucket in local dev environment

https://github.com/mozilla-services/socorro/commit/e3775e0125bc758fe4f88ed1ea445ff88f77052c
fixes bug 1432884 - change boto default for upload_crash_report_json_schema

This changes the upload_crash_report_json_schema crontabber job to use
RegionalS3ConnectionContext which is the same default we use for
TelemetryBotoS3CrashStorage (which derives from BotoS3CrashStorage)
which is what we're using in the processor.

https://github.com/mozilla-services/socorro/commit/43c4e467f1fff3c53bf2ec3dab9a763a6fe645e9
Merge pull request #4338 from willkg/1432884-tele-bucket

fixes bug 1432884 - upload_crash_report_json_schema s3 fix