Socorro apps should be able to use AWS IAM roles for credentials

RESOLVED WORKSFORME

Status

Socorro
General
RESOLVED WORKSFORME
3 months ago
3 months ago

People

(Reporter: miles, Assigned: willkg)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

3 months ago
Currently the Socorro apps need AWS access key configuration. We should move away from this practice.
The longer description of this is that the boto ConnectionContext that we use for Socorro needs to support AWS credentials whether supplied in the environment (what we're currently doing) or bound to the node which I don't think it supports right now.

Antenna can do both, so I think I can crib from Antenna to get it to work.

I'm going to try to do this in the next week.
Status: NEW → ASSIGNED
I found the PR that I remember that addressed some/most of this:

https://github.com/mozilla-services/socorro/pull/3621

That PR was merged, so theoretically things like crontabber and the processor should work fine with IAM roles.

In the PR comments, we talked about two things:

First, I had no way to test that it works with IAM roles. Theoretically, the contributor tested it out and is using it now, so it's probably fine?

Second, we talked about is how not all things use ConnectionContextBase, so I'm going to go through the codebase and see what wasn't covered by that PR. We can figure out what to do then.
There are a few things in the webapp that use boto directly. The settings suggest that AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY both default to None, so it's possible that everything could just work. I can't verify it without setting up a node in AWS.

Miles: I think it's possible everything might just work. I think we should try setting it up and when we run into problems, we'll have a system to test with.

For now, I'm going to mark this as WORKSFORME.
Status: ASSIGNED → RESOLVED
Last Resolved: 3 months ago
Resolution: --- → WORKSFORME
For archival purposes so I don't have to hunt for these again....

The ConnectionContextBase configuration settings default to None:

https://github.com/mozilla-services/socorro/blob/d8080a5e45909fbe7d0ed415d49bd23e1794939c/socorro/external/boto/connection_context.py#L123


The webapp settings all default to None:

https://github.com/mozilla-services/socorro/blob/d8080a5e45909fbe7d0ed415d49bd23e1794939c/webapp-django/crashstats/settings/base.py#L600

https://github.com/mozilla-services/socorro/blob/d8080a5e45909fbe7d0ed415d49bd23e1794939c/webapp-django/crashstats/settings/base.py#L635

https://github.com/mozilla-services/socorro/blob/d8080a5e45909fbe7d0ed415d49bd23e1794939c/webapp-django/crashstats/settings/base.py#L536
I too believe this is done work. 

The env key `socorro/common/resource.boto.access_key` (for example) is used by the webapp, the processor and crontabber (https://github.com/mozilla-services/socorro/blob/master/socorro/cron/jobs/missingsymbols.py and https://github.com/mozilla-services/socorro/blob/master/socorro/cron/jobs/upload_crash_report_json_schema.py). 

However, the S3 access used by the webapp for uploading symbols is using `socorro/webapp-django/AWS_ACCESS_KEY`. And as we, piecemeal, migrate symbols and then migrate crashes another day, I think we can 1) set up IAM roles and 2) remove all `socorro/common/resource.boto.access_*` env vars from consulate and 3) test. All without losing the ability to redirect symbol uploads to a different S3 bucket.
Duplicate of this bug: 1213428
You need to log in before you can comment on or make changes to this bug.