1378573 - Grant IAM permission to change parameter group on a new instance

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect)

Description

[Cameron Dawson [:camd]]

I admit, I'm a real n00b here.  I was created a new instance from a Prod snapshot.  I was trying to set the DB Parameter Group to "treeherder" and I got this error message:

User cdawson is not authorized to modify database instance arn:aws:rds:us-east-1:699292812394:pg:treeherder (Service: AmazonRDS; Status Code: 403; Error Code: AccessDenied; Request ID: 31227305-61dc-11e7-aa7e-fdcf6fe29777). Please check with your administrator.

However, it looks like our iam.tf file has "rds:ModifyDBParameterGroup" enabled as well as "rds:ModifyDBInstance" for "arn:aws:rds:${var.region}:${data.aws_caller_identity.current.account_id}:pg:treeherder-dev*",

I'm sure I'm not understanding this, because that would seem to indicate I should be able to make this change.

Ed, do you know what I'm doing wrong?

Comment 1

[Cameron Dawson [:camd]]

Jake-- Adding you in.  Do you know what needs to change so I can modify the db parameter group?

Comment 2

[Ed Morley [:emorley]]

IAM has a concept of a type of permission, and the name of the resource that permission is linked to.

Extracting from the error message in comment 0:
* Type of permission: "modify database instance" (which is "rds:ModifyDBInstance")
* Resource it is linked to: "pg:treeherder"

If you look at the iam permissions file:
https://github.com/mozilla-platform-ops/devservices-aws/blob/master/treeherder/iam.tf

...you can see "rds:ModifyDBInstance" has been applied to a number of resources. However the only `pg` resource listed is:
"pg:treeherder-dev*"

...which doesn't match.

As such, that entry needs to be changed to eg: "pg:treeherder*", however that would then grant all of the other permissions in that set to the main treeherder parameter group, such as "rds:ModifyDBParameterGroup", which would be bad, since we could accidentally change the parameter group for production.

Instead I'm guessing we need to create yet another IAM policy alongside the `AllowDevDBManage` one, which adds this new permission separately.

Comment 3

[Cameron Dawson [:camd]]

Attachment: PG PR

Hey Ed-- I hope I got this right.  :)

Comment 4

[Jake Watkins [:dividehex]]

I think this approach works but in the spirit of separating dev and prod, the dev instances should probably be using a separate (but identical) parameter group.  If it were named :treeherder-dev*, you'd have access to it already.  Any arguments against that approach instead?

Comment 5

[Ed Morley [:emorley]]

We deliberately want to be using the same parameter group for dev and prod, to avoid them diverging, so would prefer not to if possible? 

(Unless there's an easy way to get Terraform to inherit one resource to the other, but just give it a different name - that would also allow trialing new prefs on dev, before removing the subclass customisation etc)

Comment 6

[Jake Watkins [:dividehex]]

(In reply to Ed Morley [:emorley] from comment #5)
> We deliberately want to be using the same parameter group for dev and prod,
> to avoid them diverging, so would prefer not to if possible? 
> 
> (Unless there's an easy way to get Terraform to inherit one resource to the
> other, but just give it a different name - that would also allow trialing
> new prefs on dev, before removing the subclass customisation etc)

Ok. That's fine with me. :fubar might have better insight when he gets back from pto so we can always go back and change it.

Comment 7

[Cameron Dawson [:camd]]

Thanks!!

Comment 8

[Kendall Libby [:fubar] (he/him)]

(In reply to Cameron Dawson [:camd] from comment #0)
> I admit, I'm a real n00b here.  I was created a new instance from a Prod
> snapshot.  I was trying to set the DB Parameter Group to "treeherder" and I
> got this error message:

I think my biggest confusion here is, what was the PG set to originally? 

The IAM change also looks confusing, but I suspect that's more of an issue with IAM and me being rusty. My concern here is that it might allow changing the PG on all RDS instances you have access to.

Comment 9

[Ed Morley [:emorley]]

I don't think that's how IAM works though. An action type has to be linked to multiple resources, and in this case to modify the other instances the action would have to be associated with the `ri` resources, not the `pg` resources.

It's a shame AWS have made this all so hard to understand - no wonder there are supposedly so many orgs using insecure settings!

Comment 10

[Kendall Libby [:fubar] (he/him)]

(In reply to Ed Morley [:emorley] from comment #9)
> I don't think that's how IAM works though. An action type has to be linked
> to multiple resources, and in this case to modify the other instances the
> action would have to be associated with the `ri` resources, not the `pg`
> resources.

Sure, except the new stanza is combining the action "rds:ModifyDBInstance" with the resource(s) "pg:treeherder*". It's not explicitly stating WHICH RDS instances you're allowed to modify, just that you're allowed to change the PG. OTOH, the prior stanza only grants you ModifyDBInstance on "db:treeherder-dev*" so I'm not going to fret much. 

> It's a shame AWS have made this all so hard to understand - no wonder there
> are supposedly so many orgs using insecure settings!

Indeed.

Comment 11

[Ed Morley [:emorley]]

I agree, but I think that's the way it's meant to work.

Part of the confusion here is that the names are the same. If we instead had:
* DB instances: `db:database-prod`, `db:database-dev`
* Parameter groups: `rg:my-parameter-group`

Then giving "rds:ModifyDBInstance" to:
`db:database-dev*`
`rg:my-parameter-group`

...is much more clearly not allowing any production changes.

What's surprising to me at least, is *why* AWS make you give "rds:ModifyDBInstance" to `rg:my-parameter-group` in the first place. I would have thought the describe permission already given to the PG would be enough for a new snapshot to read from it, but that resulted in the error message in comment 0, which says that "rds:ModifyDBInstance" is required for the PG.

Comment 12

[Cameron Dawson [:camd]]

I'm deferring to Ed's explanation to answer this n-i.  :)

Comment 13

[Ed Morley [:emorley]]

The PR here landed a while ago:
https://github.com/mozilla-platform-ops/devservices-aws/commit/1f492318c52a6a1d2e06ade41fb4627332f768d0

And I've just tried creating a `treeherder-dev-emorley` test instance from a prod snapshot (note to future self: remember to disable backups on the new instance otherwise it takes even longer before you can do anything with it), and then was able to modify the parameter group to `treeherder-mysql57` just fine :-)