1179860 - Grant permissions to the AWS account/RDS instances used by Treeherder on Heroku

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P2)

Description

[Ed Morley [:emorley]]

The DB performance on Heroku is much worse than in Treeherder production, even with lower load. To debug this it would be useful to have access to the RDS instance CPU/memory graphs etc.

In addition, as part of bug 1178641, we'll want to try experimenting with schema migration, using the Heroku prototype's DB as a testcase. As such, we'll want to take backups/snapshots of the RDS instance so we can rollback each time we test.

Please can myself and Mauro have access initially.

If you could add using my moco email address (I don't know if I need to create an AWS account first?)
Alternatively, if you'd prefer, we could move the RDS instance to another AWS account (eg the a-team's?).

Comment 1

[Kendall Libby [:fubar] (he/him)]

I have no issues with giving you access to the instance on our account (or I wouldn't have put it there! ;-)). I believe I've got the permissions sorted such that you can do pretty much anything to the existing instance (except nuke it), though it may need extra tweaking if you want to play with using option/param groups and snapshots.

Console url is: https://moz-devservices.signin.aws.amazon.com/

I'll contact you and Mauro with account details; please change your passwords and set up 2FA for console access.

Will leave bug open until you confirm access.

Comment 2

[Ed Morley [:emorley]]

Thank you for sending the credentials. I'm not able to set up 2FA since I don't have permission to view the list of users, and so cannot view/change my own user.

Comment 3

[Kendall Libby [:fubar] (he/him)]

a pox on aws for only allowing two managed policies per-group. I've added a policy that should allow you to manage your own credentials; let me know if you can't.

Comment 4

[Ed Morley [:emorley]]

Thank you, though I'm now getting:

We were unable to create a virtual MFA device for you.

We encountered the following errors while processing your request:
User: arn:aws:iam::699292812394:user/emorley is not authorized to perform: iam:ListVirtualMFADevices on resource: arn:aws:iam::699292812394:mfa/

Comment 5

[Kendall Libby [:fubar] (he/him)]

AWS documentation on access policies is kinda hit or miss, but I believe I found all of the necessary bits; please try again! :-)

Comment 6

[Ed Morley [:emorley]]

That works - 2FA set up, ty :-)

Comment 7

[Ed Morley [:emorley]]

I tried to download a log of the mysql update, but got:

User: arn:aws:iam::699292812394:user/emorley is not authorized to perform: rds:DownloadCompleteDBLogFile on resource: arn:aws:rds:us-east-1:699292812394:db:treeherder-heroku

Could you add permissions for that? Thanks :-)

Comment 8

[Kendall Libby [:fubar] (he/him)]

Fixed!

Comment 9

[Ed Morley [:emorley]]

Works, ty :-)

Comment 10

[Ed Morley [:emorley]]

Please could we also have permissions to add alarms/alerts, eg:

User: arn:aws:iam::699292812394:user/emorley is not authorized to perform: cloudwatch:PutMetricAlarm (Service: AmazonCloudWatch; Status Code: 403; Error Code: AccessDenied; Request ID: a889600d-518a-11e5-9a96-4368bdadd421)

Seems like perhaps a wildcard on the actions listed here might be warranted?
http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/UsingIAM.html#UsingWithCloudWatch_Actions

Comment 11

[Kendall Libby [:fubar] (he/him)]

Ok, you should have plenty of acccess; the only cloudwatch related action you do NOT have is the ability to delete logs. If you need something removed, shout.

Comment 12

[Ed Morley [:emorley]]

(In reply to Kendall Libby [:fubar] from comment #11)
> Ok, you should have plenty of acccess; the only cloudwatch related action
> you do NOT have is the ability to delete logs. If you need something
> removed, shout.

Works great - thank you :-)

Comment 13

[Ed Morley [:emorley]]

For the treeherder-stage RDS instance I get this trying to view some of the mysql error logs:

User emorley is not authorized to download database log file arn:aws:rds:us-east-1:699292812394:db:treeherder-stage (Service: AmazonRDS; Status Code: 403; Error Code: AccessDenied; Request ID: e5042c41-1b78-11e6-a2e6-071acc238c7c). Please check with your administrator.

This works for the original treeherder-heroku RDS instance - could you sync up the permissions for them? (We'll also need these for the new prod instance when it's created, and for other Treeherder devs, so a group will likely be best. Guessing not used at present since deliberately restricting permissions for things managed by Terraform).

Comment 14

[Kendall Libby [:fubar] (he/him)]

(In reply to Ed Morley [:emorley] from comment #13)
> For the treeherder-stage RDS instance I get this trying to view some of the
> mysql error logs:
> 
> User emorley is not authorized to download database log file
> arn:aws:rds:us-east-1:699292812394:db:treeherder-stage (Service: AmazonRDS;
> Status Code: 403; Error Code: AccessDenied; Request ID:
> e5042c41-1b78-11e6-a2e6-071acc238c7c). Please check with your administrator.
> 
> This works for the original treeherder-heroku RDS instance - could you sync
> up the permissions for them? (We'll also need these for the new prod
> instance when it's created, and for other Treeherder devs, so a group will
> likely be best. Guessing not used at present since deliberately restricting
> permissions for things managed by Terraform).

yeah, there's a treeherder dev group already. the policy only had the treeherder-heroku db listed; I've changed that to treeherder-*, so when we make treeherder-prod it'll be good to go from the start.

Comment 15

[Ed Morley [:emorley]]

Great - working now, thank you :-)

Comment 16

[Ed Morley [:emorley]]

I was wanting to experiment with enhanced monitoring on the prototype Heroku RDS instance ("treeherder-heroku"), however the monitoring edit page gives me this error:

"Failed to retrieve IAM roles. User: arn:aws:iam::699292812394:user/emorley is not authorized to perform: iam:ListRoles on resource: arn:aws:iam::699292812394:role/"

The default IAM role "rds-monitoring-role" presumably already exists (since enhanced monitoring is already set up for treeherder-{stage,prod}), however I'm guessing the setup wizard is just unable to know that, since the treeherder dev group doesn't have permissions to list the roles.

Similarly, when viewing my own user in IAM, I'm not able to view my permissions, due to:

"User: arn:aws:iam::699292812394:user/emorley is not authorized to perform: iam:ListAttachedUserPolicies on resource: user emorley"

And when viewing the policies section (to try and figure out what access people in the treeherder dev group have), I get:

"User: arn:aws:iam::699292812394:user/emorley is not authorized to perform: iam:ListPolicies on resource: policy path /"

As such, would you mind adding iam:ListRoles, iam:ListAttachedUserPolicies and iam:ListPolicies to the treeherder dev group?

Many thanks :-)

Comment 17

[Ed Morley [:emorley]]

I'm also unable to create a new parameter group:

"User: arn:aws:iam::699292812394:user/emorley is not authorized to perform: rds:CreateDBParameterGroup on resource: arn:aws:rds:us-east-1:699292812394:pg:test-parameter-group (Service: AmazonRDS; Status Code: 403; Error Code: AccessDenied; Request ID: b02be3ed-5bea-11e6-b1db-4169e5fad376)"

Or copy an existing one:

"User: arn:aws:iam::699292812394:user/emorley is not authorized to perform: rds:CopyDBParameterGroup (Service: AmazonRDS; Status Code: 403; Error Code: AccessDenied; Request ID: ddbcc6a1-5bea-11e6-a298-fba6831a7c86)"

(Should we need to tweak some MySQL options in an emergency)

Comment 18

[Kendall Libby [:fubar] (he/him)]

(In reply to Ed Morley [:emorley] from comment #16)
>
> As such, would you mind adding iam:ListRoles, iam:ListAttachedUserPolicies
> and iam:ListPolicies to the treeherder dev group?

done!

(In reply to Ed Morley [:emorley] from comment #17)
> I'm also unable to create a new parameter group:
> 
> Or copy an existing one:
> 
> (Should we need to tweak some MySQL options in an emergency)

Interesting; you have rds:ModifyDBParameterGroup and rds:ModifyOptionGroup, but the Resource is incorrect (didn't think at the time that they wouldn't be attached to :db:treeherder-*). On the other hand, those groups are managed by terraform, so I'd really prefer to not change this and potentially have things get out of sync. If there's an emergency, we can be paged. (and we can have a discussion about write/PR access to the terraform bits over in 1290066 :-))

Comment 19

[Ed Morley [:emorley]]

I'm trying to open a support case on AWS about a memory leak in the admin console, but I'm getting:

User: arn:aws:iam::699292812394:user/emorley is not authorized to perform: support: (Service: AWSSupport; Status Code: 400; Error Code: AccessDeniedException; Request ID: cd6c985d-81a2-11e6-9d6d-0b951c3285c4)

Could you add that permission? :-)

Comment 20

[Kendall Libby [:fubar] (he/him)]

Done!