1496531 - Reduce cost of aws metrics automatically being sent to stackdriver

Status: RESOLVED FIXED

(Data Platform and Tools Graveyard :: Operations, enhancement, P1)

Description

[Harold Woo]

Stackdriver aws link projects automatically ship all aws metrics to stackdriver. Changing the iam policy from readonlyaccess to ec2readonlyaccess gives about 40% savings but we need more.  We can try limiting the policy to strictly ec2 instances only initially (removing autoscaling, elb, cloudwatch).  If that doesn't get us enough, we can push cloudservices-aws-prod|stage to use logging2.0. Another option is to modify the policy to only look for ec2 instances with the aws tag key = stackdriver

Then the ec2 instances would require this tag to exist so that gcp can find them. This would need to be tested as it hasn't been done before.

https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_condition.html
11:35 AM More here: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_multi-value-conditions.html
https://docs.aws.amazon.com/IAM/latest/UserGuide/list_amazonec2.html
https://docs.aws.amazon.com/IAM/latest/UserGuide/list_amazonec2.html#amazonec2-aws_RequestTag_tag-key

Comment 1

[Harold Woo]

A custom policy worked to bring our costs to almost 0. 

ec2-describe-only-for-stackdriver

{

    "Version": "2012-10-17",

    "Statement": [

        {

            "Effect": "Allow",

            "Action": "ec2:Describe*",

            "Resource": "*"

        }

    ]

}