1588757 - send system logs to stackdriver rather than papertrail

Status: NEW

(Infrastructure & Operations :: RelOps: OpenCloudConfig, task)

Description

[Rob Thijssen [:grenade (EET/UTC+0300)]]

gcp logging projects created with gcloud

# create project folder (folder 482144716317 maps to /firefox.gcp/mozilla.com/relops/applications)
gcloud alpha resource-manager folders create --folder 482144716317 --display-name fxci-logging
logging_folder_id=$(basename $(gcloud alpha resource-manager folders list --folder 482144716317 --filter DISPLAY_NAME:fxci-logging --format=json | jq -r '.[0].name'))
# create logging projects
for environment in staging production; do
  gcloud projects create fxci-${environment}-logging --folder=${logging_folder_id}
  # set project ownership to relops group
  gcloud projects add-iam-policy-binding fxci-${environment}-logging --member group:[e-mail] --role roles/owner
  # grant project viewer access
  for user_name in bpitts bstack edunham; do
    gcloud projects add-iam-policy-binding fxci-${environment}-logging --member user:${user_name}@mozilla.com --role roles/viewer
  done
done

stackdriver workspaces created with console

note: no gcloud cli command or api call exists at this time to create workspaces. it must be done manually at: https://app.google.stackdriver.com

• fxci-production-logging for worker instance projects:
  • fxci-production-test-workers
  • fxci-production-level1-workers
  • fxci-production-level3-workers
  • aws-stackdriver-log-1571127027 (bridge project to accept ec2 logs)
• fxci-staging-logging for worker instance projects:
  • fxci-staging-test-workers
  • fxci-staging-level1-workers
  • fxci-staging-level3-workers

set ownership on auto-created bridge project

gcloud projects add-iam-policy-binding aws-stackdriver-log-1571127027 --member group:[e-mail] --role roles/owner

service account for ec2 logging created with gcloud

gcloud iam service-accounts create taskcluster-worker-ec2 --display-name "taskcluster worker in ec2" --project aws-stackdriver-log-1571127027
gcloud iam service-accounts keys create ~/gcp-keys/[e-mail] --iam-account [e-mail]
gpg2 -e -u 0x1C09AC24C113C7F080DD4AA5B3C5A958508A43F2 -r 0x1C09AC24C113C7F080DD4AA5B3C5A958508A43F2 -r 0x4D8F28D5C2CA60C9EECD422E7A35F7E8F9132A40 ~/gcp-keys/[e-mail]
gcloud projects add-iam-policy-binding aws-stackdriver-log-1571127027 --member serviceAccount:[e-mail] --role roles/logging.logWriter
gcloud projects add-iam-policy-binding aws-stackdriver-log-1571127027 --member serviceAccount:[e-mail] --role roles/monitoring.metricWriter

aws resources created for stackdriver integration

• stackdriver iam role

• stackdriver iam policy (taken from docs at: minimal-aws-permissions)

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Action": [
                    "autoscaling:Describe*",
                    "cloudfront:Get*",
                    "cloudfront:List*",
                    "cloudwatch:Describe*",
                    "cloudwatch:Get*",
                    "cloudwatch:List*",
                    "dynamodb:Describe*",
                    "dynamodb:Get*",
                    "dynamodb:List*",
                    "ec2:Describe*",
                    "ec2:Get*",
                    "elasticache:Describe*",
                    "elasticache:List*",
                    "elasticloadbalancing:Describe*",
                    "es:Describe*",
                    "es:List*",
                    "events:Describe*",
                    "events:List*",
                    "health:Describe*",
                    "health:Get*",
                    "health:List*",
                    "kinesis:Describe*",
                    "kinesis:Get*",
                    "kinesis:List*",
                    "lambda:Get*",
                    "lambda:List*",
                    "rds:Describe*",
                    "rds:List*",
                    "redshift:Describe*",
                    "redshift:Get*",
                    "redshift:View*",
                    "s3:Get*",
                    "s3:List*",
                    "ses:Get*",
                    "ses:List*",
                    "ses:Describe*",
                    "sns:Get*",
                    "sns:List*",
                    "sqs:Get*",
                    "sqs:List*"
                ],
                "Effect": "Allow",
                "Resource": "*"
            }
        ]
    }
  

Comment 1

[Rob Thijssen [:grenade (EET/UTC+0300)]]

provision stackdriver key to ec2 instances:
https://github.com/mozilla-releng/OpenCloudConfig/commit/d6de54e