1593709 - Create a solution for ingesting stackdriver metrics for services running in AWS

Status: RESOLVED FIXED

(Cloud Services :: Operations: Metrics/Monitoring, task)

Description

[Brian Pitts]

People need to be able to import stackdriver metrics into influxdb for some projects that live in AWS. Examples are projects that use log-based metrics (e.g. bugzilla) or use a gcp service such as pubsub (socorro).

I had originally thought that projects could each add the appropriate telegraf config to a singleton instance, but during the team meeting people pointed out that not all projects currently have one. The consensus what that it's a better to have a single place to do this for all AWS services that need it.

I think this place could be either an EC2 instance or a GKE cluster where we make it easy for people to configure one or more telegrafs that run an instance of the stackdriver input plugin (https://github.com/influxdata/telegraf/tree/master/plugins/inputs/stackdriver) for each GCP project that corresponds to an AWS service. By default that plugin should collect just logging metrics, but it should be configurable to collect other metrics as well.

Rather than requiring people to generate a GCP service account per-project with permissions to read from stackdriver, we should be able to create one that has permissions at the folder level to read stackdriver metrics from all our projects.

Our existing AWS telegraf config is at https://github.com/mozilla-services/cloudops-deployment/tree/master/libs/puppet/modules/telegraf and our existing GKE telegraf config is at https://github.com/mozilla-services/cloudops-infra/blob/master/libs/influx/k8s/values/values.yaml

Bob volunteered to take a crack at this since he'll want it for Autograph soon.

Comment 1

[Brian Pitts]

Last week, Bob and I met and discussed this. The plan is for him to build out a new project in cloudops-deployment that will deploy a telegraf that users can configure via hiera to enable stackdriver metrics collections for their projects.

After that, I set up a new user with folder-level permissions to read stackdriver from all our projects and send bo the service account key.

Comment 2

[Bob Micheletto [:bobm]]

Filed PR to collect these metrics. The telegraf service is presently running and collecting metrics for some services. Documentation on adding new services is forthcoming.

Comment 3

[Bob Micheletto [:bobm]]

(In reply to Bob Micheletto [:bobm] from comment #2)

Filed PR to collect these metrics. The telegraf service is presently running and collecting metrics for some services. Documentation on adding new services is forthcoming.

Still addressing PR.

Comment 4

[Brian Pitts]

Bob, this just needs documentation and then it can be closed, right?

Comment 5

[Bob Micheletto [:bobm]]

(In reply to Brian Pitts from comment #4)

Bob, this just needs documentation and then it can be closed, right?

Should I just add something to the Influx FAQs? There's already a question: How can I derive metrics from my log files?

Comment 6

[Brian Pitts]

Yeah, that sounds like a good start. I need to create a page describing our monitoring architecture in AWS. Once we have that, we could put some info there too.

On further thought we probably also need to set up some monitoring for this instance. Did you want to be responsible for it, or should I just put it in the standard monitoring policy which is me with fallback to erik?

Comment 7

[Bob Micheletto [:bobm]]

(In reply to Brian Pitts from comment #6)

Yeah, that sounds like a good start. I need to create a page describing our monitoring architecture in AWS. Once we have that, we could put some info there too.

Addressed in cloudops-docs PR 77.

On further thought we probably also need to set up some monitoring for this instance. Did you want to be responsible for it, or should I just put it in the standard monitoring policy which is me with fallback to erik?

I think this should be owned by yourselves, since it's pretty tied up with the Monitoring Project.

Comment 8

[Bob Micheletto [:bobm]]

PR 77 merged. Closing bug.