1201063 - Get the Amazon RDS instance used by Heroku reporting to New Relic

Status: RESOLVED WONTFIX

(Tree Management :: Treeherder: Infrastructure, defect, P3)

Description

[Ed Morley [:emorley]]

...or alternatively set up some RDS alerts using the Amazon dashboard in the meantime.

Comment 1

[Ed Morley [:emorley]]

In the meantime I've set up a CloudWatch alert for if there is <20GB free space on RDS.

Comment 2

[Ed Morley [:emorley]]

As an alternative, we can enable RDS's new enhanced monitoring feature, which feeds into cloudwatch:
https://aws.amazon.com/blogs/aws/new-enhanced-monitoring-for-amazon-rds-mysql-5-6-mariadb-and-aurora/

I'm unable to switch it on now, since I don't have enough permissions.

Comment 3

[Kendall Libby [:fubar] (he/him)]

(In reply to Ed Morley [:emorley] from comment #2)
> As an alternative, we can enable RDS's new enhanced monitoring feature,
> which feeds into cloudwatch:
> https://aws.amazon.com/blogs/aws/new-enhanced-monitoring-for-amazon-rds-
> mysql-5-6-mariadb-and-aurora/
> 
> I'm unable to switch it on now, since I don't have enough permissions.

Enabled enhanced monitoring on both the treeherder-heroku and treeherder-stage instances.

Comment 4

[Ed Morley [:emorley]]

The treeherder-stage RDS instance event log says:

Amazon RDS has been unable to create credentials for enhanced monitoring and this feature has been disabled. This is likely due to the rds-monitoring-role not being present and configured correctly in your account. Please refer to the troubleshooting section in the Amazon RDS documentation for further details.

Was this before all the changes were made, and as such can be ignored? :-)

Comment 5

[Kendall Libby [:fubar] (he/him)]

yes

Comment 6

[Kendall Libby [:fubar] (he/him)]

Are we cool with cloudwatch, or do we still want to look at getting RDS data into newrelic?

Comment 7

[Ed Morley [:emorley]]

Bug 1306597 is now for the CloudWatch alerts, I guess it depends on how useful we think the MySQL-specific stats (like breakdown of SELECTS vs ... and things like temp table counts, innodb mutexes etc) are.

We have some of them at the app level in NR, via the python agent, but it doesn't have everything.

If we were to do this, it would mean running an EC2 instance somewhere that used the third party NR MySQL plugin to poll the RDS instances and report to NR.

Perhaps just not worth the hassle?

Comment 8

[Ed Morley [:emorley]]

I'll double back to this later, but for now some links I've found so far:

https://www.datadoghq.com/blog/monitoring-rds-mysql-performance-metrics/
https://www.datadoghq.com/blog/how-to-collect-rds-mysql-metrics/
https://www.datadoghq.com/blog/monitor-rds-mysql-using-datadog/

https://newrelic.com/aws-monitoring-tips
 -> https://newrelic.com/plugins/new-relic-inc/63
 -> https://aws.amazon.com/marketplace/pp/B01KLLCQZE
https://rpm.newrelic.com/accounts/677903/plugins/directory/52
 -> https://github.com/newrelic-platform/newrelic_mysql_java_plugin

Comment 9

[Ed Morley [:emorley]]

New Relic now have a new Infrastructure feature, that can directly monitor RDS without needing a separate EC2 node running a plugin:
https://docs.newrelic.com/docs/infrastructure
https://docs.newrelic.com/docs/infrastructure/infrastructure-integrations/amazon-integrations/aws-rds-integration
https://docs.newrelic.com/docs/infrastructure/infrastructure-integrations/getting-started/connect-aws-integrations-infrastructure

Comment 10

[Ed Morley [:emorley]]

This would have sped up the diagnosis of bug 1386331.

Comment 11

[Joel Maher ( :jmaher ) (UTC -8)]

cleaning up old bugs we haven't fixed