977409 - [monitoring-story] monitor auth server memory statistics in production

Status: RESOLVED FIXED

(Cloud Services :: Server: Firefox Accounts, defect)

Description

[Katie Parlante]

As a developer, I want to monitor server memory statistics, so that I can debug slow memory leaks.

Discussed here: https://github.com/mozilla-services/puppet-config/issues/146#issuecomment-35434579

I believe the plan is to have a heka dashboard on the aggregator, with a stacked view, not unlike this telemetry example: http://ec2-50-112-66-71.us-west-2.compute.amazonaws.com:4352/#sandboxes/TelemetryServerMetricsHostAggregator/outputs/TelemetryServerMetricsHostAggregator.RequestStatisticsSuccess.cbuf

and/or separate views for each host.

Comment 1

[Katie Parlante]

:trink's dashboards are now on production: https://heka.fxa.us-west-2.prod.mozaws.net/#plugins/filters/FxaAuthMemory

I believe he also has anomaly detection set up for this, alerts currently going to trink and whd as he tweaks it.

:dcoates, I'd like to close out this bug, let me know if this meets your needs.

Comment 2

[Danny Coates [:dcoates]]

looks great! thanks y'all :)