1141993 - Pick & set thresholds for alerts using the rabbitmq New Relic plugin

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P2)

Description

[Ed Morley [:emorley]]

Now that bug 1093757 has installed the rabbitmq plugin, we can set up alerts for a number of rabbitmq related things:
* Messages Available (# messages)
* Publish Rate (messages/sec)
* Delivery Rate (messages/sec)
* Open Channels
* Consumers

We'll need to leave the plugin running for a while to figure out sensible thresholds for these for both prod and stage.

https://rpm.newrelic.com/accounts/677903/dashboard/6293241/page/4

Comment 1

[Ed Morley [:emorley]]

Done today as part of dealing with bug 1152681.

Prod + stage:
* Messages Available (# messages): warn=50, crit=100
* Publish Rate (messages/sec): warn=60, crit=100
* Delivery Rate (messages/sec): warn=60, crit=100
* Open Channels: warn=80
* Consumers: warn=60

The "Messages Available" stat is the most critical.

To change them later, use the settings cog on:
https://rpm.newrelic.com/accounts/677903/plugins/16138

The email alerts for plugins don't use the new alert section, so have to be set up here:
https://rpm.newrelic.com/accounts/677903/integrations?page=alerting#tab-integrations=_mobile_amp_plugin_alert_email

I've added my email to start with, and once I know how noisy it will/won't be (eg the text help on that page contradicts itself a few times - not clear what other non-plugin notifications will be sent to the email there as well) I can try adding others.

Comment 2

[Ed Morley [:emorley]]

But the alerts/thresholds themselves seem to be working well, eg see the red shaded areas here:
https://rpm.newrelic.com/accounts/677903/dashboard/6293241?tw[dur]=last_6_hours&tw[end]=1428587506

Comment 3

[Ed Morley [:emorley]]

We were getting alerts on stage, since the recent new queue additions have bumped up the totals. As such I've updated stage/prod to the following:

Note: higher values=worse (ie: the alert goes off when the actual value goes above the alert value).

(In reply to Ed Morley [:emorley] from comment #1)
> * Open Channels: warn=80

warn=110, crit=130

> * Consumers: warn=60

warn=65, crit=75

These values had previously been updated to be more strict, including here for reference:

> * Publish Rate (messages/sec): warn=60, crit=100

warn=50, crit=80

> * Delivery Rate (messages/sec): warn=60, crit=100

warn=50, crit=80