1307465 - Switch from the old style availability monitoring to New Relic Synthetics

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P2)

Description

[Ed Morley [:emorley]]

The availability monitoring feature is EOL:
https://rpm.newrelic.com/accounts/677903/applications/4180461/ping_targets

Synthetics is its replacement, and offers many more features:
https://synthetics.newrelic.com/accounts/677903/synthetics

Notably:
* Ability to control ping frequency and location
* Ability to ping more than one URL (eg can test both /api/ and index.html)

Comment 1

[Ed Morley [:emorley]]

> Notably:

Plus graphing of response times, response size etc.

I've set up some initial monitors for:
* stage API
* stage index.html
* prod API
* prod index.html

eg:
https://synthetics.newrelic.com/accounts/677903/monitors/ec23a19d-852d-44f7-9e16-cba039796c60

Currently they only report to me, once proven non-spammy, I'll send them to treeherder-internal too.

Comment 2

[Ed Morley [:emorley]]

I've been iterating on alert types/thresholds/locations over the last 2 months.

I've just disabled the NR Synthetics locations that correspond to Linode hosted monitors, since they intermittently fail (causing false positive email alerts, that only I'm receiving at present) and are much noisier for response time graphs.

Presuming this quietens the email alert noise, I'll change over the recipient from me directly to the treeherder-internal@ list instead, at some point soon.

Comment 3

[Ed Morley [:emorley]]

(In reply to Ed Morley [:emorley] from comment #2)
> Presuming this quietens the email alert noise, I'll change over the
> recipient from me directly to the treeherder-internal@ list instead, at some
> point soon.

Done.