1059325 - [Meta] Improve monitoring/alert coverage of treeherder (eg New Relic, Nagios, CloudWatch)

Status: RESOLVED INVALID

(Tree Management :: Treeherder: Infrastructure, task, P3)

Description

[Ed Morley [:emorley]]

This bug is for the long term fix for this problem, since even if this particular instance clears up, getting this much behind is a tree-closing event & so something that blocks switching to treeherder.

Comment 1

[Ed Morley [:emorley]]

To clarify: looking at say mozilla-inbound on treeherder right now, I see jobs still marked as running as far back as https://treeherder.mozilla.org/ui/#/jobs?repo=mozilla-inbound&revision=2ccb65865db7 - even though they appear completed on TBPL & buildapi says they finished ~10 hours ago.

Comment 2

[Mauro Doglio [:mdoglio]]

This backlog in the data ingestion is due to a problem we had on production since the last code push yesterday. I saw an error in the routine that processes the incoming jobs but it was not present on dev/stage. Everything got fixed pushing the chief red button without any code change. One way to mitigate this kind of problems would be to have a staging environment that matches the production architecture in my opinion. Also, we could write a command to re-process those jobs that resulted in a failure during the ingestion in the last week or so.

Comment 3

[Ed Morley [:emorley]]

Now that we have access to newrelic, this isn't a regression over TBPL (we're already better off than having to read the import logs on tbpl-dev/cache/...).

Comment 4

[Ed Morley [:emorley]]

Guess we need to decide whether newrelic alerts are sufficient, or if we should get coverage via nagios too?

Comment 5

[Mauro Doglio [:mdoglio]]

We should instrument the rabbitmq instance on the admin node to report to new relic.
In that way we would be able to see how many tasks we have in each queue, etc.
Maybe :fubar can help us?

Comment 6

[Ed Morley [:emorley]]

Ah great idea :-)

Comment 7

[Kendall Libby [:fubar] (he/him)]

As it happens, we are/were missing monitoring pieces for treeherder in nagios. :-(  I've added standard http/s checks for zeus and the webheads. Currently looking at what nagios can do for rabbitmq, etc.

newrelic is already set up on the admin node

Comment 8

[Ed Morley [:emorley]]

Making this bug more generic, since it sounds like we need to increase coverage for things other than data ingestion too.

(In reply to Mauro Doglio [:mdoglio] from comment #5)
> We should instrument the rabbitmq instance on the admin node to report to
> new relic.
> In that way we would be able to see how many tasks we have in each queue,
> etc.

Using something like this?
http://newrelic.com/plugins/pivotal/95

Comment 9

[Ed Morley [:emorley]]

I've just updated the Heroku metrics alerts (see https://devcenter.heroku.com/articles/metrics#threshold-alerting) to go to treeherder-internal@ rather than only me.

Comment 10

[Armen [:armenzg]]

The two bugs are within the components' queue and it is not massive.
If we ever plan to tackle them they're filed. No need for a meta bug for only two bugs.