1330331 - [tracker] runbooks and ack-able alerts and SLOs, oh my

Status: RESOLVED FIXED

(Taskcluster :: Operations and Service Requests, task)

Description

[Brian Stack [:bstack]]

Note: According to https://madamgrammar.com/2016/10/21/oh-my-could-we-stop-using-this-cliche/, this bug title is cliche. Oh well, I'm sticking to it.

We've decided to get serious about monitoring the health of Taskcluster. With that in mind, we're going to have to start discussing:

* gather new metrics about how tc is functioning
* adding alerting on metrics we're gathering
* figuring out what to do when those alerts fire
* figuring out how to communicate within ourselves and to the outside world during a fire.
* figuring out how to take bug reports from the outside world
* ????
* and more!

This will be a meta-bug to track our work on this. As I type this we're working on a retrospective for a slowdown we had last night and we'll update this bug with the results of our discussion.