Bugzilla

(In reply to Mark Reid [:mreid] from comment #6)
> This query queue can be monitored using the Redash API at [3], is that
> something we can easily add to DataDog? Inside that JSON response, we're
> interested in tracking the length of the "waiting" element.
> 
> It would also be interesting to keep track of the age of the oldest item in
> that array as an indicator of "time in queue" per Sunah's suggestion above.
> 
> [1] https://sql.telemetry.mozilla.org/admin/status
> [2] https://sql.telemetry.mozilla.org/admin/queries/tasks#waiting
> [3] https://sql.telemetry.mozilla.org/api/admin/queries/tasks

Thanks I didn't realize that this data was available. We should be able to poll this and add it into datadog and alerting around it. 

A large waiting queue may indicate data sources are overloaded and based on that we can figure out what to do next.

(In reply to Mark Reid [:mreid] from comment #6)
> This query queue can be monitored using the Redash API at [3], is that
> something we can easily add to DataDog? Inside that JSON response, we're
> interested in tracking the length of the "waiting" element.
> 
> It would also be interesting to keep track of the age of the oldest item in
> that array as an indicator of "time in queue" per Sunah's suggestion above.
> 
> [1] https://sql.telemetry.mozilla.org/admin/status
> [2] https://sql.telemetry.mozilla.org/admin/queries/tasks#waiting
> [3] https://sql.telemetry.mozilla.org/api/admin/queries/tasks

Thanks I didn't realize that this data was available. We should be able to poll this, add it into datadog and add alerting around it. 

A large waiting queue may indicate data sources are overloaded and based on that we can figure out what to do next.