1481419 - dashboards often stop to update, some queries stuck with "Query in queue"

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

On Sunday, https://sql.telemetry.mozilla.org/dashboard/sheriffing-activity---overview had all its elements stop updating. They remained stuck until yesterday and manually starting such a query showed only "Query in queue" and it would remain in that state for hours. Later the queries worked again, but ~7h ago they stopped updating again.

Other (tabular) STMO queries which use the same database (Treeherder) succeed, e.g. https://sql.telemetry.mozilla.org/queries/53332/source#141520

Comment 1

[Jason Thomas [:jason]]

Most data sources on STMO (including Treeherder) use a shared queue for query execution. Usually this is fine until queries against a specific datasource has issues and blocks other datasource queries from being executed in a timely manner. There is a shared queue for 'adhoc' and 'scheduled' queries so that is why you are seeing inconsistent behavior. 

I've moved 'Treeherder' datasource onto it's own adhoc and scheduled queue. This will allow the queries to be serviced in a more timely fashion.

I've also moved 'DSMO-RS' datasource (the problematic one in this case) to it's own queue so it doesn't effect other datasources in the shared queue.

Let me know if you continue to have issues.