queries against treeherder url in redash (STMO) became much slower, often timing out
Categories
(Tree Management :: Treeherder: Infrastructure, defect)
Tracking
(Not tracked)
People
(Reporter: aryx, Unassigned)
References
Details
Some of my queries against Treeherder in Redash started to time out after hitting the 180s limit.
These two queries run on a schedule:
https://sql.telemetry.mozilla.org/queries/62003#table
https://sql.telemetry.mozilla.org/queries/62004#table
When I opened them yesterday, redash indicated the last successful run was on Sunday ~5pm UTC. They succeeded after running them twice (often the second attempt succeeded in the past - caching?).
Today https://sql.telemetry.mozilla.org/queries/62004#table failed even after the secind run (it has less data to process than 62003).
Also https://sql.telemetry.mozilla.org/queries/65355/source#166269 timed out for the first time ever.
Return of bug 1579416?
|
||
Comment 1•6 years ago
|
||
Hi Sebastian,
The MySql queries might need furhter optimizations to not do full table scans. I can't help you with that as I'm not good at it. Perhaps Kyle can help. In general, you are pushing the limits of what selects can do for you.
I've tried using direct access to MySql via MySql Workbench, however, I face the same issues as with Redash. All queries that take more than 30 seconds get killed. If the query ever succeeds is by small chance.
I believe breaking that select into multiple stages is the way to go.
|
|
||
Comment 2•6 years ago
|
||
As it is, this is not an infrastructure issue but a user MySql optimization.
|
Reporter | |
Comment 3•6 years ago
|
||
As far as I know where is no further optimization possible. And if a query runs for the last 20 days without issues, that should not be an issue with the query.
FTR as mentioned in comment 0, the redash limit is 3 minutes.
And a run of https://sql.telemetry.mozilla.org/queries/62003#table finished in 39s just now.
Jason, was there higher load than normal on redash for the last 2 days.
|
|
||
Comment 4•6 years ago
|
||
I forgot to mention that I reproduced the issue directly against the MySql replica. It's a MySql query issue.
Load on the DB changes throughout the day and the average number of jobs keeps increasing month over month.
|
||
Comment 5•6 years ago
|
||
There was indeed an issue with Redash and querys may not have been executing in a timely mannor. I restarted the scheduler yesterday and it seems to have resovled the issue. I will be adding additional monitoring to capture this failure scenario.
Description
•