1407377 - [meta] Triage the top N web and worker New Relic slow transactions to find quick wins

Status: NEW

(Tree Management :: Treeherder: Infrastructure, enhancement, P2)

Description

[Ed Morley [:emorley]]

Given that Treeherder's workload has been changing over time (eg taskcluster transition, schema changes, favouring pulse over REST API, adding new features) we're unsurprisingly now exercising parts of the codebase that have previously not been analysed as closely for performance.

Looking at the New Relic slow transactions for both web and non-web (ie celery worker) transactions, there look to be several easy wins that would make a big difference to both API performance (and thus real/perceived frontend responsiveness) as well as data ingestion load/capacity.

Using a data-driven approach will mean we can get the biggest ROI from our time, given we're short on headcount. For example bug 1407365 appears in the top 3 slow transactions and is actively affecting sheriffs.

We should:
1) Go through the top N transactions on New Relic
2) File bugs for each blocking this one
3) Once a proportion of the issues are fixed, return to step 1 to see what's left

Comment 1

[Ed Morley [:emorley]]

We should also check out:
https://aws.amazon.com/blogs/database/tuning-amazon-rds-for-mysql-with-performance-insights/

Comment 2

[Sarah Clements [:sclements]]

I've done some work here but I don't think this is high priority at the moment.