1490759 - Treeherder is slow while loading bug suggestions and logs

Status: RESOLVED WORKSFORME

(Tree Management :: Treeherder: API, defect, P1)

Comment 1

[Ed Morley [:emorley]]

Yeah we've been having a few response time alerts.

Load caused by the jobs API has jumped up, starting about 90 minutes ago:
https://screenshots.firefox.com/L9FKoImaIY0aH3EC/rpm.newrelic.com
https://screenshots.firefox.com/rVIPWEb6IvMmh8xd/rpm.newrelic.com

...and a corresponding load on the DB from selects on the jobs table:
https://screenshots.firefox.com/8JEezhMMQsDzTX2o/rpm.newrelic.com
https://screenshots.firefox.com/5PKbOMYCqZvpvrpY/rpm.newrelic.com

...and the prod RDS instance has much higher CPU load than normal:
https://screenshots.firefox.com/eXkEIfRULYnqw4j8/console.aws.amazon.com

...but none of the other stats have jumped up (eg throughput shown on New Relic, read/write I/O on the DB), which is strange. Perhaps AWS noisy neighbour?

The DB CPU load appears to have improved slightly over the last 5-10 mins.

Comment 2

[Ed Morley [:emorley]]

Things have returned to normal since:
https://screenshots.firefox.com/hXz3Lf2ZpGNlqmQi/rpm.newrelic.com

It's still not entirely clear what the root cause was.

Comment 3

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Ryan and Andreea experience the issue again. Does newrelic show the issue for users in general?

Comment 4

[Ed Morley [:emorley]]

New Relic doesn't show any further spikes:
https://screenshots.firefox.com/dAaOlEF39hVkfxnj/rpm.newrelic.com

Testing prod Treeherder now, it responds fast for me.

It would be useful to know if the issues they see are UI load times/responsiveness or duration for API XHR calls related.

Comment 5

[Andreea Pavel [:apavel]]

Hi.

The issues i'm having are regarding the duration of API XHR calls. The most frequent ones are regarding failure selection/retrieving data for them (summary/suggestions), classification saving - switching to a next failure.

Comment 6

[Armen [:armenzg]]

How bad is the impact of this?

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

IMHO a worksforme, bug 1553199 is the issue observed these days.