1560657 - High database spikes affect UI response time

Status: RESOLVED DUPLICATE of bug 1553199

(Tree Management :: Treeherder: Infrastructure, defect)

Description

[Armen [:armenzg]]

There were a couple of Heroku 95th percentile emails. This bug is only analyzing the latest one.

This spike does not seem to be related to a high number of talos jobs completing:
https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&fromchange=e053c95bb1b4a99407040a33be906b9465c5fb2b&searchStr=talos
https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&searchStr=talos&fromchange=99f1b315fa89db68ef3e7f8875bbd0a4b818423d

You can see the spike on MySql job selects:
https://cl.ly/3ace8cb7afbb

You can see the JobsView set spiking:
https://cl.ly/65e6ba9e749a

Now, once I look closely at log parser:
https://cl.ly/a4918e40c9e2
You will see that first we start having problem with MySql line failures and then it starts affecting other insert operations. This next screenshots only leaves MySql related operation and you will notice it more clearly
https://cl.ly/ded35e84aa41

I believe my proposal to use the replica for read operations is gaining weight.

Comment 1

[Armen [:armenzg]]

We will first try what we mentioned in bug 1553199.
Maybe I can do the same for failure line inserts.

Comment 2

[Armen [:armenzg]]

No need to have multiple bugs for the same root issue.