465621 - Socorro reporter is timing out on just about every query

Status: RESOLVED DUPLICATE of bug 444749

(mozilla.org Graveyard :: Server Operations, task)

Description

[Michael Morgan [:morgamic]]

This is atypical and it's off-peak.  We need to be able to do better forensics on the breakpad db server to understand what is causing this load.

Comment 1

[Justin Fitzhugh]

stuff is basically broken now - needs to be fixed asap.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The timeout or better connection reset message comes for nearly each search you can start. Even the topcrasher list will not be displayed. I have to hit reload several times to get the results. Looks like more than severity minor to me.

Comment 3

[Michael Morgan [:morgamic]]

Aravind sent us the slow query logs.  We had multiple queries with 3-month time spans across all products and all branches sitting in the queue.  That combined with a db dump and multiple report spams for other queries amounted to the server issues causing delays for even simple queries.

We've got some low-hanging fruit to pick before Lars completes larger db changes for bug 432450.  Not much IT can do currently, but we're going to divert more traffic to topcrashes (and improve the topcrash report list) and limit query time spans to periods that the system can actually handle until we have improved partitioning.

In the meantime, I'll dupe this with 444749 which is the same issue.