948772 - socorro master hit high load and query count

Status: RESOLVED FIXED

(Socorro :: General, task)

Description

[Matt Pressman [:mpressman]]

At 17:52 nagios reported high load on socorro1.db.phx1
[17:52:37] <nagios-phx1> Tue 17:52:37 PST [1073] socorro1.db.phx1.mozilla.com:Load is CRITICAL: CRITICAL - load average: 73.29, 62.44, 32.18 (http://m.allizom.org/Load)

A manual view showed load even higher:
load average: 248.94, 154.60, 83.92

The query count showed as many as 401
postgres=# select count(*) from pg_stat_activity ;
 count
-------
   401

Comment 1

[Matt Pressman [:mpressman]]

solarce found this debug right around the start of the high load
2013-12-10 16:52:22,243 DEBUG - MainThread - about to run <class 'socorro.cron.jobs.matviews.ReportsCleanCronApp'>

Additionally a number of backfill_matviews seemed to have been running

Comment 2

[Robert Kaiser]

After the fix to bug 946347 we needed to have matviews re-run to correct the Nightly and Metro numbers, that could play into this.

Comment 3

[Selena Deckelmann :selenamarie :selena]

This was due to bug 946347.