938661 - TCBS pages time out loading on prod

Status: RESOLVED INVALID

(Socorro :: General, task)

Description

[Robert Kaiser]

When I load top crasher pages linked from https://crash-stats.mozilla.com/home/products/Firefox#date_range_type=report they take quite long to load. 28.0a1 comes up after a bit, 27.0a2 take very long, 26.0b and 25.0 time out with the browser telling me "The connection to the server was reset while the page was loading."

This blocks crash investigations right now.

Comment 1

[Selena Deckelmann :selenamarie :selena]

Happened again today starting around 12:50pm UTC.

Comment 2

[Selena Deckelmann :selenamarie :selena]

Here's what was going on with crontabber, so not clear that any crontabber jobs were involved in the pileup: 

breakpad=# select app_name, log_time from crontabber_log  
WHERE 
log_time BETWEEN '2013-11-15 16:00:00+00' and '2013-11-15 20:00:00+00'
order by log_time desc;
       app_name        |           log_time            
-----------------------+-------------------------------
 server-status         | 2013-11-15 19:50:07.424093+00
 reports-clean         | 2013-11-15 19:46:36.150051+00
 server-status         | 2013-11-15 19:43:47.747186+00
 ftpscraper            | 2013-11-15 19:43:42.741626+00
 server-status         | 2013-11-15 19:30:54.86221+00
 automatic-emails      | 2013-11-15 19:30:50.093192+00
 bugzilla-associations | 2013-11-15 19:25:03.436885+00
 server-status         | 2013-11-15 19:15:19.118158+00
 duplicates            | 2013-11-15 19:15:14.278316+00
 server-status         | 2013-11-15 19:10:07.078868+00
 server-status         | 2013-11-15 19:00:07.725946+00
 server-status         | 2013-11-15 18:55:07.954847+00
 server-status         | 2013-11-15 18:46:46.573188+00
 reports-clean         | 2013-11-15 18:46:41.581062+00
 ftpscraper            | 2013-11-15 18:44:16.533093+00
 ftpscraper            | 2013-11-15 18:39:42.87604+00
 server-status         | 2013-11-15 18:26:49.537139+00
 automatic-emails      | 2013-11-15 18:26:44.155486+00
 bugzilla-associations | 2013-11-15 18:20:09.23077+00
 server-status         | 2013-11-15 18:10:23.128679+00
 duplicates            | 2013-11-15 18:10:17.549291+00
 server-status         | 2013-11-15 18:00:07.860816+00
 server-status         | 2013-11-15 17:55:07.511431+00
 server-status         | 2013-11-15 17:45:07.939284+00
 server-status         | 2013-11-15 17:36:24.8906+00
 reports-clean         | 2013-11-15 17:36:20.009892+00
 ftpscraper            | 2013-11-15 17:34:12.630219+00
 server-status         | 2013-11-15 17:21:36.755265+00
 automatic-emails      | 2013-11-15 17:21:31.379984+00
 bugzilla-associations | 2013-11-15 17:15:05.259669+00
 server-status         | 2013-11-15 17:10:24.421838+00
 duplicates            | 2013-11-15 17:10:19.360907+00
 server-status         | 2013-11-15 17:05:08.254711+00
 server-status         | 2013-11-15 17:00:07.962199+00
 server-status         | 2013-11-15 16:55:08.078909+00
 server-status         | 2013-11-15 16:50:07.245295+00
 server-status         | 2013-11-15 16:40:07.60942+00
 server-status         | 2013-11-15 16:34:36.562266+00
 reports-clean         | 2013-11-15 16:34:31.504053+00
 reports-clean         | 2013-11-15 16:32:04.507896+00
 ftpscraper            | 2013-11-15 16:29:40.940444+00
 server-status         | 2013-11-15 16:21:27.661981+00
 automatic-emails      | 2013-11-15 16:21:22.548726+00
 bugzilla-associations | 2013-11-15 16:15:04.557952+00
 server-status         | 2013-11-15 16:05:21.712782+00
 duplicates            | 2013-11-15 16:05:16.968251+00
(46 rows)

Comment 3

[Lonnen :lonnen]

This is no longer an issue.