383757 - Crash reporting tool is not responding

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Michael Morgan [:morgamic]]

The crash reporter is not responding due to (possibly) the netapp disk problems or db connection limits.  Could someone take a look and investigate?  :)

Comment 1

[:Gavin Sharp [email: [e-mail]]]

Probably the same problem as bug 383742?

Comment 2

[Benjamin Smedberg]

Aravind, there is a new version of the app available on the stable branch, but this is a significant upgrade of the collector/processor/reporter and requires some config changes and a database upgrade. If we can limp along over the weekend (perhaps kill/restart the reporter every hour on a cron?) I can file the upgrade bug for next week sometime.

Comment 3

[Benjamin Smedberg]

ok, let's do it now:

The code is at socorro.googlecode.com/svn/branches/stable

The following config option is new:
# Tell the collector where the reporter lives (optional)
# reporterURL = 'http://crash-stats.mozilla.com'
reporterURL = None

If we can have crash-stats.mozilla.com now, let's use that, otherwise it should be 'https://crash-reports.mozilla.com/reports'

Also, I believe that we have collected enough sample data for load testing, so please set saveProcessedMinidumps = False

Finally, you will need to re-run "paster setup-app production.ini", with the same caution as last time about having to comment out the "paste:prefix" lines temporarily.

Comment 4

[Aravind Gottipati [:aravind]]

switched repos and updated everything to rev 213.

Comment 5

[Benjamin Smedberg]

I think there's a permission error with the cache directory: I'm getting

OSError: [Errno 13] Permission denied: 'None'

from the reporter logs. Did you leave the default cache_dir=$(here)s/data setting, and does the app have permissions to write to that directory?

Comment 6

[Benjamin Smedberg]

You're welcome to set cache_dir to any appropriate directory, of course, as long as the app user has access to it.

Comment 7

[Aravind Gottipati [:aravind]]

Does it store stuff here?  or is it just a temporary location?  Does it matter if stuff is wiped from there?

Comment 8

[Robert Sayre]

(In reply to comment #7)
> Does it store stuff here?  or is it just a temporary location?  Does it matter
> if stuff is wiped from there?
> 

It is a disk-based cache for report lists. It is not permanent data. The front pages and the top crash page are cached in memory.

Individual reports are not cached by the app, but they are given long Expires: headers for use by proxy caches (once we're off ssl) and the browser cache.

Comment 9

[Benjamin Smedberg]

The new deployment is broken, but I've debugged it back to a code error of some sort. So I'm going to mark this bug fixed and work with sayrer on getting a fix to you.

FWIW, the reason it's broken is because I was testing with beaker 0.6.x and you are deploying beaker 0.7.2 (which we should work with, probably).