No crash reports could be sent due to newly set-up NFS backend server

VERIFIED FIXED

Status

task
--
critical
VERIFIED FIXED
10 years ago
4 years ago

People

(Reporter: whimboo, Assigned: aravind)

Tracking

Details

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.2b1pre) Gecko/20091011 Namoroka/3.6b1pre ID:20091011033822

Today I tried to investigate a crasher bug and had to send a crash report. Sadly nothing happened when clicking the crash id in about:crashes. The status bar only tells me that it is waiting for the server. Checking the log file shows me the following entry:

[Tue Oct 13 14:49:09 2009] Crash report submission failed: lost network connection

Something broken on the server so we don't accept crash reports?
Yeah, we've been seeing some issues with the storage servers, I'm probably going to wait for Aravind to wake up and take a look at this.
Henrik,

Can you please check now and see if the problem still exists?
Works fine again. See http://crash-stats.mozilla.com/report/pending/2189e99b-4e76-4665-9760-ac5702091013.

Can we call it fixed or has some more work to be done?
Keeping it open for some investigation.  Punting to Aravind.
Assignee: server-ops → aravind
Crash reporter was broken last night for a couple of hours.
Assignee

Comment 6

10 years ago
We switched to a new nfs backend server and that seems to have been the root cause.  I changed some params on the nfs server.  Hopefully, we won't see these problems again.  Please re-open if you continue to notice these issues.

I also logged bug 522103 to deal with another aspect of the collector code that will make this stuff work better.
Status: NEW → RESOLVED
Last Resolved: 10 years ago
Resolution: --- → FIXED
Thanks Aravind. I would have another idea how to make it more safe. Can we run an automated test on all platforms which tries to send a crash report to the server after major changes have been made? Or can we attach Nagios if possible?
Status: RESOLVED → VERIFIED
OS: Mac OS X → All
Hardware: x86 → All
Summary: Crash reports cannot be submitted on OS X → No crash reports could be sent due to newly set-up NFS backend server
Assignee

Comment 8

10 years ago
The changes worked and were tested.  We already have a nagios monitoring job that submits crashes every so often.  The failure case was triggered under heavy load.  We don't have an easy way to simulate this sustained heavy load on the crash collection system.
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.