522016 - No crash reports could be sent due to newly set-up NFS backend server

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.2b1pre) Gecko/20091011 Namoroka/3.6b1pre ID:20091011033822

Today I tried to investigate a crasher bug and had to send a crash report. Sadly nothing happened when clicking the crash id in about:crashes. The status bar only tells me that it is waiting for the server. Checking the log file shows me the following entry:

[Tue Oct 13 14:49:09 2009] Crash report submission failed: lost network connection

Something broken on the server so we don't accept crash reports?

Comment 1

[Shyam Mani [:fox2mike]]

Yeah, we've been seeing some issues with the storage servers, I'm probably going to wait for Aravind to wake up and take a look at this.

Comment 2

[Shyam Mani [:fox2mike]]

Henrik,

Can you please check now and see if the problem still exists?

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Works fine again. See http://crash-stats.mozilla.com/report/pending/2189e99b-4e76-4665-9760-ac5702091013.

Can we call it fixed or has some more work to be done?

Comment 4

[matthew zeier [:mrz]]

Keeping it open for some investigation.  Punting to Aravind.

Comment 5

[Jeremy Orem [:oremj]]

Crash reporter was broken last night for a couple of hours.

Comment 6

[Aravind Gottipati [:aravind]]

We switched to a new nfs backend server and that seems to have been the root cause.  I changed some params on the nfs server.  Hopefully, we won't see these problems again.  Please re-open if you continue to notice these issues.

I also logged bug 522103 to deal with another aspect of the collector code that will make this stuff work better.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Thanks Aravind. I would have another idea how to make it more safe. Can we run an automated test on all platforms which tries to send a crash report to the server after major changes have been made? Or can we attach Nagios if possible?

Comment 8

[Aravind Gottipati [:aravind]]

The changes worked and were tested.  We already have a nagios monitoring job that submits crashes every so often.  The failure case was triggered under heavy load.  We don't have an easy way to simulate this sustained heavy load on the crash collection system.