570906 - Camino 2.0.3 topcrash is "(null signature)" composed of unprocessed valid crash reports or "[Errno 24] Too many open files"

Status: RESOLVED DUPLICATE of bug 570478

(Socorro :: General, task)

Description

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

The current topcrash for Camino 2.0.3 is our old friend "(null signature)": http://crash-stats.mozilla.com/topcrasher/byversion/Camino/2.0.3

Interestingly, when I load the URL there (see URL field), I mostly see one of two things:

1) Unprocessed crashes (I thought Camino wasn't being throttled at all), which are processed into actual reports, e.g. bp-49aff946-c8d5-4088-942d-4bda82100606

2) Crashes with 1 of 3 "processor" error conditions:
  a) "WARNING: Json file missing Add-ons; [Errno 24] Too many open files", by far the most common; e.g. bp-59e01018-e04f-4a89-a0ec-a17192100607
  b) "WARNING: Json file missing Add-ons; expected string or buffer", e.g. bp-d4e26a58-da8e-47b0-bb7a-94eac2100608
  c) "WARNING: Json file missing Add-ons; /home/processor/stackwalk/bin/stackwalk.sh returned no header lines for reportid: 131199574; No thread was identified as the cause of the crash; No signature could be created because we do not know which thread crashed; /home/processor/stackwalk/bin/stackwalk.sh returned no frame lines for reportid: 131199574; /home/processor/stackwalk/bin/stackwalk.sh failed with return code 1 when processing dump 49901e66-58b8-432d-8380-c14e02100608", e.g. bp-49901e66-58b8-432d-8380-c14e02100608

I think 2c is the "normal" corrupt dump error.  

The leading "WARNING: Json file missing Add-ons" part of all of the error messages has been normal ever since Add-on tracking has been around (although I see that in working Camino crash reports, it has currently disappeared).

In all 2a/b/c cases, the Raw Dump in the web view is entirely empty; I thought in the past the dump for real "null signature" crashes typically had some data in it, and "No crash" or corrupted threads, or something.  I have no idea if the physical/downloadable raw dumps are actually entirely empty or not because accessing them apparently is a separate permissions bit (beyond s-g) that I don't have.

In any case, this smells like a server/processor bug (and one introduced recently, as a few weeks ago we didn't see these sorts of high numbers, but now it's the topcrash for 2.0.3 *and* 2.0.2), or our users are somehow hitting some extraordinarily nasty crashes, but given the unprocessed reports, that seems less likely.

Comment 1

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

I just found bug 570478, and it sounds like this is a dupe of that.

Problem reports began for us on Thursday, 3 Jun at 01:56, and stopped on Monday 7 Jun at 09:25; the three reports before that time and the three after that time are all the "normal" 2c corrupt dump error.