840632 - Custom report: breakdown of empty crashes

Status: RESOLVED FIXED

(Socorro :: Data request, task)

Description

[Benjamin Smedberg]

The crashes with an empty dump need to be broken down more:

(Signature: "[@ EMPTY: no crashing thread identified; corrupt dump]")

1) Although we don't *directly* know the OS (because we normally get that from the dump), we can deduce some things: the memory annotations "AvailableVirtualMemory" and "AvailablePhysicalMemory" are only collected/annotated on Windows, so we can probably break this down into Windows/non-Windows by that fact alone.

2) We should check the breakdown of these crashes by process type (browser or plugin)

3) We should check the breakdown of these crashes by IsGarbageCollecting

4) It would be good too check our hunch whether most of these are caused by OOM conditions on Windows, so we should provide buckets or histograms of the crashes by both AvailableVirtualMemory and AvailablePhysicalMemory numbers.

Comment 1

[(not currently active) Ted Mielczarek]

We should also find what percentage of these crashes have an OOMAllocationSize annotation, to see whether we're OOMing as a result of a failure to malloc.

Comment 2

[Selena Deckelmann :selenamarie :selena]

Here's a rough look at the 'corrupt dump' crashes from Feb 3, 4 and 12.


1) 
looking at available_virtual_memory IS NOT NULL:
 total_crashes | estimated_windows_crashes | percentage windows 
---------------+---------------------------+--------------------
        113061 |                    104823 |              92.71

looking at available_physical_memory IS NOT NULL:
 total_crashes | estimated_windows_crashes | percentage windows 
---------------+---------------------------+--------------------
        113061 |                     95060 |              84.08

2) From looking at processed crashes, every crash is a browser crash.

3) 

Percentage of crashes that are garbage collecting:
 total_crashes | is_garbage_collecting_present | percentage is_garbage_collecting present 
---------------+-------------------------------+------------------------------------------
        113061 |                         17139 |                                    15.16

4) 

Percentage of crashes that have oom_allocation_size annotation:
 total_crashes | oom_allocation_size_present | percentage oom_allocation_size present 
---------------+-----------------------------+----------------------------------------
        113061 |                       43210 |                                  38.22


What versions seem to crash the most with a corrupt dump:
selena=# select count(*), version from crashes group by version order by count desc limit 10; 
 count | version 
-------+---------
 30236 | 18.0.1
 22553 | 19.0
 22350 | 18.0.2
  4916 | 18.0
  4371 | 17.0.2
  1997 | 17.0.1
  1945 | 12.0
  1711 | 16.0.2
  1710 | 20.0a2
  1441 | 11.0
(10 rows)


Will attach compressed data.

Comment 3

[Selena Deckelmann :selenamarie :selena]

Attachment: Collection of 'corrupt dump' crashes from Feb 3, 4 and 12 (tar, bz)

Comment 4

[Selena Deckelmann :selenamarie :selena]

I'm not so great at making creative visualizations for this data yet - so Ted, if you've got additional instructions for me, happy to follow them.

Is there anything else left to do for this ticket?

Comment 5

[Benjamin Smedberg]

The bucketing/histogram by available_virtual_memory and available_physical_memory.

Comment 6

[Selena Deckelmann :selenamarie :selena]

Attachment: Histogram of available physical memory, about 50k crashes, excluding a few crashes that are > 4e10

Comment 7

[Selena Deckelmann :selenamarie :selena]

Attachment: Histogram of available virtual memory, excluding reported VM > 1e10 (only 3 records)

Comment 8

[Selena Deckelmann :selenamarie :selena]

Attachment: Histogram of available virtual memory, with smaller buckets

Comment 9

[(not currently active) Ted Mielczarek]

(In reply to Selena Deckelmann :selenamarie :selena from comment #7)
> Created attachment 715761 [details]
> Histogram of available virtual memory, excluding reported VM > 1e10 (only 3
> records)

This is really fascinating in that there are two peaks: one near zero and one near 2GB. The one near zero is what I'd expect: we're actually OOMing and thus aborting (and failing to write a minidump in the process). The peak near 2GB is really interesting. Since the Firefox binary is an x86 binary but marked as LARGEADDRESSAWARE (I believe), it will have 2GB of VM available on a standard 32-bit Windows. (On 64-bit Windows the same binary will have 4GB of VM avaailable.)

In any event, I think what this means is that we have a significant fraction of these crashes that are happening when almost all of the virtual memory is available. That is very unexpected to me.

Comment 10

[Selena Deckelmann :selenamarie :selena]

We don't have further questions to ask at this time. Happy to open a new bug and do more analysis as needed.