724046 - Breakpad should ask jemalloc to free up some heap memory so it's less likely to OOM itself

Status: RESOLVED WONTFIX

(Toolkit :: Crash Reporting, defect)

Description

[Justin Lebar (not reading bugmail)]

In dev.planning, Bob Clary wrote:

> One of the most common types of crashes I can reproduce in automation relate to 
> pages where many images appear on the page.
>
> Due to memory exhaustion these crashes can appear in socorro as an empty or 
> corrupt dump. Currently the top crasher for Firefox 10 with 8.4% of all crashes 
> are empty or corrupt dumps.

bsmedberg clarified:

> The problem is that the Windows crash reporter calls into the API 
> "MinidumpWriteDump" which internally has to allocate memory. Whether or not 
> the crash is a null-pointer deref or some other exception, if we are truly out 
> of physical memory or out of address space, MinidumpWriteDump fails to write a 
> usable minidump and all we get is an empty report.

I think we can fix this without the nuclear option of moving the crash reporter into a separate process, by asking jemalloc to free X mb of the heap, when the crash reporter starts up.

This jemalloc code should be relatively simple.  We'd actually hand the memory back to the OS; ted explains that the crash reporter doesn't allocate using jemalloc, so the memory will need to be OS-level free'd in order for the crash reporter to have access to it.

But because breakpad stores some data on the heap before the crash, we'd need to have some way to mark segments as "please don't free this".  Ensuring that we've marked all relevant memory might be tough.

This won't guarantee that the crash reporter won't OOM: If some other process gobbles up the memory we free'd before we get to it, we may still not have sufficient memory to build the crash report.  But it should be a significant improvement.

Comment 1

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Are the other threads paused at this point?

This is all pretty terrifying, but if we wanted to forcibly deallocate a chunk of memory, I think we should drop the JS GC heap.  That pretty much guarantees that the crashreporter itself will be ok.

Comment 2

[(not currently active) Ted Mielczarek]

Do we have that at a known address that we could just drop on the floor? Apparently we're seeing a spike in empty minidumps now, so we'd like to find a quick fix we could use to see if we can band-aid the problem.

Comment 3

[(not currently active) Ted Mielczarek]

We solved this on Windows in bug 837835 by using VirtualAlloc to reserve a block of virtual memory that we then free when an exception is hit (but before writing a minidump). bug 943051 fixed it to actually work, and it seems to have reduced our empty dump reports by quite a bit.

Comment 4

[(not currently active) Ted Mielczarek]

I'm going to go ahead and say we don't need this in light of that.