491734 - Hookup OOM for MMGC_USE_SYSTEM_MALLOC defined case

Status: VERIFIED FIXED

(Tamarin Graveyard :: Garbage Collection (mmGC), defect, P1)

Description

[Rishit Shah]

Currently, when MMGC_USE_SYSTEM_MALLOC is define the global operator new invokes VMPI_alloc that could return NULL.

OOM hooks need to be added under this case to abort if VMPI_alloc returns NULL.

Comment 1

[Lars T Hansen]

What do clients (Air?) assume if VMPI_alloc returns NULL?

Should we hook into our OOM system - run GC, decommit pages, etc; abort in the last instance?

Basically remains investigative for now.

We need to figure out how the profiler deals with this and if the profiler should be able to shut down the system or if it should just shut itself off.

Comment 2

[Lars T Hansen]

Attachment: Patch

This fixes the problem by creating a hook in GCHeap for SystemNew to signal OOM; restructuring the code around SystemNew a little bit; and it also fixes one bug where SystemDelete would call a profiling method on a NULL pointer during shutdown.

Comment 3

[Dan Schaffer]

ran the vm testsuite with AVMFEATURE_USE_SYSTEM_MALLOC with and without the patch.  In mainline (without patch) the mmgc/outofmemory.as testcase fails due to the vm crashing.  With the patch the testcase passes since the vm exits gracefully when OOM is reached with message error: out of memory and an exit code 128.

Comment 4

[Lars T Hansen]

redux changeset:   2614:1d34004af549