559483 - MMGC allocations close to 4GB will OOM on canFail calls

Status: RESOLVED WONTFIX

(Tamarin Graveyard :: Garbage Collection (mmGC), defect)

Description

[Gavin]

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_2; en-us) AppleWebKit/531.22.7 (KHTML, like Gecko) Version/4.0.5 Safari/531.22.7
Build Identifier: 

THe MMGC has various checks for allocations near 4GM that force aborts. if the allocation is a canFail allocate, it should return null instead. The APi should allow for future 64 bit platform allocations. As long as there is a limit, there should be a method to get the maximum valid allocation size so that calling code can avoid erroneous values (and throw exceptions) -- Update ByteArray glue to do the right thing when this gets fixed.

Reproducible: Always

Steps to Reproduce:
1. set length of a ByteArray to 4GB without a workaround in the ByteArray glue.
2.
3.
Actual Results:  
Calls Abort and displays OOM triangle

Expected Results:  
returns NULL.

Comment 1

[Felix S. Klock II [:pnkfelix, :fklock]]

Attachment: avmshell program concretely illustrating the bug

This is my attempt at a test case to illustrate the problem more concretely.  The code illustrates the current boundaries where the behavior switches between three different cases (at least for a 64 bit build): at the most extreme of length >= 4 GB = 4294967296, the length is truncated to 0, presumably because we don't actually support such a large uint, at least in the current 64 bit build.  At length <= 4294963199 = 4 GB - 4097, the byte array is set to the indicated length (with a nice long pause, presumably because we're explicitly setting the bytes to 0 rather than allow uninitialized bits to leak through, but that's just an educated guess on my part).  In the space in between these extremes, we get the OOM.

Here's the output I get on my 64-bit Mac build:
bash-3.2$ java -jar /Users/fklockii/Dev/asc/lib/asc.jar -import /Users/fklockii/Dev/tamarin-redux/objdir-64/../core/builtin.abc -import /Users/fklockii/Dev/tamarin-redux/objdir-64/../shell/shell_toplevel.abc -d -strict bytes-oom.as

bytes-oom.abc, 730 bytes written
bash-3.2$ /Users/fklockii/Dev/tamarin-redux/objdir-64/shell/avmshell bytes-oom.abc
attempt 4GB
4GB: 4294967296 bytearray.length: 0
attempt 3GB+1023MB+1019KB+1023
3GB+1023MB+1019KB+1023: 4294963199 bytearray.length: 4294963199
attempt 3GB+1023MB+1020KB
Implementation limit exceeded: attempting to allocate too-large object
error: out of memory
OUT OF MEMORY


(Feel free to let me know if I have misinterpreted Gavin's description.  I believe there are separate issues: updating the ByteArrayGlue to use an interface supporting the canFail flag, and also making sure that the mmgc code respects canFail.)

Comment 2

[Felix S. Klock II [:pnkfelix, :fklock]]

Ah, the ByteArrayGlue.cpp in tamarin-redux, which is what avmshell uses, may not be synced up with the flash player source.  Need to check with the team about that.

Comment 3

[Steven Johnson]

Unifying those two is a long-standing issue we should really tackle.

Comment 4

[Gavin]

The player version currently works around the mmgc OOM with the following check
in Grow()

        if (minimumCapacity > (MMgc::GCHeap::kMaxObjectSize -
MMgc::GCHeap::kBlockSize*2))
            ThrowMemoryError();

When this is fixed in mmgc, this test should be removed. This only prevents
setting the size to a large value, I think you could still generate the error
by creating the bytearray with large size.

Comment 5

[Felix S. Klock II [:pnkfelix, :fklock]]

Okay, so that change is just meant as a temporary measure.  Maybe my filing bug 564248 was premature.

Comment 6

[Felix S. Klock II [:pnkfelix, :fklock]]

The workaround (for mmgc OOM that gavin is referring to) was put in to fix Watson 2597057.

The workaround itself was flashfarm CL 660620

Comment 7

[Sylvestre Ledru [:Sylvestre]]

Tamarin is a dead project now. Mass WONTFIX.

Comment 8

[Sylvestre Ledru [:Sylvestre]]

Tamarin isn't maintained anymore. WONTFIX remaining bugs.