Created attachment 531111 [details]
The attached script is array2 from emscripten's test suite. With |-m -n| on JM, it enters an apparently infinite loop, and memory usage explodes out of control (be careful when running the script). Works ok with just -m.
Can you still reproduce this? Works for me with 32-bit and 64-bit OS X shell, both |-m -n| and |-m -a -n|.
I still see this on Linux 32. Might be a platform specific thing then.
I get the same on most other emscripten tests as well (this is among the first in the list, so I saw it first).
Ah this is interesting, debug build works on Linux 32-bit, but I can reproduce with a release build. Linux 32-bit tinderbox has the same problem, orange caused by time-outs. I'll reduce this now.
Created attachment 531766 [details] [diff] [review]
There are different versions of JS_FLOOR_LOG2 macro, apparently the optimized version returned an invalid value if the result is signed. The register was not in freeMask, causing an infinite loop in FrameState::merge (it was busy generating code, causing the memory explosion).
Comment on attachment 531766 [details] [diff] [review]
Weird, any other hits on this? possible to put a comment on the macro?
I think the fix is correct, but it looks like JS_FLOOR_LOG2 is a red herring. The problem seems to be somewhere outside peekReg. Probably a bug in GCC 4.5 with -O3, I'll look closer tomorrow.
Created attachment 531880 [details] [diff] [review]
The (RegisterID)ireg cast was confusing GCC when ireg is a FP register. So maybe it's not a GCC bug but undefined behavior. We don't even need the int -> unsigned change but I left it in, it seems to be the right type.
There's another (RegisterID) cast in FrameState.cpp, but it's valid because RegisterID ranges from 0 to TotalRegisters.