Created attachment 603048 [details] [diff] [review] Required patch for JS OOM testing After applying the attached patch to the JS engine, the following command crashes/aborts on mozilla-central revision 4b728a090880: js/src/debug64-bt/js -m -n -a -A 1327 -f /home/decoder/LangFuzz/mozilla-central/js/src/jit-test/tests/debug/Object-defineProperty-06.js In order to build, use CFLAGS="-rdynamic -Wno-error" CXXFLAGS="-rdynamic -Wno-error" (I promise to make the patch more clean, it's just a hack right now^^). The patch adds two things: First it adds the JS_OOM_POSSIBLY_FAIL macro to LifoAlloc (which is required for this issue to reproduce more reliably) and secondly, it adds the possibility to print backtraces for alloc failures. Use MOZ_OOM_BTPRINT=1 MOZ_OOM_BTDEPTH=8 js/src/debug64-bt/js -m -n -a -A 1327 -f /home/decoder/LangFuzz/mozilla-central/js/src/jit-test/tests/debug/Object-defineProperty-06.js to see traces of allocation failures. This will hopefully make debugging OOM failures much easier. I'll also attach a small perl script that allows filtering the output to add symbols. In the future, the patch can hopefully be enhanced and merged to m-c, and the script will be part of a larger set of JS shell OOM testing tools. I'm filing this now because I keep hitting this bug extremely often and I don't know how to fix it.
Brian just pointed me to bug 732496 and the patch in there fixes this problem too.