818655 - BaselineCompiler: Add stack checks to raise over-recursion error

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect)

Description

[Kannan Vijayan [:djvj]]

Not performing stack checks in baseline-jitted code caused jit-test/tests/basic/testOverflowOOMFixupArity.js to fail.

Add stack checks to baseline jitcode.

Comment 1

[Kannan Vijayan [:djvj]]

Attachment: Add StackCheck IC to do stack checks.

Adds an IC to do stack checks.  The IC is simple in that it has no optimized stubs.  There's an inline check of the stack pointer against ionStackLimit.  If the check fails, we enter the IC (which contains only the fallback stub), which does the JS recursion check unconditionally.

Comment 2

[Kannan Vijayan [:djvj]]

Originally, I attempted to do this by emitting the vmcall for the stack check C++ function inline in the main jitcode.

TO help with this, I generalized |EmitCreateStubFrameDescriptor| to just |EmitCreateFrameDescriptor| taking an extra FrameType argument, and some other interface changes in BaselineHelpers.

It ended up being difficult to emit the vmcall inline anyway (mainly because for baseline jitcode, the frame iterator needs to be able to map return addresses to ICEntries).

So the changes to BaselineHelpers-* are not used.  If you think they're undesirable I can take them out, but personally I don't feel that they hurt even if they've become residual in this case.

Comment 3

[Jan de Mooij [:jandem]]

Comment on attachment 688922 [details] [diff] [review]
Add StackCheck IC to do stack checks.

Review of attachment 688922 [details] [diff] [review]:
-----------------------------------------------------------------

Cool.

::: js/src/ion/x86/BaselineHelpers-x86.h
@@ +93,5 @@
>  {
>      // Compute stub frame size. We have to add two pointers: the stub reg and previous
>      // frame pointer pushed by EmitEnterStubFrame.
>      masm.movl(BaselineFrameReg, reg);
>      masm.addl(Imm32(sizeof(void *) * 2), reg);

Passing the FrameType as argument makes sense, but the 2 pointers we add here are specific to the stub frames. We could add this as argument but for now it seems best to revert until we need these changes.

Comment 4

[Kannan Vijayan [:djvj]]

BaselineHelpers changes undone.  Other changes pushed:
https://hg.mozilla.org/projects/ionmonkey/rev/542abe927012