734196 - Crash due to call stack overflow [@ js::ArrayBuffer::obj_trace]

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect)

Description

[Christian Holler (:decoder)]

The following test crashes on mozilla-central revision 453d5c733caa (options -m -n -a):


var x = new ArrayBuffer(2);
gczeal(4);
actual = [].concat(x).toString();
var count2 = countHeap();


In a debug build, the recursion looks like this:

#7  0x00000000005b7809 in js::ArrayBuffer::obj_trace (trc=0x7ffff7fae220, obj=0x7ffff61031a0) at /srv/repos/mozilla-central/js/src/jstypedarray.cpp:334
#8  0x0000000000445044 in js::ObjectImpl::privateWriteBarrierPre (this=0x7ffff61031a0, old=0x7ffff6103238) at ../vm/ObjectImpl-inl.h:105
#9  0x0000000000445653 in JSObject::setPrivate (this=0x7ffff61031a0, data=0x7ffff6105080) at ../jsobjinlines.h:119
#10 0x00000000005b781c in js::ArrayBuffer::obj_trace (trc=0x7ffff7fae220, obj=0x7ffff61031a0) at /srv/repos/mozilla-central/js/src/jstypedarray.cpp:335
#11 0x0000000000445044 in js::ObjectImpl::privateWriteBarrierPre (this=0x7ffff61031a0, old=0x7ffff6103238) at ../vm/ObjectImpl-inl.h:105
#12 0x0000000000445653 in JSObject::setPrivate (this=0x7ffff61031a0, data=0x7ffff6105080) at ../jsobjinlines.h:119
#13 0x00000000005b781c in js::ArrayBuffer::obj_trace (trc=0x7ffff7fae220, obj=0x7ffff61031a0) at /srv/repos/mozilla-central/js/src/jstypedarray.cpp:335

while in an opt build, it's just setPrivate and js::ArrayBuffer::obj_trace and the rest seems inlined.

Comment 1

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

This looks similar to bug 733372, Terrence. I think the best fix is to call setPrivate only if IS_GC_MARKING_TRACER(trc). Hopefully that fix would have worked for bug 733372 as well.

Comment 2

[Terrence Cole [:terrence]]

Attachment: v0: A complete fix.

I audited all call sites of setPrivate and modified all of the ones that could possibly exhibit this problem.  This should fix all possible instances of this particular problem now and in the future, without hindering our ability to move objects as we will need to later.

Comment 3

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Comment on attachment 604209 [details] [diff] [review]
v0: A complete fix.

Thanks!

Comment 4

[Terrence Cole [:terrence]]

http://hg.mozilla.org/integration/mozilla-inbound/rev/1587745454a6

Comment 5

[Marco Bonardo [:mak]]

https://hg.mozilla.org/mozilla-central/rev/1587745454a6

Comment 6

[Christian Holler (:decoder)]

A testcase for this bug was automatically identified at js/src/jit-test/tests/basic/bug734196.js.