525028 - Crash [@ js_Interpret] reading from 0x30000006 or similar

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect)

Description

[Jason Orendorff [:jorendorff]]

On this line of code in js/src/jsops.cpp:
    1486 do_getprop_with_obj:
    1487   do {
    1488     JSObject *aobj;
    1489     JSPropCacheEntry *entry;
    1490
    1491     aobj = js_GetProtoIfDenseArray(cx, obj);
 >> 1492     if (JS_LIKELY(aobj->map->ops->getProperty == js_GetProperty)) { 

From the beta:
http://crash-stats.mozilla.com/report/index/4593b607-5b65-4e1d-ae76-c81602091026
http://crash-stats.mozilla.com/report/index/766a7e12-6021-4a90-8136-fab202091027

From nightlies:
http://crash-stats.mozilla.com/report/index/23f82a44-0843-4d00-aa32-328732091028
http://crash-stats.mozilla.com/report/index/d302acb6-049a-489c-990f-318132091027

Apparently Windows-only.

0x30000006 looks like a JSString::length to me (atomized deflated string of length 6), suggesting aobj points to a JSString. A handful of the other crashes are at other addresses that also look like JSString::length values:

0x30000005
http://crash-stats.mozilla.com/report/index/2af30fc3-b7bb-420d-90e0-1792e2091028

0x20000002
http://crash-stats.mozilla.com/report/index/766a7e12-6021-4a90-8136-fab202091027

But they all have a stack that shows js_fun_apply being called recursively.

Comment 1

[David Mandelin [:dmandelin]]

There is a Windows minidump saved for this one:

http://crash-stats.mozilla.com/report/index/5e8d46b8-d241-4c66-b937-2dab52090928

Comment 2

[David Mandelin [:dmandelin]]

The first trunk crash of this kind, in both crash time and build time, is this:

http://crash-stats.mozilla.com/report/index/4dea5e4e-8ff7-40c9-a73e-b5d702090827

The build date is 08-26-2009. Following that, there's a crash for about every build and every day, so most likely the problem came in on 8-25. Sadly, 132 TM changesets hit m-c that day:

http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=a0b2a2ffa124&tochange=189759c41621

Comment 3

[David Mandelin [:dmandelin]]

From the minidump, I found that it is crashing on the second load (reading the field 'ops' from aobj->map. I suspect it's related to tracing arguments |arguments| because:

- There are several patches to fix bugs in tracing arguments in the merge I listed above.
- Previously, bugs relating to arguments have caused values to get boxed with the wrong type tags (e.g., a JSString* getting boxed up as a JSObject*).
- fun_apply is on the stack, and fun_apply is expected to be seen when |arguments| is being used.

Comment 4

[David Mandelin [:dmandelin]]

I read over my arguments patches and they seemed unlikely causes: they are all minor bug fixes. Based on advice from dvander, I'm now looking more closely at bug 510518 and bug 509093.

Comment 5

[Tony Chung [:tchung]]

As mentioned in bug 519363 comment 30, it seems like my repro steps there can apply to a testcase for this stack trace.   Tracking it here.

Comment 6

[David Mandelin [:dmandelin]]

I've bisected the problem to here:

changeset:   31470:8702299aa4dc
user:        Igor Bukanov <[e-mail]>
date:        Fri Aug 14 20:43:16 2009 +0200
description:
bug 495061 -  js_PutArgsObject and js_PutCallObject that never fail. r=brendan

Also, the problem occurs only if tracing is enabled. Still working on the exact cause.

Comment 7

[David Mandelin [:dmandelin]]

Forgot to add Igor yesterday. Btw, this bug is pretty fiendish--I can repro only in optimized builds and it goes away if I add a printf to print when tracing starts. I found that there is a certain JSOP_ARGUMENTS that if I make it not trace, the crash does not occur, but the native code for that JSOP_ARGUMENTS does seem to be executed (so presumably it is something from earlier on in the trace).

Comment 8

[David Mandelin [:dmandelin]]

OK, this is geting weird.

+ nocrash if: I make all JSOP_ARGUMENTS return JSRS_STOP immediately
- crash if: If I make all JSOP_ARGUMENTS return JSRS_STOP immediately except for a certain one, (identified by filename and bytecode pos)
- crash if: I make all JSOP_ARGUMENTS return JSRS_STOP immediately, and I have the special one record LIR and then return JSRS_STOP. 
- crash if: I only trace the special one, and for that one I only run 3 LIR generating statements and then return JSRS_STOP.
+ no crash if: I only trace the special one, and for that one I only run *2* LIR generating statements and then return JSRS_STOP.
- crash if: I remove just the one LIR statement I removed for the test described on the last line (and only trace the special one, and return stop).

Especially given that the trace doesn't ever run, what could be the variable here that makes it crash or not? The amount of LIR generated is the obvious idea, but I have no idea if that's right. And why the amount of LIR generated would matter is also strange. Random ideas:

- generating more LIR causes a buffer overrun that mangles a typemap or something similar.
- generating more LIR causes a record buffer OOM that in turn causes something bad to happen
- ???

If anyone has any ideas, let me know. I don't think either of the above are very promising.

Another idea I had is assert that object tag bits are not placed on the value 0x30000006, in the hope of catching something interesting sooner.

Comment 9

[David Anderson [:dvander] - inactive, e-mail if emergency]

Managed to test case this based on dmandelin's debugging efforts:

function f() {
    var _L1 = arguments;
    for (var i = 0; i < _L1.length; i++) {
        if (typeof _L1[i] == "string")
            _L1[i] = new Object();
    }
}

f(1, 2, "3", 4, 5);
f(1, 2, "3", 4, 5);
f(1, 2, "3", 4, 5);
f(1, 2, "3", 4, 5);

With TMFLAGS=full, wrong type tag screws up:

ntering trace at x.js:3@14, native stack slots: 15 code: 0x2ebeeb
args0: function<0x2d77e0:f> args1: object<0x2d9000:global> args2: int<1> args3: int<2> args4: string<0x1d8018> args5: int<4> args6: int<5> arguments0: object<0x513280:Object> var0: object<0x513280:Object> var1: int<0> global0: function<0x2d8150:Object> 
leaving trace at x.js:3@14, op=trace, lr=0x8503ac, exitType=UNSTABLE_LOOP, sp=0, calldepth=0, cycles=0
args0=function<0x2d77e0:f> args1=object<0x2d9000:global> args2=int<1> args3=int<2> args4=
Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x00000030

Comment 10

[David Mandelin [:dmandelin]]

++dvander -- I thought this bug was way too fragile to be test-cased like that.

Comment 11

[David Mandelin [:dmandelin]]

Attachment: Patch

Setting properties of an |arguments| object that corresponds to an on-trace frame doesn't work (and wasn't intended to work). The reason it's hard is that it can alter the type of local variables, which would mess up the tracker. The original code protected against this in ArgSetter by deep bailing if the arguments object had a private member that referred to a trace frame.

The problem is that that check doesn't cover the case of an |arguments| object created in the interpreter and then used as part of the trace entry frame. This patch fixes that problem by deep bailing always.

Comment 12

[David Mandelin [:dmandelin]]

http://hg.mozilla.org/tracemonkey/rev/06101a9ca642
http://hg.mozilla.org/tracemonkey/rev/6667543dbe27

(I forgot to hg add the test case in the first commit.)

Comment 13

[Jason Orendorff [:jorendorff]]

Nice work here. Wow.

Comment 15

[David Mandelin [:dmandelin]]

http://hg.mozilla.org/releases/mozilla-1.9.2/rev/427aba6240c0

Comment 16

[Robert Sayre]

http://hg.mozilla.org/mozilla-central/rev/06101a9ca642
http://hg.mozilla.org/mozilla-central/rev/6667543dbe27
http://hg.mozilla.org/mozilla-central/rev/29c7e9f73f7a