652438 - TM: Crash [@ JSString::isLinear] on out-of-memory

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect)

Description

[Christian Holler (:decoder)]

The following testcase (IMPORTANT: run with "-A 500 -m -j" to limit the memory usage) crashes on TM revision a03a4fea1679 (tested on 32 bit):

function f()
{
}
function dosubst()
{
  var s = f;
  for (var i = 0; i < 18; i++)
  {
    s += s;
  }
  var index = s.indexOf(f);
  while(true) {
    dosubst();
  }
}
dosubst();


Backtrace (looks like a safe null-pointer deref):

out of memory
==2747== Invalid read of size 4
==2747==    at 0x807BBEE: JSString::isLinear() const (jsstr.h:289)
==2747==    by 0x807BEBC: JSLinearString::chars() const (jsstr.h:421)
==2747==    by 0x807BFAB: JSString::getChars(JSContext*) (jsstr.h:697)
==2747==    by 0x81B228E: str_indexOf(JSContext*, unsigned int, js::Value*) (jsstr.cpp:1448)
==2747==    by 0x57B1788: ???
==2747==    by 0x82857D0: js::mjit::EnterMethodJIT(JSContext*, JSStackFrame*, void*, js::Value*) (MethodJIT.cpp:684)
==2747==    by 0x82858F5: CheckStackAndEnterMethodJIT(JSContext*, JSStackFrame*, void*) (MethodJIT.cpp:713)
==2747==    by 0x82859CA: js::mjit::JaegerShot(JSContext*) (MethodJIT.cpp:730)
==2747==    by 0x832E3A9: js::Interpret(JSContext*, JSStackFrame*, unsigned int, JSInterpMode) (jsinterp.cpp:4699)
==2747==    by 0x8234FFC: js::RecordTracePoint(JSContext*, js::TraceMonitor*, unsigned int&, bool*, bool) (jstracer.cpp:16741)
==2747==    by 0x82356CD: js::MonitorTracePoint(JSContext*, unsigned int&, bool*, void**, unsigned int*, unsigned int*, unsigned int) (jstracer.cpp:16898)
==2747==    by 0x82F2984: RunTracer(js::VMFrame&, js::mjit::ic::TraceICInfo&) (InvokeHelpers.cpp:981)
==2747==  Address 0x0 is not stack'd, malloc'd or (recently) free'd

Comment 1

[Luke Wagner [:luke]]

Attachment: fix

Well, it looks like in the heat of bug 613457 I seem to have "refactored" out the OOM check I added in bug 609440.  Sheesh, you'd think if anyone would know about fallible string chars...

Thanks for finding this!  The pesky string limits make it hard to make a fast-running shell test case that doesn't require special flags.

Comment 2

[Luke Wagner [:luke]]

http://hg.mozilla.org/tracemonkey/rev/a3e69f698ce3

Comment 3

[Chris Leary [:cdleary] (not checking bugmail)]

http://hg.mozilla.org/mozilla-central/rev/a3e69f698ce3

Comment 4

[Daniel Veditz [:dveditz]]

Comment on attachment 528058 [details] [diff] [review]
fix

We should consider landing this safe fix in Firefox 5: see bug 659920 comment 7

Comment 5

[Luke Wagner [:luke]]

It's definitely low risk.

Comment 6

[Asa Dotzler [:asa]]

Comment on attachment 528058 [details] [diff] [review]
fix

too late in the Beta cycle for non critical bugs.