1411302 - Assertion failure: false (owner_.compareExchange(nullptr, this)), at dist/include/js/Utility.h:327 or various crashes

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect)

Description

[Christian Holler (:decoder)]

The following testcase crashes on mozilla-central revision d49501f258b1 (build with --enable-posix-nspr-emulation --enable-valgrind --enable-gczeal --disable-tests --enable-stdcxx-compat --disable-profiling --enable-debug --enable-optimize, run with --fuzzing-safe --cpu-count=2):

let lfPreamble = `
    value:{
`;
try {
    evaluate("");
    evalInWorker("");
} catch (exc) {}
try {
    evalInWorker("");
} catch (exc) {}
try {
    oomTest(function() {
        eval("function testDeepBail1() {");
    });
} catch (exc) {}


Backtrace:

received signal SIGSEGV, Segmentation fault.
#0  0x0000000000475a2c in js::AutoEnterOOMUnsafeRegion::AutoEnterOOMUnsafeRegion (this=0x7ffff4dfdfc0) at dist/include/js/Utility.h:327
#1  0x0000000000e70c2e in JS::Zone::getUniqueIdInfallible (this=this@entry=0x7ffff69db000, cell=0x7ffff4707040) at js/src/gc/Zone.h:615
#2  0x0000000000e71217 in js::MovableCellHasher<JSObject*>::match (k=@0x7ffff4dfe0a0: 0x7ffff4707040, l=@0x7ffff4dfe098: 0x7ffff4707040) at js/src/gc/Barrier.cpp:195
#3  0x0000000000bff602 in js::MovableCellHasher<js::TaggedProto>::match (l=..., k=...) at js/src/vm/TaggedProto.h:82
#4  js::ObjectGroupCompartment::NewEntry::match (lookup=..., key=...) at js/src/vm/ObjectGroup.cpp:418
#5  js::detail::HashTable<js::ObjectGroupCompartment::NewEntry const, js::HashSet<js::ObjectGroupCompartment::NewEntry, js::ObjectGroupCompartment::NewEntry, js::SystemAllocPolicy>::SetOps, js::SystemAllocPolicy>::match (l=..., e=...) at /srv/jenkins/jobs/mozilla-central-build-jsshell/workspace/arch/64/compiler/gcc/instrumentation/none/type/debug/dist/include/js/HashTable.h:1378
#6  js::detail::HashTable<js::ObjectGroupCompartment::NewEntry const, js::HashSet<js::ObjectGroupCompartment::NewEntry, js::ObjectGroupCompartment::NewEntry, js::SystemAllocPolicy>::SetOps, js::SystemAllocPolicy>::lookup (this=this@entry=0x7ffff69471a0, l=..., keyHash=keyHash@entry=2879862646, collisionBit=collisionBit@entry=1) at /srv/jenkins/jobs/mozilla-central-build-jsshell/workspace/arch/64/compiler/gcc/instrumentation/none/type/debug/dist/include/js/HashTable.h:1406
#7  0x0000000000bff715 in js::detail::HashTable<js::ObjectGroupCompartment::NewEntry const, js::HashSet<js::ObjectGroupCompartment::NewEntry, js::ObjectGroupCompartment::NewEntry, js::SystemAllocPolicy>::SetOps, js::SystemAllocPolicy>::lookupForAdd (this=this@entry=0x7ffff69471a0, l=...) at /srv/jenkins/jobs/mozilla-central-build-jsshell/workspace/arch/64/compiler/gcc/instrumentation/none/type/debug/dist/include/js/HashTable.h:1779
#8  0x0000000000bffefc in js::HashSet<js::ObjectGroupCompartment::NewEntry, js::ObjectGroupCompartment::NewEntry, js::SystemAllocPolicy>::lookupForAdd (l=..., this=0x7ffff69471a0) at dist/include/js/HashTable.h:408
#9  JS::WeakCache<JS::GCHashSet<js::ObjectGroupCompartment::NewEntry, js::ObjectGroupCompartment::NewEntry, js::SystemAllocPolicy> >::lookupForAdd (this=0x7ffff6947180, l=...) at dist/include/js/GCHashTable.h:701
#10 0x0000000000bf78d3 in js::ObjectGroup::lazySingletonGroup (cx=cx@entry=0x7ffff69c3000, clasp=<optimized out>, proto=...) at js/src/vm/ObjectGroup.cpp:622
#11 0x0000000000a4543c in JSObject::setSingleton (obj=..., cx=0x7ffff69c3000) at js/src/jsobjinlines.h:160
#12 NewObject (cx=0x7ffff69c3000, group=..., kind=js::gc::AllocKind::FIRST, newKind=js::SingletonObject, initialShapeFlags=<optimized out>) at js/src/jsobj.cpp:739
#13 0x0000000000a45af6 in js::NewObjectWithClassProtoCommon (cx=cx@entry=0x7ffff69c3000, clasp=clasp@entry=0x1f29420 <JSFunction::class_>, protoArg=..., protoArg@entry=..., allocKind=allocKind@entry=js::gc::AllocKind::FIRST, newKind=newKind@entry=js::SingletonObject) at js/src/jsobj.cpp:852
#14 0x00000000009f387b in js::NewObjectWithClassProto (newKind=js::SingletonObject, allocKind=js::gc::AllocKind::FIRST, proto=..., clasp=0x1f29420 <JSFunction::class_>, cx=0x7ffff69c3000) at js/src/jsobjinlines.h:687
#15 js::NewObjectWithClassProto<JSFunction> (newKind=js::SingletonObject, allocKind=js::gc::AllocKind::FIRST, proto=..., cx=0x7ffff69c3000) at js/src/jsobjinlines.h:712
#16 js::NewFunctionWithProto (cx=0x7ffff69c3000, native=0xb37e50 <DebuggerScript_clearAllBreakpoints(JSContext*, unsigned int, JS::Value*)>, nargs=0, flags=flags@entry=JSFunction::NATIVE_FUN, enclosingEnv=enclosingEnv@entry=..., atom=..., proto=..., allocKind=js::gc::AllocKind::FIRST, newKind=js::SingletonObject) at js/src/jsfun.cpp:2083
#17 0x000000000060fd76 in js::NewNativeFunction (cx=<optimized out>, native=<optimized out>, nargs=<optimized out>, atom=..., atom@entry=..., allocKind=allocKind@entry=js::gc::AllocKind::FIRST, newKind=newKind@entry=js::SingletonObject) at js/src/jsfun.h:709
#18 0x0000000000990658 in JS::NewFunctionFromSpec (cx=0x7ffff69c3000, fs=fs@entry=0x1f2f580 <DebuggerScript_methods+320>, id=id@entry=...) at js/src/jsapi.cpp:3633
#19 0x0000000000a3fb2f in DefineFunctionFromSpec (intrinsic=js::NotIntrinsic, flags=0, fs=0x1f2f580 <DebuggerScript_methods+320>, obj=..., cx=0x7ffff69c3000) at js/src/jsobj.cpp:3075
#20 js::DefineFunctions (cx=cx@entry=0x7ffff69c3000, obj=..., fs=0x1f2f580 <DebuggerScript_methods+320>, fs@entry=0x1f2f440 <DebuggerScript_methods>, intrinsic=intrinsic@entry=js::NotIntrinsic) at js/src/jsobj.cpp:3091
#21 0x00000000009960f6 in JS_DefineFunctions (cx=cx@entry=0x7ffff69c3000, obj=..., fs=fs@entry=0x1f2f440 <DebuggerScript_methods>) at js/src/jsapi.cpp:3849
#22 0x0000000000b9854b in js::DefinePropertiesAndFunctions (cx=0x7ffff69c3000, obj=obj@entry=..., ps=ps@entry=0x1f2f620 <DebuggerScript_properties>, fs=fs@entry=0x1f2f440 <DebuggerScript_methods>) at js/src/vm/GlobalObject.cpp:609
#23 0x0000000000a68ba4 in DefineConstructorAndPrototype (ctorKind=js::gc::AllocKind::FIRST, ctorp=0x0, static_fs=0x0, static_ps=0x0, fs=0x1f2f440 <DebuggerScript_methods>, ps=0x1f2f620 <DebuggerScript_properties>, nargs=<optimized out>, constructor=0xb19a60 <DebuggerScript_construct(JSContext*, unsigned int, JS::Value*)>, clasp=0x1f2fee0 <DebuggerScript_class>, protoProto=..., atom=..., key=JSProto_Null, obj=..., cx=0x7ffff69c3000) at js/src/jsobj.cpp:1918
#24 js::InitClass (cx=0x7ffff69c3000, obj=..., obj@entry=..., protoProto_=..., clasp=clasp@entry=0x1f2fee0 <DebuggerScript_class>, constructor=constructor@entry=0xb19a60 <DebuggerScript_construct(JSContext*, unsigned int, JS::Value*)>, nargs=nargs@entry=0, ps=0x1f2f620 <DebuggerScript_properties>, fs=0x1f2f440 <DebuggerScript_methods>, static_ps=0x0, static_fs=0x0, ctorp=0x0, ctorKind=js::gc::AllocKind::FIRST) at js/src/jsobj.cpp:1976
#25 0x0000000000b4fa04 in JS_DefineDebuggerObject (cx=0x7ffff69c3000, obj=...) at js/src/vm/Debugger.cpp:11528
#26 0x000000000046d48c in NewGlobalObject (cx=0x7ffff69c3000, options=..., principals=principals@entry=0x0) at js/src/shell/js.cpp:7927
#27 0x000000000046de1f in WorkerMain (arg=<optimized out>) at js/src/shell/js.cpp:3597
#28 0x0000000000474f12 in js::detail::ThreadTrampoline<void (&)(void*), WorkerInput*&>::callMain<0ul> (this=0x7ffff691e160) at js/src/threading/Thread.h:239
#29 js::detail::ThreadTrampoline<void (&)(void*), WorkerInput*&>::Start (aPack=0x7ffff691e160) at js/src/threading/Thread.h:232
#30 0x00007ffff7bc16fa in start_thread (arg=0x7ffff4dff700) at pthread_create.c:333
#31 0x00007ffff6c38b5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
rax	0x0	0
rbx	0x1f564d8	32859352
rcx	0x7ffff6c28a2d	140737333332525
rdx	0x0	0
rsi	0x7ffff6ef7770	140737336276848
rdi	0x7ffff6ef6540	140737336272192
rbp	0x7ffff4dfdfa0	140737301700512
rsp	0x7ffff4dfdf70	140737301700464
r8	0x7ffff6ef7770	140737336276848
r9	0x7ffff4dff700	140737301706496
r10	0x58	88
r11	0x7ffff6b9f750	140737332770640
r12	0x7ffff69db000	140737330917376
r13	0x7ffff4707040	140737294397504
r14	0x7ffff69db000	140737330917376
r15	0xaba73776	2879862646
rip	0x475a2c <js::AutoEnterOOMUnsafeRegion::AutoEnterOOMUnsafeRegion()+204>
=> 0x475a2c <js::AutoEnterOOMUnsafeRegion::AutoEnterOOMUnsafeRegion()+204>:	movl   $0x0,0x0
   0x475a37 <js::AutoEnterOOMUnsafeRegion::AutoEnterOOMUnsafeRegion()+215>:	ud2



The test in comment 0 is slightly intermittent, you might have to run it a few times. Overall, this issue is highly frequent though and causing a whole series of crashes, some of them look s-s.

Comment 1

[Jan de Mooij [:jandem]]

Jon do you know if AutoEnterOOMUnsafeRegion's threading behavior changed recently?

Comment 2

[Jon Coppeard (:jonco)]

Bug 1406455 disallowed OOM simulation on workers so that seems related.  Maybe not all worker threads end up with the new worker thread type.

Comment 3

[Jon Coppeard (:jonco)]

No, it's not that.  The crashing thread has thread type THREAD_TYPE_WORKER and js::oom::targetThread is THREAD_TYPE_NONE.  Maybe it's because there is no synchronisation between the OOM test infrastructure and worker threads.

Comment 4

[Fuzzing Team]

JSBugMon: Bisection requested, result:
autoBisect shows this is probably related to the following changeset:

The first bad revision is:
changeset:   https://hg.mozilla.org/mozilla-central/rev/f160ec9ff8bd
user:        Kevin Pellet (Ilphrin)
date:        Wed Oct 11 19:06:10 2017 -0400
summary:     Bug 1406488 - Use a set instead of array to store current visits in `_recordToPlaceInfo` r=kitcambridge

This iteration took 272.849 seconds to run.

Comment 5

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

The bisection window in comment 4 is likely not valid due to the intermittent testcase.

Comment 6

[Fuzzing Team]

JSBugMon: Cannot process bug: Unable to automatically reproduce, please track manually.

Comment 7

[Christian Holler (:decoder)]

Jon, can you fix this? If not, please assign this fuzzblocker to someone else. Thanks.

Comment 8

[Jon Coppeard (:jonco)]

(In reply to Christian Holler (:decoder) from comment #7)
I didn't see this was a fuzzblocker.  I'll take another look.

Comment 9

[Jon Coppeard (:jonco)]

Attachment: bug1411302-oom-test-worker

The problem is that running oomTest() on the main thread will attempt to simulate OOM on worker threads, which belong to a different runtime and execute concurrently.  This is the opposite problem to bug 1406455 where we disabled running oomTest() on worker threads because it would attempt to simulate OOM on the main thread.

The fix is to not simulate OOM on workers.

Comment 10

[Jan de Mooij [:jandem]]

Comment on attachment 8922812 [details] [diff] [review]
bug1411302-oom-test-worker

Review of attachment 8922812 [details] [diff] [review]:
-----------------------------------------------------------------

Nice, hopefully this will fix bug 1412285 too.

::: js/public/Utility.h
@@ +86,5 @@
> +
> +// Define the range of threads tested by simulated OOM testing and the
> +// like. Testing worker threads is not supported.
> +const ThreadType FirstThreadTypeToTest = THREAD_TYPE_COOPERATING;
> +const ThreadType LastThreadTypeToTest = THREAD_TYPE_WASM_TIER2;

Might make sense to add a comment to the ThreadType enum, above THREAD_TYPE_WORKER, reminding people to update this constant when they add a new thread type. Or we could add THREAD_TYPE_SOMETHING = THREAD_TYPE_WASM_TIER2 to the enum.

Comment 11

[Jon Coppeard (:jonco)]

Unmarking s-s since this is shell only.

Comment 12

[Jon Coppeard (:jonco)]

(In reply to Jan de Mooij [:jandem] from comment #10)
Good idea, I'll add a comment.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/9641170641e5
Don't try and OOM test worker threads r=jandem

Comment 14

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/9641170641e5