624366 - Assertion failure: compartment mismatched, at jscntxtinlines.h:541

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect)

Description

[Martin Stránský [:stransky] (ni? me)]

FF 4.0b9pre
debug build

It crashes in firefox test:

./cfx testall -a firefox -b /dist/bin/firefox

..console: Invalid use of the preferences on a background thread!
console: Invalid use of the preferences on a background thread!
............................................WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file nsChromeRegistry.cpp, line 250
..............................................WARNING: Cannot be used on a background thread!: file nsPrefService.cpp, line 983
.....console: Invalid use of the preferences on a background thread!
......*** Compartment mismatch 0x7f7c383d7000 vs. 0x7f7c245ec000
Assertion failure: compartment mismatched, at jscntxtinlines.h:541

bactrace:
#2  0x00007f7c4f2994a8 in ah_crap_handler (signum=6) at nsSigHandlers.cpp:132
#3  0x00007f7c4f29dcb2 in nsProfileLock::FatalSignalHandler (signo=6, info=0x7fffb04a38f0, context=0x7fffb04a37c0)
    at nsProfileLock.cpp:226
#4  <signal handler called>
#5  0x00007f7c5215c36b in raise (sig=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/pt-raise.c:42
#6  0x00007f7c50c5b958 in JS_Assert (s=0x7f7c514754c4 "compartment mismatched", file=0x7f7c51475350 "jscntxtinlines.h", ln=
    541) at jsutil.cpp:83
#7  0x00007f7c50afa8cd in js::CompartmentChecker::fail (c1=0x7f7c383d7000, c2=0x7f7c245ec000) at jscntxtinlines.h:541
#8  0x00007f7c50afa9b9 in js::CompartmentChecker::check (this=0x7fffb04a3d50, c=0x7f7c245ec000) at jscntxtinlines.h:557
#9  0x00007f7c50afa9ff in js::CompartmentChecker::check (this=0x7fffb04a3d50, obj=0x7f7c24e4ba18) at jscntxtinlines.h:565
#10 0x00007f7c50afaa40 in js::CompartmentChecker::check (this=0x7fffb04a3d50, v=...) at jscntxtinlines.h:570
#11 0x00007f7c50afaa70 in js::CompartmentChecker::check (this=0x7fffb04a3d50, v=...) at jscntxtinlines.h:574
#12 0x00007f7c50afef58 in js::assertSameCompartment<JSObject*, jsval_layout, JSValueArray> (cx=0x7f7c34c4b400, t1=
    0x7f7c32515e38, t2=..., t3=...) at jscntxtinlines.h:652
#13 0x00007f7c50af2320 in JS_CallFunctionValue (cx=0x7f7c34c4b400, obj=0x7f7c32515e38, fval=..., argc=3, argv=
    0x7fffb04a3e50, rval=0x7fffb04a3eb8) at jsapi.cpp:5008
#14 0x00007f7c50449ff1 in mozilla::jetpack::JetpackActorCommon::RecvMessage (this=0x7f7c0c68cec8, cx=0x7f7c34c4b400, 
    messageName=..., data=..., results=0x7fffb04a4060) at JetpackActorCommon.cpp:474
#15 0x00007f7c5043f9a3 in mozilla::jetpack::JetpackParent::AnswerCallMessage (this=0x7f7c0c68cc00, messageName=..., data=
    ..., results=0x7fffb04a4060) at JetpackParent.cpp:242
#16 0x00007f7c507a9462 in mozilla::jetpack::PJetpackParent::OnCallReceived (this=0x7f7c0c68cc00, __msg=..., __reply=
    @0x7fffb04a4148) at PJetpackParent.cpp:374
#17 0x00007f7c506ce426 in mozilla::ipc::RPCChannel::DispatchIncall (this=0x7f7c0c68cc10, call=...) at RPCChannel.cpp:517
#18 0x00007f7c506ce33e in mozilla::ipc::RPCChannel::Incall (this=0x7f7c0c68cc10, call=..., stackDepth=0)
    at RPCChannel.cpp:503
#19 0x00007f7c506ce055 in mozilla::ipc::RPCChannel::OnMaybeDequeueOne (this=0x7f7c0c68cc10) at RPCChannel.cpp:434
#20 0x00007f7c506d3fee in void DispatchToMethod<mozilla::ipc::RPCChannel, bool (mozilla::ipc::RPCChannel::*)()>(mozilla::ipc::RPCChannel*, bool (mozilla::ipc::RPCChannel::*)(), Tuple0 const&) () from /home/komat/tmp590-b7/src/dist/bin/libxul.so
#21 0x00007f7c506d3f3e in RunnableMethod<mozilla::ipc::RPCChannel, bool (mozilla::ipc::RPCChannel::*)(), Tuple0>::Run() ()
   from /home/komat/tmp590-b7/src/dist/bin/libxul.so
#22 0x00007f7c506cfa11 in mozilla::ipc::RPCChannel::RefCountedTask::Run() ()
   from /home/komat/tmp590-b7/src/dist/bin/libxul.so
#23 0x00007f7c506cfb14 in mozilla::ipc::RPCChannel::DequeueTask::Run() () from /home/komat/tmp590-b7/src/dist/bin/libxul.so
#24 0x00007f7c5096a638 in MessageLoop::RunTask (this=0x7f7c3aa0d6a0, task=0x7f7c0bfd0380) at ./src/base/message_loop.cc:343
#25 0x00007f7c5096a6a8 in MessageLoop::DeferOrRunPendingTask (this=0x7f7c3aa0d6a0, pending_task=...)
    at ./src/base/message_loop.cc:351
#26 0x00007f7c5096aa8c in MessageLoop::DoWork (this=0x7f7c3aa0d6a0) at ./src/base/message_loop.cc:451
#27 0x00007f7c506cb8a7 in mozilla::ipc::DoWorkRunnable::Run (this=0x7f7c3aa09f70) at MessagePump.cpp:70
#28 0x00007f7c508ff44c in nsThread::ProcessNextEvent (this=0x7f7c461f3f00, mayWait=0, result=0x7fffb04a453c)
    at nsThread.cpp:633
#29 0x00007f7c508891d8 in NS_ProcessNextEvent_P (thread=0x7f7c461f3f00, mayWait=0) at nsThreadUtils.cpp:250
#30 0x00007f7c506cbb26 in mozilla::ipc::MessagePump::Run (this=0x7f7c3aa02500, aDelegate=0x7f7c3aa0d6a0)
    at MessagePump.cpp:110
#31 0x00007f7c5096a143 in MessageLoop::RunInternal (this=0x7f7c3aa0d6a0) at ./src/base/message_loop.cc:219
#32 0x00007f7c5096a0c8 in MessageLoop::RunHandler (this=0x7f7c3aa0d6a0) at ./src/base/message_loop.cc:202

Comment 1

[Andreas Gal :gal]

We can't block on this component. Bugzilla, you are just weird.

Comment 2

[Andreas Gal :gal]

This is really a jetpack bug, but if we still want jetpack to be compatible with 4 and we still block on that, than this has to block. Jetpack doesn't seem to have the ability to block product though. Weirdness.

Comment 3

[Andreas Gal :gal]

Do we have to make jetpack work with FF4? Is that still a mandatory product goal?

Comment 4

[chris hofmann]

myk, do you have the equivelent of jetpack blocking for the firefox 4 release?

Comment 5

[Andreas Gal :gal]

I looked at the code. It _seems_ to do the right thing (it enters a compartment), so this must be a compartment mismatch of arguments. We need a testcase for this.

Comment 6

[Myk Melez [:myk] [@mykmelez]]

chofmann: there isn't a special flag that says "this should block because it breaks Jetpack", I just use the regular blocking flags along with bug dependencies.  So when there's a bug in Gecko/Firefox that prevents the Add-on SDK from working correctly, I request that it block Gecko 2/Firefox 4, create an Add-on SDK bug about the problem, and make the Add-on SDK bug depend on the Gecko/Firefox bug.

Andreas: when you say that this is "really a Jetpack bug", do you mean that the problem is in the SDK codebase?

Martin: can you try running tests with the --verbose option?  The output from that should enable us to pinpoint the specific test triggering the assertion, which I'm not seeing.

Comment 7

[Brendan Eich [:brendan]]

We should have automated tests in 24:7 tinderbox coverage for any must-not-break stuff that ships with Firefox 4. Do we?

/be

Comment 8

[Andreas Gal :gal]

As per discussion with Myk, we will treat this bug and all bugs like it as a FF4 blocker until there is proof that its not a bug in the platform.

Comment 9

[Andreas Gal :gal]

I installed the latest jetpack from the repository. cfx testall runs 71 tests successfully and then says:

Ran 71 tests in 1.673s

OK
Testing reading-data...
Using binary at '/usr/bin/firefox'.
Using profile at '/tmp/tmp7m8c3d.mozrunner'.

and opens a firefox and then just sits there with the cursor waiting in an empty url bar.

Comment 10

[Andreas Gal :gal]

I get the same behavior if I run the tests against 3.6.13. Myk, this is why I wanted the SDK in our tree as a point of reference. It seems your tests are broken in your tree right now afaict and I have to juggle two trees trying to figure out whats going on.

I ran the tests on a linux box. I will try macosx next.

Comment 11

[Myk Melez [:myk] [@mykmelez]]

(In reply to comment #9)
> Using binary at '/usr/bin/firefox'.

Is /usr/bin/firefox the latest nightly build?  If it's 3.6.13, then it makes sense that the test run hangs at that point, as Add-on SDK does not support that version of Firefox.

Comment 12

[Myk Melez [:myk] [@mykmelez]]

Note: to make sure you are running the Add-on SDK test suite with the version of Firefox you want to test it with, use the --binary option, f.e.:

cfx testall --binary /home/me/builds/nightly/dist/bin/firefox
cfx testall --binary /Applications/Minefield.app/
cfx testall --binary C:\Program Files\Minefield\firefox.exe
cfx testall --binary /c/Program\ Files/Minefield/firefox

Comment 13

[Andreas Gal :gal]

Reproduced.

Comment 14

[Andreas Gal :gal]

471	  for (PRUint32 i = 0; i < snapshot.Length(); ++i) {
472	    Variant* vp = results ? results->AppendElement() : NULL;
473	    rval.set(JSVAL_VOID);
474	    if (!JS_CallFunctionValue(cx, implGlobal, snapshot[i], argc, argv,
475	                              rval.jsval_addr())) {
476	      (void) JS_ReportPendingException(cx);
477	      if (vp)
478	        *vp = void_t();
(gdb) p snapshot[i]
$9 = (jsval_layout &) @0x7fff8e361cb0: {asBits = 18445618044974160408, debugView = {payload47 = 140608659806744, tag = JSVAL_TAG_OBJECT}, s = {
    payload = {i32 = 20470296, u32 = 20470296, why = 20470296, word = 18445618044974160408}}, asDouble = -nan(0xbffe201385a18), 
  asPtr = 0xfffbffe201385a18}
(gdb) p JSVAL_TO_OBJECT(snapshot[i])
$10 = (JSObject *) 0x7fe201385a18
(gdb) p JSVAL_TO_OBJECT(snapshot[i])->compartment()
warning: can't find linker symbol for virtual table for `JSObject' value
warning: can't find linker symbol for virtual table for `JSObject' value
warning: can't find linker symbol for virtual table for `js::gc::Cell' value
warning: can't find linker symbol for virtual table for `js::gc::Cell' value
$11 = (JSCompartment *) 0x7fe200f70000
(gdb) p cx->compartment
$12 = (JSCompartment *) 0x7fe209567000
(gdb) p Killed

And thats where my debug session ended. There is a 300s limit on debug sessions in the harness. Looks like snapshot[i] is in the wrong compartment.

Comment 15

[Andreas Gal :gal]

Alright, I will have to double check with Myk but if I read the state correctly a background thread is doing IPC with the foreground thread and is shipping jsvals wrapped into a variant. That is no longer supported by the platform. Sharing JS heap objects between threads is illegal. This is a major API usage bug. It went undetected because we don't have jetpack tests on our tree (where we run debug builds that are able to catch this), and the jetpack guys probably don't ever run their stuff against debug builds (Martin did, thats why he caught this).

This will need major surgery to be fixed. I will confer with Myk. And someone please send Martin a t-shirt for saving our butts here and running a test we should have been running all along.

Comment 16

[Brendan Eich [:brendan]]

Comment 14 shows JS_ReportPendingException(cx)'s r.v. being ignored. Who reviewed this JS API client code?

/be

Comment 17

[Andreas Gal :gal]

I wasn't able to get hold of Myk, but for the test case that crashes here sender and receiver seem to be the same thread. Its two different compartments through. I fixed the call site and wrap my way out of this. I am now crashing in jetpack::EvalInSandbox. Looking into that next.

Comment 18

[Andreas Gal :gal]

I fixed the EvalInSandbox stuff and now I am dying in JetpackParent::SendMessage. Looking into it.

Comment 19

[Andreas Gal :gal]

Dying in TearDown now. Looks fixable, too. I will upload a patch in a bit. The fact that this was this severely broken and went unnoticed for this long is terrifying.

Comment 20

[Andreas Gal :gal]

Attachment: patch

Comment 21

[Andreas Gal :gal]

The attached patch fixes all compartment mismatches. With this I pass all jetpack tests with a linux debug build. This doesn't solve the threading issue. I did some code review but my mental image of the Jetpack architecture is too incomplete at this point to tell whether all compartments involved are always made on the main thread or not (they _seem_ to be all chrome compartments, so I am praying they are). I will check with Myk.

Comment 22

[Andreas Gal :gal]

Trying to figure out who the module owner is to get a review. In the meantime asking jst, he has done similar reviews elsewhere before.

Comment 23

[Andreas Gal :gal]

Looks like I was using an old version of the SDK from december. It moved to a new location in the repository. I checked with the new API too (addon-sdk), that works as well with the patch.

Comment 24

[Andreas Gal :gal]

http://hg.mozilla.org/tracemonkey/rev/9d8a15c1f22f

Comment 25

[Chris Leary [:cdleary] (not checking bugmail)]

cdleary-bot mozilla-central merge info:
http://hg.mozilla.org/mozilla-central/rev/9d8a15c1f22f