921171 - crash in js::NukeCrossCompartmentWrappers(JSContext*, js::CompartmentFilter const&, js::CompartmentFilter const&, js::NukeReferencesToWindow)

Status: RESOLVED FIXED

(Core :: JavaScript Engine, defect)

Description

[Tracy Walker [:tracy]]

This bug was filed from the Socorro interface and is 
report bp-2bf5e082-ddee-4c0d-adc0-4f9052130926.
=============================================================

This has climbed back topcrash on Fx26 and Fx27

Frame 	Module 	Signature 	Source
0 	mozjs.dll 	js::NukeCrossCompartmentWrappers(JSContext *,js::CompartmentFilter const &,js::CompartmentFilter const &,js::NukeReferencesToWindow) 	js/src/jswrapper.cpp
1 	xul.dll 	WindowDestroyedEvent::Run() 	dom/base/nsGlobalWindow.cpp
2 	xul.dll 	nsThread::ProcessNextEvent(bool,bool *) 	xpcom/threads/nsThread.cpp
3 	xul.dll 	NS_ProcessNextEvent(nsIThread *,bool) 	xpcom/glue/nsThreadUtils.cpp
4 	xul.dll 	mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate *) 	ipc/glue/MessagePump.cpp
5 	xul.dll 	MessageLoop::RunHandler() 	ipc/chromium/src/base/message_loop.cc
6 	xul.dll 	MessageLoop::Run() 	ipc/chromium/src/base/message_loop.cc
7 	xul.dll 	nsBaseAppShell::Run() 	widget/xpwidgets/nsBaseAppShell.cpp
8 	xul.dll 	nsAppShell::Run() 	widget/windows/nsAppShell.cpp
9 	nss3.dll 	nss3.dll@0x7960 	
10 	xul.dll 	XREMain::XRE_main(int,char * * const,nsXREAppData const *) 	toolkit/xre/nsAppRunner.cpp
11 	xul.dll 	XRE_main 	toolkit/xre/nsAppRunner.cpp
12 	firefox.exe 	do_main 	browser/app/nsBrowserApp.cpp
13 	firefox.exe 	NS_internal_main(int,char * *) 	browser/app/nsBrowserApp.cpp
14 	firefox.exe 	wmain 	toolkit/xre/nsWindowsWMain.cpp
15 	firefox.exe 	__tmainCRTStartup 	f:/dd/vctools/crt_bld/self_x86/crt/src/crtexe.c
16 	kernel32.dll 	kernel32.dll@0x1336a 	
17 	ntdll.dll 	ntdll.dll@0x39f72 	
18 	ntdll.dll 	ntdll.dll@0x39f45

Comment 2

[Robert Kaiser]

Starting with frame 3, the stacks vary.

Not sure if all those are high memory consumption as described in bug 789568 so here's a different new bug.

This spiked on 26 on 2013-09-16 and has been high on Aurora 26 and Nightly 27 since then, right now it ranks as the #3 top crash over Aurora builds from the last 3 days.

Comment 3

[Robert Kaiser]

Find more of those reports in https://crash-stats.mozilla.com/report/list?product=Firefox&signature=js%3A%3ANukeCrossCompartmentWrappers%28JSContext*%2C+js%3A%3ACompartmentFilter+const%26%2C+js%3A%3ACompartmentFilter+const%26%2C+js%3A%3ANukeReferencesToWindow%29

Comment 4

[Johnny Stenback (:jst)]

Bobby, this is #3 of our top orange factor bugs and needs investigating. Can you dig in? If this is starting to look like a bug in the JS engine itself, you know who to talk to :)

Comment 5

[Benjamin Smedberg]

See https://crash-analysis.mozilla.com/bsmedberg/NukeCrossCompartmentWrappers-nightly.csv There is a clear regression on the 20130914030203 nightly. The regression range link is:

http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=b9029b1de410&tochange=53d5e43e23cc

Comment 6

[Benjamin Smedberg]

Candidates that jump out at me:

f7bfa3ed149b	Stephen Pohl — Bug 817700 - Make <canvas>.toBlob run asynchronously. r=seth,roc,bz
5b89a959713b	Brian Hackett — Bug 915485 - Set compileAndGo when parsing scripts off thread for an Evaluate, r=luke.
3cb16a4bf227	Jason Orendorff — Bug 897403 - "Assertion failure: !((attrs ^ shape->attrs) & 0x40) || !(attrs & 0x40)" with bound function proxy. r=Waldo.
54247b7b87e6	Boris Zbarsky — Bug 915419. Add support for "object" types in owning unions (so union return values and unions in dictionaries and sequences. r=dzbarsky, smaug Adds RootedUnion and NullableRootedUnion structs that are used on the stack for union return values when the union needs rooting. It also adds a TraceUnion method on union return values, which is called by (Nullable)RootedUnion or by dictionary tracing. TraceUnion traces typed array and object members of unions (the only things unions can contain so far that might need tracing). Union return values are changed to store raw JSObject* or typed array structs instead of Rooted versions of both; the tracing is now handled via TraceUnion. The wrapping code for dictionary arguments to constructors is fixed to actually work. This required adding GetAs* methods to union return values that return references to the internal data. The SetToObject method is adjusted to actually work for union return value structs and not assume it's being generated for a union-conversion struct, and we now generate SetToObject methods as needed for union return values.
9a8916b16e6e	Dale Harvey — Bug 911195 - Properly compartment scroll event object. r=bz
1193eb976c83	Boris Zbarsky — Bug 915368. Give dictionaries copy constructors and assignment operators when it's safe. r=khuey

Comment 7

[Susanne Jaeger]

I can reproduce this crash at startup of Seamonkey on 2 systems as soon as adblockplus or adblockedge are installed and enabled. 
User agent: Mozilla/5.0 (X11; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0 SeaMonkey/2.23a2
Build identifier: 20131001013001
When I disable addon new start is ok, enabling adblockxxx after start is ok - no crash. 
newest in a series of crash-reports: https://crash-stats.mozilla.com/report/index/1572716c-821c-424e-a6af-8a00f2131001

Comment 8

[Bobby Holley (:bholley)]

(In reply to Susanne Jaeger from comment #7)
> I can reproduce this crash at startup of Seamonkey on 2 systems as soon as
> adblockplus or adblockedge are installed and enabled. 

Awesome! Can you bisect to determine the culprit push?

Comment 9

[Susanne Jaeger]

(In reply to Bobby Holley (:bholley) from comment #8)
> Awesome! Can you bisect to determine the culprit push?

I tested with precompiled builds from Aurora channel. 
Last good: User agent: Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0 SeaMonkey/2.22a2
Build identifier: 20130917013001

Any 2.23a2 build from buildid=20130918013001 to buildid=20131001013001 works for the first time it is started with  existing profile - checking addon compatibility for new version takes some time - Second start without compatibility check leads to crash, if adblockedge is active.

Comment 10

[Kyle Wooten]

Thanks!

Comment 11

[Bobby Holley (:bholley)]

(In reply to Susanne Jaeger from comment #7)
> I can reproduce this crash at startup of Seamonkey on 2 systems as soon as
> adblockplus or adblockedge are installed and enabled. 
> User agent: Mozilla/5.0 (X11; Linux x86_64; rv:26.0) Gecko/20100101
> Firefox/26.0 SeaMonkey/2.23a2
> Build identifier: 20131001013001
> When I disable addon new start is ok, enabling adblockxxx after start is ok
> - no crash. 

I installed this build of SeaMonkey on OSX, but didn't get a crash. Maybe it's an OS-dependent issue?

(In reply to Susanne Jaeger from comment #9)
> (In reply to Bobby Holley (:bholley) from comment #8)
> > Awesome! Can you bisect to determine the culprit push?
> 
> I tested with precompiled builds from Aurora channel. 
> Last good: User agent: Mozilla/5.0 (X11; Linux x86_64; rv:25.0)
> Gecko/20100101 Firefox/25.0 SeaMonkey/2.22a2
> Build identifier: 20130917013001

This is just the uplift merge, which means it contains 6 weeks worth of changes. If the only known way to reproduce this is on SeaMonkey, we need someone to bisect and build the changesets from the regression range in comment 5 to figure out the culprit changeset. If it's reproducible in Firefox somehow, we could just use the inbound builds, which would mean that whoever does the bisecting wouldn't have to recompile at each step.

To proceed here, we either need clear STR, or a finer-granularity regression range.

Comment 12

[Catalin Varga [QA][:VarCat]]

I've tried reproducing this issue without any success on the following environments:

Build identifier: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0
Build Id:20130910160258
OS: Win 7 x32

Build identifier: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0
Build Id:20130910160258
OS: Win Xp x32

User agent: Mozilla/5.0 (X11; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0 SeaMonkey/2.23a2
Build identifier: 20131001013001
OS: Ubuntu 12.04 x64

Based on the fact that 44.29 % of crashes happened just after Firefox started most of my testing was done by restarting Firefox in different scenarios(after an add-on installed or having several tabs opened with memory intensive pages and restoring the previous session).

Please note that based on comment 7 I installed adblock plus on both Firefox and SeaMonkey and still did not manage to reproduce the issue.

If there are any suggestions on what else should I try please tell me

Comment 13

[Bobby Holley (:bholley)]

Susanne, do you have any tips for Catalin on how you managed to reproduce the crash?

Comment 14

[Susanne Jaeger]

unfortunately it's not so easy. I still get the crash on every start of Seamonkey with my existing profile, which contains loads of email and right now 19 extensions. Dis-/Enabling adblock triggers the startup crash, but this alone won't do in a new testing profile, after installing and activating more extensions step by step I managed to get the crash, but I can't give a fixed point which other prerequisites are required, this seems to be a floating target. 
I also wasn't successful to get an error window, trying out precompiled nightlies with my working profile last working build was 
User agent: Mozilla/5.0 (X11; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0 SeaMonkey/2.23a1
Build identifier: 20130829003001
but after that I got crashes in other places and can't tell when exactly this one happened first.

Comment 15

[Bobby Holley (:bholley)]

If you duplicate the profile where you can reproduce this, and delete the emails and other sensitive information, does it still reproduce?

Comment 16

[Susanne Jaeger]

next attempt. I created a new profile, added all used extensions and search engines and disabled addon.compatibility check. After that the profile crashes at startup. Disabling adblock or wikilook stops the crashing but these two extensions alone aren't sufficient to trigger it. Crashing copy of this profile: http://sujag.de/mozilla/j39e06q7.crashtest.tar.gz I'll leave this online for a few days, will remove it at the end of the week. Some of the extensions contain platform specific parts, this is Linux x86_64.

Comment 17

[Bobby Holley (:bholley)]

Great! Catalin, can you give that profile a try (SeaMonkey linux64, if I understand correctly) and see if you can reproduce?

Comment 18

[Catalin Varga [QA][:VarCat]]

I've used the profile provided by Susanne and the following environment:

User agent: Mozilla/5.0 (X11; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0 SeaMonkey/2.23a2
Build identifier: 20131001013001
OS: Ubuntu 12.04 x64

but I still did not manage to reproduce the bug.

Comment 19

[Bobby Holley (:bholley)]

Darn. Just to confirm - Susanne, does the setup in setup in comment 18 match yours? And is it still the case (per comment 7) that you can reproduce the issue on two different systems with that profile?

Comment 20

[Susanne Jaeger]

I rechecked the Notebook. Yes both systems crash on startup 
Desktop: http://crash-stats.mozilla.com/report/index/bp-2d007c3d-154b-49c2-bd51-3bb5d2131017
and
Notebook: Crash ID: bp-c7d13a29-a90a-438b-93d2-575122131017 crash on Notebook on second start.
Both machines are running Debian Wheezy amd64 this shouldn't be too different from Ubuntu

Comment 21

[Bobby Holley (:bholley)]

(In reply to Susanne Jaeger from comment #20)
> I rechecked the Notebook. Yes both systems crash on startup 
> Desktop:
> http://crash-stats.mozilla.com/report/index/bp-2d007c3d-154b-49c2-bd51-
> 3bb5d2131017
> and
> Notebook: Crash ID: bp-c7d13a29-a90a-438b-93d2-575122131017 crash on
> Notebook on second start.
> Both machines are running Debian Wheezy amd64 this shouldn't be too
> different from Ubuntu

Kairo, can you pick out anything in Susanne's crash report that might explain why she can reproduce this issue with the attacehd profile but Catalin can't?

Comment 22

[Robert Kaiser]

(In reply to Bobby Holley (:bholley) from comment #21)
> Kairo, can you pick out anything in Susanne's crash report that might
> explain why she can reproduce this issue with the attacehd profile but
> Catalin can't?

No idea, but then I'm also more concerned with those crashes on Firefox Windows than on SeaMonkey Linux even if that happens with all products and OSes.
Also, we had this signature around for a lot of versions but it has jumped in 26 on 2013-09-16.

Comment 23

[Alice0775 White]

I can reproduce in 64bit build on Windows7 64bit and Ubuntu 64bit.

Regression window(m-c-win64)
Good:
http://hg.mozilla.org/mozilla-central/rev/f73bed2856a8
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:26.0) Gecko/20100101 Firefox/26.0 ID:20130910172959
Bad:
http://hg.mozilla.org/mozilla-central/rev/f9e8e8ce552c
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:26.0) Gecko/20100101 Firefox/26.0 ID:20130911003123
Pushlog:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=f73bed2856a8&tochange=f9e8e8ce552c



Regression window(m-i-linux64)
Good:
http://hg.mozilla.org/integration/mozilla-inbound/rev/d0a0127e099e
Mozilla/5.0 (X11; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0 ID:20130910161058
Bad:
http://hg.mozilla.org/integration/mozilla-inbound/rev/4cb81ac52275
Mozilla/5.0 (X11; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0 ID:20130910162258
Pushlog:
http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=d0a0127e099e&tochange=4cb81ac52275

Suspected: Bug 912914 , Bug 912719

Comment 24

[Bobby Holley (:bholley)]

Awesome! Thanks Alice!

Given that the other bug is a buildconfig patch, it seems pretty clear that this is a regression from bug 912719.

Brian - this is a high priority for stability and tree-management. Can you have a look? Alice can probably give you the STR used in comment 23.

Comment 25

[Hartmut Figge]

Using my archive of self compiled SM Trunk builds, Linux x86_64, and the crashtest profile of Susanne, i can confirm the crash. My regression window will not help because of a huge gap in my archive due to bustages, but there is another related information.

Up to 2013-09-26 16:36:00 PDT a Trunk SM with a fresh crashtest profile, the first start of SM shows the check for compatibility and then SM is up. Subsequent starts of SM ends with a crash.

Starting with 2013-09-27 16:04:00 PDT this behavior has changed. First start as before, the next two starts crash, but with the next start the window asking 'Would you like to restore your session?' comes up. If you choose 'Start New Session' SM will not crash again.

Comment 26

[Tracy Walker [:tracy]]

topcrash is being replaced by more precise keywords per https://bugzilla.mozilla.org/show_bug.cgi?id=927557#c3

Comment 27

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment 23 has the regression range (as does comment 24) and comment 25 looks to have STR so there looks to be enough info here to proceed - Brian is this on your priority list? We're merging FF26 to Beta on Monday Oct 28 and so will probably see a lot more crashes once this hits the larger population there.

Comment 28

[Brian Hackett [Laid off!]]

Sorry, this hasn't been on my radar at all, I wasn't needinfo'ed (please do that!) and didn't notice comment 24.  Alice, how did you reproduce this?

Comment 29

[Alice0775 White]

(In reply to Brian Hackett (:bhackett) from comment #28)
> Sorry, this hasn't been on my radar at all, I wasn't needinfo'ed (please do
> that!) and didn't notice comment 24.  Alice, how did you reproduce this?

STR
1. Extract archived profile of Comment#16 
2. Launch Firefox with -P
3. Create Profile and Choose folder which is created at step1, and Start
4. Click "Don't check" in Incompatible Add-ons dalog

Actual Results
Crash

Comment 30

[Robert Kaiser]

Putting ni? back to Brian, hoping those STR in comment #29 help him dig into it.

Comment 31

[Brian Hackett [Laid off!]]

Attachment: potential patch

I can't reproduce the crash, but in a debug build I do get assertion failures because we're trying to end up parsing a script off thread using a context with a pending exception --- wrappers for the exception get created in the compartment for the dummy global used for the off thread parse.  The wrapper code doesn't like this and it's also something that should never happen for these dummy compartments/zones, so it's conceivable this could lead to the later crash.  The attached patch watches for this case and doesn't permit off thread parsing on contexts with pending exceptions.

A larger issue though is why this pending exception is showing up at all.  Presuming this behavior was triggered by bug 912719, the cause is almost certainly the use of AutoSafeJSContext to report syntax errors etc. in XUL scripts (the pending exception triggering the assertions is a syntax error).

Comment 32

[Naveed Ihsanullah [:naveed]]

Unsetting myself since bhackett is on it.

Comment 33

[Bobby Holley (:bholley)]

Comment on attachment 821880 [details] [diff] [review]
potential patch

Review of attachment 821880 [details] [diff] [review]:
-----------------------------------------------------------------

As discussed on IRC, i think that this function should MOZ_ASSERT(!cx->isExceptionPending()). The caller shouldn't be invoking this function if there's a pending exception on the cx (it should have either propagated the failure or cleared the exception). So we need to figure out where this is coming from and fix that non-kosher usage of JSAPI.

Comment 34

[Brian Hackett [Laid off!]]

Attachment: potential patch

Also as discussed on IRC, the reason this AutoSafeJSContext ends up with a pending exception is that there is no AutoLastFrameCheck wrapping the generation of the syntax error.  This patch fixes the latter issue and will hopefully fix the crash.

Comment 35

[Bobby Holley (:bholley)]

Can we also MOZ_ASSERT(!cx->isExceptionPending()) in the other function?

Comment 36

[Brian Hackett [Laid off!]]

(In reply to Bobby Holley (:bholley) from comment #35)
> Can we also MOZ_ASSERT(!cx->isExceptionPending()) in the other function?

We will end up asserting this in JS_NewGlobalObject, which will always be called by JS::CompileOffThread.

Comment 37

[Brian Hackett [Laid off!]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/2c247c354e8a

Comment 38

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/2c247c354e8a

Comment 39

[Brian Hackett [Laid off!]]

Alice, can you still reproduce the crash on trunk?

Comment 40

[Alice0775 White]

I can confirm that the following Nightly build(since Oct-27) does not crash anymore.

http://hg.mozilla.org/mozilla-central/rev/a80dce1126db
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:27.0) Gecko/20100101 Firefox/27.0 ID:20131027030205

http://hg.mozilla.org/mozilla-central/rev/a80dce1126db
Mozilla/5.0 (X11; Linux x86_64; rv:27.0) Gecko/20100101 Firefox/27.0 ID:20131027030205

Comment 41

[Brian Hackett [Laid off!]]

Comment on attachment 822497 [details] [diff] [review]
potential patch

[Approval Request Comment]
Bug caused by (feature/regressing bug #): 912719
User impact if declined: crashes
Testing completed (on m-c, etc.): on m-c
Risk to taking this patch (and alternatives if risky): none

Comment 42

[Brian Hackett [Laid off!]]

(In reply to Alice0775 White from comment #40)
> I can confirm that the following Nightly build(since Oct-27) does not crash
> anymore.

Thanks!

Comment 43

[Ryan VanderMeulen [:RyanVM]]

This hit m-c before Aurora became Fx27. This needs approval for beta now. Also, any reason this bug is still open?

Comment 44

[Brian Hackett [Laid off!]]

Comment on attachment 822497 [details] [diff] [review]
potential patch

[Approval Request Comment]
See comment 43

Comment 45

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/ae7ca0cd3cda

Comment 46

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-b2g26_v1_2/rev/ae7ca0cd3cda

Comment 47

[Tracy Walker [:tracy]]

this is affecting on Fx 25.*

Comment 48

[Bogdan Maris, Desktop Test Engineering]

I reproduced the crash on Windows 8 x64 and Ubuntu 12.04 x64 with the profile from comment 16.

I am getting different signatures for each OS:
Windows
https://crash-stats.mozilla.com/report/index/3ecb6880-4682-4764-946a-8676b2131119
https://crash-stats.mozilla.com/report/index/128fb013-18f7-4b78-9acc-bd8902131119

Linux
https://crash-stats.mozilla.com/report/index/a0edae4f-82c9-4d2f-8898-1d0d92131119
https://crash-stats.mozilla.com/report/index/8f17713b-5388-4432-a7c6-bd3972131119

The regression from comment 23 is the same with this crash so I guess this is the same crash. Am I wrong?
This is verified as fixed on Firefox 26 beta 6 and latest Nightly.

Comment 49

[Brian Hackett [Laid off!]]

(In reply to Bogdan Maris, QA [:bogdan_maris] from comment #48)
> I reproduced the crash on Windows 8 x64 and Ubuntu 12.04 x64 with the
> profile from comment 16.
> 
> I am getting different signatures for each OS:
> Windows
> https://crash-stats.mozilla.com/report/index/3ecb6880-4682-4764-946a-
> 8676b2131119
> https://crash-stats.mozilla.com/report/index/128fb013-18f7-4b78-9acc-
> bd8902131119
> 
> Linux
> https://crash-stats.mozilla.com/report/index/a0edae4f-82c9-4d2f-8898-
> 1d0d92131119
> https://crash-stats.mozilla.com/report/index/8f17713b-5388-4432-a7c6-
> bd3972131119
> 
> The regression from comment 23 is the same with this crash so I guess this
> is the same crash. Am I wrong?
> This is verified as fixed on Firefox 26 beta 6 and latest Nightly.

All those crashes are from nightlies dating well before this fix went in.

Comment 50

[Bogdan Maris, Desktop Test Engineering]

Yes that is correct, I just tried to reproduce the crash on older builds in order to see if this is verified as fixed on latest builds.

Comment 51

[Bogdan Maris, Desktop Test Engineering]

Also verified as fixed on Windows XP x32, Windows 8 x32, Ubuntu 12.04 x32, Mac OS X 10.8.5 using latest Aurora 27.0a2 (buildID: 20131128004001).