startup crash spike in js::CreateRegExpMatchResult with Firefox 42.0b9

RESOLVED DUPLICATE of bug 1218473

Status

()

Core
JavaScript Engine
--
critical
RESOLVED DUPLICATE of bug 1218473
2 years ago
a year ago

People

(Reporter: Robert Kaiser, Unassigned)

Tracking

({crash})

Trunk
x86
Windows NT
crash
Points:
---

Firefox Tracking Flags

(firefox42+ affected)

Details

(crash signature)

(Reporter)

Description

2 years ago
[Tracking Requested - why for this release]:

This bug was filed from the Socorro interface and is 
report bp-2fbc02f7-de63-439f-a485-f12992151024.
=============================================================


Top stack frames:
0 		@0x10f43280 	
1 	xul.dll 	js::CreateRegExpMatchResult(JSContext*, JS::Handle<JSString*>, js::MatchPairs const&, JS::MutableHandle<JS::Value>) 	js/src/builtin/RegExp.cpp
2 	xul.dll 	regexp_exec_impl 	js/src/builtin/RegExp.cpp
3 	xul.dll 	regexp_exec_impl 	js/src/builtin/RegExp.cpp
4 	xul.dll 	js::regexp_exec(JSContext*, unsigned int, JS::Value*) 	js/src/builtin/RegExp.cpp
5 	xul.dll 	js::Invoke(JSContext*, JS::CallArgs, js::MaybeConstruct) 	js/src/vm/Interpreter.cpp
6 	xul.dll 	Interpret 	js/src/vm/Interpreter.cpp
7 	xul.dll 	js::RunScript(JSContext*, js::RunState&) 	js/src/vm/Interpreter.cpp
8 	xul.dll 	js::Invoke(JSContext*, JS::CallArgs, js::MaybeConstruct) 	js/src/vm/Interpreter.cpp
9 	xul.dll 	nsXPCWrappedJSClass::CallMethod(nsXPCWrappedJS*, unsigned short, XPTMethodDescriptor const*, nsXPTCMiniVariant*) 	js/xpconnect/src/XPCWrappedJSClass.cpp
10 	xul.dll 	nsXPCWrappedJS::CallMethod(unsigned short, XPTMethodDescriptor const*, nsXPTCMiniVariant*) 	js/xpconnect/src/XPCWrappedJS.cpp
11 	xul.dll 	PrepareAndDispatch 	xpcom/reflect/xptcall/md/win32/xptcstubs.cpp
[...]

The three crash signatures attached to the bug are highly spiking in the last beta of this cycle, 42.0b9, and are ~40% of the early crashes for this version. Those crashes are practically all early in startup and the stacks for js::CreateRegExpMatchResult and js::AddTypePropertyId are similar (the latter has 2 more frames or so on top), the js::regexp_exec are heavily truncated.

The signatures did exist previously at much lower volume but exploded as startup crashes in this version. The regression range is https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=FIREFOX_42_0b8_RELEASE&tochange=FIREFOX_42_0b9_RELEASE - I'm not sure what can cause this.

This is bad enough to block release and we are only a week before release with this, so we need urgent action here (there is a reason why I file this on a weekend).
(Reporter)

Comment 1

2 years ago
Actually, I just found that the `anonymous namespace''::CreateWindowExWHook signature - which is another 5% of 42.0b9 crashes and is startup as well - shows stacks like this (from bp-253e63ac-dc63-4d6c-bd9f-cd20b2151024):

0 	xul.dll 	`anonymous namespace'::CreateWindowExWHook(unsigned long, wchar_t const*, wchar_t const*, unsigned long, int, int, int, int, HWND__*, HMENU__*, HINSTANCE__*, void*) 	ipc/glue/WindowsMessageLoop.cpp
1 	xul.dll 	js::CreateRegExpMatchResult(JSContext*, JS::Handle<JSString*>, js::MatchPairs const&, JS::MutableHandle<JS::Value>) 	js/src/builtin/RegExp.cpp
2 	xul.dll 	regexp_exec_impl 	js/src/builtin/RegExp.cpp
3 	xul.dll 	js::regexp_exec(JSContext*, unsigned int, JS::Value*) 	js/src/builtin/RegExp.cpp
4 	xul.dll 	js::Invoke(JSContext*, JS::CallArgs, js::MaybeConstruct) 	js/src/vm/Interpreter.cpp

So I guess this might the same thing.

I also found that that signature also exists in 43.0a2 builds from 10/22 and later, and there it is a breakpoint instead of an exception and has a stack like this (bp-8a8fbab7-3fd0-4737-aa9d-ae2b32151024):

0 	xul.dll 	`anonymous namespace'::CreateWindowExWHook(unsigned long, wchar_t const*, wchar_t const*, unsigned long, int, int, int, int, HWND__*, HMENU__*, HINSTANCE__*, void*) 	ipc/glue/WindowsMessageLoop.cpp
1 	xul.dll 	base::MessagePumpForUI::InitMessageWnd() 	ipc/chromium/src/base/message_pump_win.cc
2 	xul.dll 	base::MessagePumpForUI::MessagePumpForUI() 	ipc/chromium/src/base/message_pump_win.cc
3 	xul.dll 	MessageLoop::MessageLoop(MessageLoop::Type) 	ipc/chromium/src/base/message_loop.cc
4 	xul.dll 	base::Thread::ThreadMain() 	ipc/chromium/src/base/thread.cc

That sounds a bit more sane to me, and I'd immediately wonder if bug 1213567 might be connected.

Aaron, bob, could that be?
Crash Signature: [@ js::CreateRegExpMatchResult ] [@ js::AddTypePropertyId ] [@ js::regexp_exec ] → [@ js::CreateRegExpMatchResult ] [@ js::AddTypePropertyId ] [@ js::regexp_exec ] [@ `anonymous namespace''::CreateWindowExWHook ]
Flags: needinfo?(bobowen.code)
Flags: needinfo?(aklotz)

Comment 2

2 years ago
this seems to happen nearly exclusively on devices with an intel/nvidia dual gpu setup, as can bee seen in the "app notes" facet for https://crash-stats.mozilla.com/search/?product=Firefox&version=42.0b9&signature=%3Djs%3A%3AAddTypePropertyId&signature=%3Djs%3A%3ACreateRegExpMatchResult&signature=%3D%60anonymous+namespace%27%27%3A%3ACreateWindowExWHook&signature=%3Djs%3A%3Aregexp_exec&_facets=signature&_facets=user_comments&_facets=platform&_facets=version&_facets=adapter_vendor_id&_facets=app_notes&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-app_notes 

the 42.0b9 crashes correlation for modules also contains some hints pointing in that direction:
  js::AddTypePropertyId|EXCEPTION_ACCESS_VIOLATION_READ (2315 crashes)
    100% (2311/2315) vs.  46% (6572/14314) _etoured.dll
    100% (2311/2315) vs.  46% (6632/14314) nvd3d9wrap.dll
    100% (2311/2315) vs.  46% (6632/14314) nvdxgiwrap.dll
    100% (2311/2315) vs.  46% (6645/14314) nvumdshim.dll
    100% (2311/2315) vs.  47% (6687/14314) nvinit.dll

  js::CreateRegExpMatchResult|EXCEPTION_ACCESS_VIOLATION_EXEC (1809 crashes)
    100% (1807/1809) vs.  46% (6572/14314) _etoured.dll
    100% (1807/1809) vs.  46% (6632/14314) nvd3d9wrap.dll
    100% (1807/1809) vs.  46% (6632/14314) nvdxgiwrap.dll
    100% (1807/1809) vs.  46% (6645/14314) nvumdshim.dll
    100% (1807/1809) vs.  47% (6687/14314) nvinit.dll

Comment 3

2 years ago
here's the aurora pushlog from 10/21 to 10/22 as well, which is a bit more compact than from 42.0b8 to 42.0b9: https://hg.mozilla.org/releases/mozilla-aurora/pushloghtml?fromchange=42c40ea06488&tochange=3f6b26ed69fd

Updated

2 years ago
Crash Signature: [@ js::CreateRegExpMatchResult ] [@ js::AddTypePropertyId ] [@ js::regexp_exec ] [@ `anonymous namespace''::CreateWindowExWHook ] → [@ js::CreateRegExpMatchResult ] [@ js::AddTypePropertyId ] [@ js::regexp_exec ] [@ `anonymous namespace''::CreateWindowExWHook ] [@ CreateWindowInternal ]

Comment 4

2 years ago
noise
Jason, any chance this could be caused by bug 1218128?
Flags: needinfo?(jorendorff)

Comment 5

2 years ago
noise
Jason, any chance this could be caused by bug 1206700?
(with the correct bug number)
(Reporter)

Comment 6

2 years ago
noise
(In reply to Sylvestre Ledru [:sylvestre] from comment #5)
> Jason, any chance this could be caused by bug 1206700?
> (with the correct bug number)

I think that6 actually did not land for this beta yet.

Comment 7

2 years ago
noise
Indeed, my bad. Sorry about the noise.
Flags: needinfo?(jorendorff)

Comment 8

2 years ago
I don't see how CreateRegExpMatchResult could trigger the create window hooks.

InitMessageWnd calls CreateWindow, so it's odd that the next one is the CreateWindowEx hook. Maybe one calls the other. 

It's also not clear to me why the graphics card setup would affect the hooks, but to be fair Aaron is likely to have a better idea anyway.
Flags: needinfo?(bobowen.code)

Comment 9

2 years ago
(In reply to Bob Owen (:bobowen) from comment #8)

> InitMessageWnd calls CreateWindow, so it's odd that the next one is the
> CreateWindowEx hook. Maybe one calls the other. 

It seems that CreateWindow is compiled directly into a CreateWindowEx call, so that explains that.

Nearly all the crashes seem to be on Windows 8 or later as well.
tracking-firefox42: ? → +
(Reporter)

Comment 10

2 years ago
philipp found two more signatures that are definitely the same thing and point even stronger to the window neutering patch.
Crash Signature: [@ js::CreateRegExpMatchResult ] [@ js::AddTypePropertyId ] [@ js::regexp_exec ] [@ `anonymous namespace''::CreateWindowExWHook ] [@ CreateWindowInternal ] → [@ js::CreateRegExpMatchResult ] [@ js::AddTypePropertyId ] [@ js::regexp_exec ] [@ `anonymous namespace''::CreateWindowExWHook ] [@ CreateWindowInternal ] [@ mozilla::widget::TaskbarWindowPreview::Release ] [@ mozilla::ipc::SuppressedNeuteri&hellip;
Hannes, bbouvier told me that you might know when it is failing in  js::CreateRegExpMatchResult .
Thanks
Flags: needinfo?(hv1989)
I think there are a couple of issues at play here, and they're tangling with each other in a way that is confusing us:

This stack:
https://crash-stats.mozilla.com/report/index/253e63ac-dc63-4d6c-bd9f-cd20b2151024

Makes no sense. JS RegExp stuff has nothing to do with Win32 GUI goop. Either breakpad could not symbolicate properly or the JS code is incorrectly jumping into it somehow; either way the CreateRegExpMatchResult crash is unrelated to the window neutering stuff and should be addressed by a JS person.

I think that the stacks in https://bugzilla.mozilla.org/show_bug.cgi?id=1218128#c1 and Bob's remark in https://bugzilla.mozilla.org/show_bug.cgi?id=1218128#c9 are more indicative of a problem. I'd be OK for a backout for beta in that case. But we really should separate these two issues to two separate bugs :-)
Flags: needinfo?(aklotz)
Blocks: 1218473

Comment 13

2 years ago
One other thing I forgot to mention in comment 9, is that I couldn't find this crash at all on Nightly, which also seems odd.
Regarding https://crash-stats.mozilla.com/report/index/253e63ac-dc63-4d6c-bd9f-cd20b2151024:

The function call was correct in the sense that the disassembly shows that the correct function was called and the correct parameter was pushed on the stack. Everything there indicates to me that something caused the CPU to jump to CreateWindowExWHook.

Oddly enough, when I disassemble CreateWindowExWHook, I don't see the instruction modifications that we would expect when hooking CreateWindowExW. Independently of each other, I would not consider these two issues related, but in combination I suspect that the API interception code wrote a jmp to the wrong location. It is likely that this is related to the other code after all.
Sorry, "Oddly enough, when I disassemble CreateWindowExWHook" should read, "Oddly enough, when I disassemble user32!CreateWindowExW"
This report definitely seems strange.

In the changelog there is nothing that could have affected "regexp". Also the backtrace shows we are in the Interpreter in regular c++ code. I don't understand how we could jump from "CreateRegExpMatchResult" to "CreateWindowEx". 

The only reason I can find to create a popup, is for the "slow running dialog". Given e10s is enabled on nightly it might skew the results when that happens. I don't know if we still run the "slow running dialog" on e10s. Which would explain why we don't see that crash signature on nightly.
=> But on issue with this explanation is that I cannot find any of the callees of "CreateRegExpMatchResult" checking "JS_CheckForInterrupt" ... In gc (gcIfNeededPerAllocation) we check for 'pendingInterrupt', but we don't execute the interrupt callback. Looks like this reason is therefore debunked?


If I ignore the "CreateRegExpMatchResult" and only look at the cases which have "CreateWindowExWHook". There is strong evidence bug 1213567 is related. Given that the patch changes some of the interactions around that callsite. Now the question is what is worse. The original intent of this patch was to fix a crash reports...
Flags: needinfo?(hv1989)
I found the root cause of this and the reason why we were seeing this signature in https://bugzilla.mozilla.org/show_bug.cgi?id=1218473#c44
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 1218473
No longer blocks: 1218473
You need to log in before you can comment on or make changes to this bug.