[Tracking Requested - why for this release]: This bug was filed from the Socorro interface and is report bp-2fbc02f7-de63-439f-a485-f12992151024. ============================================================= Top stack frames: 0 @0x10f43280 1 xul.dll js::CreateRegExpMatchResult(JSContext*, JS::Handle<JSString*>, js::MatchPairs const&, JS::MutableHandle<JS::Value>) js/src/builtin/RegExp.cpp 2 xul.dll regexp_exec_impl js/src/builtin/RegExp.cpp 3 xul.dll regexp_exec_impl js/src/builtin/RegExp.cpp 4 xul.dll js::regexp_exec(JSContext*, unsigned int, JS::Value*) js/src/builtin/RegExp.cpp 5 xul.dll js::Invoke(JSContext*, JS::CallArgs, js::MaybeConstruct) js/src/vm/Interpreter.cpp 6 xul.dll Interpret js/src/vm/Interpreter.cpp 7 xul.dll js::RunScript(JSContext*, js::RunState&) js/src/vm/Interpreter.cpp 8 xul.dll js::Invoke(JSContext*, JS::CallArgs, js::MaybeConstruct) js/src/vm/Interpreter.cpp 9 xul.dll nsXPCWrappedJSClass::CallMethod(nsXPCWrappedJS*, unsigned short, XPTMethodDescriptor const*, nsXPTCMiniVariant*) js/xpconnect/src/XPCWrappedJSClass.cpp 10 xul.dll nsXPCWrappedJS::CallMethod(unsigned short, XPTMethodDescriptor const*, nsXPTCMiniVariant*) js/xpconnect/src/XPCWrappedJS.cpp 11 xul.dll PrepareAndDispatch xpcom/reflect/xptcall/md/win32/xptcstubs.cpp [...] The three crash signatures attached to the bug are highly spiking in the last beta of this cycle, 42.0b9, and are ~40% of the early crashes for this version. Those crashes are practically all early in startup and the stacks for js::CreateRegExpMatchResult and js::AddTypePropertyId are similar (the latter has 2 more frames or so on top), the js::regexp_exec are heavily truncated. The signatures did exist previously at much lower volume but exploded as startup crashes in this version. The regression range is https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=FIREFOX_42_0b8_RELEASE&tochange=FIREFOX_42_0b9_RELEASE - I'm not sure what can cause this. This is bad enough to block release and we are only a week before release with this, so we need urgent action here (there is a reason why I file this on a weekend).
Actually, I just found that the `anonymous namespace''::CreateWindowExWHook signature - which is another 5% of 42.0b9 crashes and is startup as well - shows stacks like this (from bp-253e63ac-dc63-4d6c-bd9f-cd20b2151024): 0 xul.dll `anonymous namespace'::CreateWindowExWHook(unsigned long, wchar_t const*, wchar_t const*, unsigned long, int, int, int, int, HWND__*, HMENU__*, HINSTANCE__*, void*) ipc/glue/WindowsMessageLoop.cpp 1 xul.dll js::CreateRegExpMatchResult(JSContext*, JS::Handle<JSString*>, js::MatchPairs const&, JS::MutableHandle<JS::Value>) js/src/builtin/RegExp.cpp 2 xul.dll regexp_exec_impl js/src/builtin/RegExp.cpp 3 xul.dll js::regexp_exec(JSContext*, unsigned int, JS::Value*) js/src/builtin/RegExp.cpp 4 xul.dll js::Invoke(JSContext*, JS::CallArgs, js::MaybeConstruct) js/src/vm/Interpreter.cpp So I guess this might the same thing. I also found that that signature also exists in 43.0a2 builds from 10/22 and later, and there it is a breakpoint instead of an exception and has a stack like this (bp-8a8fbab7-3fd0-4737-aa9d-ae2b32151024): 0 xul.dll `anonymous namespace'::CreateWindowExWHook(unsigned long, wchar_t const*, wchar_t const*, unsigned long, int, int, int, int, HWND__*, HMENU__*, HINSTANCE__*, void*) ipc/glue/WindowsMessageLoop.cpp 1 xul.dll base::MessagePumpForUI::InitMessageWnd() ipc/chromium/src/base/message_pump_win.cc 2 xul.dll base::MessagePumpForUI::MessagePumpForUI() ipc/chromium/src/base/message_pump_win.cc 3 xul.dll MessageLoop::MessageLoop(MessageLoop::Type) ipc/chromium/src/base/message_loop.cc 4 xul.dll base::Thread::ThreadMain() ipc/chromium/src/base/thread.cc That sounds a bit more sane to me, and I'd immediately wonder if bug 1213567 might be connected. Aaron, bob, could that be?
this seems to happen nearly exclusively on devices with an intel/nvidia dual gpu setup, as can bee seen in the "app notes" facet for https://crash-stats.mozilla.com/search/?product=Firefox&version=42.0b9&signature=%3Djs%3A%3AAddTypePropertyId&signature=%3Djs%3A%3ACreateRegExpMatchResult&signature=%3D%60anonymous+namespace%27%27%3A%3ACreateWindowExWHook&signature=%3Djs%3A%3Aregexp_exec&_facets=signature&_facets=user_comments&_facets=platform&_facets=version&_facets=adapter_vendor_id&_facets=app_notes&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-app_notes the 42.0b9 crashes correlation for modules also contains some hints pointing in that direction: js::AddTypePropertyId|EXCEPTION_ACCESS_VIOLATION_READ (2315 crashes) 100% (2311/2315) vs. 46% (6572/14314) _etoured.dll 100% (2311/2315) vs. 46% (6632/14314) nvd3d9wrap.dll 100% (2311/2315) vs. 46% (6632/14314) nvdxgiwrap.dll 100% (2311/2315) vs. 46% (6645/14314) nvumdshim.dll 100% (2311/2315) vs. 47% (6687/14314) nvinit.dll js::CreateRegExpMatchResult|EXCEPTION_ACCESS_VIOLATION_EXEC (1809 crashes) 100% (1807/1809) vs. 46% (6572/14314) _etoured.dll 100% (1807/1809) vs. 46% (6632/14314) nvd3d9wrap.dll 100% (1807/1809) vs. 46% (6632/14314) nvdxgiwrap.dll 100% (1807/1809) vs. 46% (6645/14314) nvumdshim.dll 100% (1807/1809) vs. 47% (6687/14314) nvinit.dll
here's the aurora pushlog from 10/21 to 10/22 as well, which is a bit more compact than from 42.0b8 to 42.0b9: https://hg.mozilla.org/releases/mozilla-aurora/pushloghtml?fromchange=42c40ea06488&tochange=3f6b26ed69fd
Jason, any chance this could be caused by bug 1218128?
Jason, any chance this could be caused by bug 1206700? (with the correct bug number)
(In reply to Sylvestre Ledru [:sylvestre] from comment #5) > Jason, any chance this could be caused by bug 1206700? > (with the correct bug number) I think that6 actually did not land for this beta yet.
Indeed, my bad. Sorry about the noise.
I don't see how CreateRegExpMatchResult could trigger the create window hooks. InitMessageWnd calls CreateWindow, so it's odd that the next one is the CreateWindowEx hook. Maybe one calls the other. It's also not clear to me why the graphics card setup would affect the hooks, but to be fair Aaron is likely to have a better idea anyway.
(In reply to Bob Owen (:bobowen) from comment #8) > InitMessageWnd calls CreateWindow, so it's odd that the next one is the > CreateWindowEx hook. Maybe one calls the other. It seems that CreateWindow is compiled directly into a CreateWindowEx call, so that explains that. Nearly all the crashes seem to be on Windows 8 or later as well.
philipp found two more signatures that are definitely the same thing and point even stronger to the window neutering patch.
Hannes, bbouvier told me that you might know when it is failing in js::CreateRegExpMatchResult . Thanks
I think there are a couple of issues at play here, and they're tangling with each other in a way that is confusing us: This stack: https://crash-stats.mozilla.com/report/index/253e63ac-dc63-4d6c-bd9f-cd20b2151024 Makes no sense. JS RegExp stuff has nothing to do with Win32 GUI goop. Either breakpad could not symbolicate properly or the JS code is incorrectly jumping into it somehow; either way the CreateRegExpMatchResult crash is unrelated to the window neutering stuff and should be addressed by a JS person. I think that the stacks in https://bugzilla.mozilla.org/show_bug.cgi?id=1218128#c1 and Bob's remark in https://bugzilla.mozilla.org/show_bug.cgi?id=1218128#c9 are more indicative of a problem. I'd be OK for a backout for beta in that case. But we really should separate these two issues to two separate bugs :-)
One other thing I forgot to mention in comment 9, is that I couldn't find this crash at all on Nightly, which also seems odd.
Regarding https://crash-stats.mozilla.com/report/index/253e63ac-dc63-4d6c-bd9f-cd20b2151024: The function call was correct in the sense that the disassembly shows that the correct function was called and the correct parameter was pushed on the stack. Everything there indicates to me that something caused the CPU to jump to CreateWindowExWHook. Oddly enough, when I disassemble CreateWindowExWHook, I don't see the instruction modifications that we would expect when hooking CreateWindowExW. Independently of each other, I would not consider these two issues related, but in combination I suspect that the API interception code wrote a jmp to the wrong location. It is likely that this is related to the other code after all.
Sorry, "Oddly enough, when I disassemble CreateWindowExWHook" should read, "Oddly enough, when I disassemble user32!CreateWindowExW"
This report definitely seems strange. In the changelog there is nothing that could have affected "regexp". Also the backtrace shows we are in the Interpreter in regular c++ code. I don't understand how we could jump from "CreateRegExpMatchResult" to "CreateWindowEx". The only reason I can find to create a popup, is for the "slow running dialog". Given e10s is enabled on nightly it might skew the results when that happens. I don't know if we still run the "slow running dialog" on e10s. Which would explain why we don't see that crash signature on nightly. => But on issue with this explanation is that I cannot find any of the callees of "CreateRegExpMatchResult" checking "JS_CheckForInterrupt" ... In gc (gcIfNeededPerAllocation) we check for 'pendingInterrupt', but we don't execute the interrupt callback. Looks like this reason is therefore debunked? If I ignore the "CreateRegExpMatchResult" and only look at the cases which have "CreateWindowExWHook". There is strong evidence bug 1213567 is related. Given that the patch changes some of the interactions around that callsite. Now the question is what is worse. The original intent of this patch was to fix a crash reports...
I found the root cause of this and the reason why we were seeing this signature in https://bugzilla.mozilla.org/show_bug.cgi?id=1218473#c44