1015341 - startup crash in KiUserCallbackDispatcher on 32-bit versions of Windows 7 and Vista

Status: RESOLVED WORKSFORME

(Core :: Widget, defect)

Description

[u279076]

This bug was filed from the Socorro interface and is 
report bp-55ab279a-2ebf-4c05-bf61-e73d22140516.
=============================================================
0 		@0x77bb4ec3 	
1 	ntdll.dll 	KiUserCallbackDispatcher 	
2 	ntdll.dll 	KiUserApcDispatcher 	
3 	user32.dll 	_CreateWindowEx 	
4 	user32.dll 	CreateWindowExW 	
5 	ole32.dll 	InitMainThreadWnd() 	
6 	ole32.dll 	CoVrfNotifyCoInit() 	
7 	shell32.dll 	kfapi::CCoInitializer::Init() 	
8 	shell32.dll 	kfapi::CFolderIDListBuilder::GetIDList(_GUID const &,kfapi::KNOWNFOLDER_DEFINITION_EX const &,void *,unsigned long,kfapi::CFolderCache *,_ITEMIDLIST * *) 	
9 	shell32.dll 	CTSmartObj<_ITEMIDLIST *,CTSmartPtr_PolicyComplete<CTContainer_PolicyCoTaskMem> >::Attach(_ITEMIDLIST * const &) 	
10 	shell32.dll 	kfapi::CKFFacade::GetFolderIDList(_GUID const &,unsigned long,void *,_ITEMIDLIST * *) 	
11 	shell32.dll 	SHGetKnownFolderIDList_Internal 	
12 	shell32.dll 	SHGetFolderLocation 	
13 	shell32.dll 	SHGetSpecialFolderLocation 	
14 	xul.dll 	GetShellFolderPath 	toolkit/xre/nsXREDirProvider.cpp
15 	xul.dll 	nsXREDirProvider::GetUserDataDirectoryHome(nsIFile * *,bool) 	toolkit/xre/nsXREDirProvider.cpp
16 	xul.dll 	nsXREDirProvider::GetUserDataDirectory(nsIFile * *,bool,nsACString_internal const *,nsACString_internal const *,nsACString_internal const *) 	toolkit/xre/nsXREDirProvider.cpp
17 	xul.dll 	nsXREDirProvider::GetUserAppDataDirectory(nsIFile * *) 	toolkit/xre/nsXREDirProvider.h
18 	xul.dll 	XREMain::XRE_mainInit(bool *) 	toolkit/xre/nsAppRunner.cpp
19 	xul.dll 	XREMain::XRE_main(int,char * * const,nsXREAppData const *) 	toolkit/xre/nsAppRunner.cpp
20 	xul.dll 	XRE_main 	toolkit/xre/nsAppRunner.cpp
21 	firefox.exe 	do_main 	browser/app/nsBrowserApp.cpp
22 	firefox.exe 	NS_internal_main(int,char * *) 	browser/app/nsBrowserApp.cpp
23 	firefox.exe 	wmain 	toolkit/xre/nsWindowsWMain.cpp
24 	firefox.exe 	__tmainCRTStartup 	f:/dd/vctools/crt_bld/self_x86/crt/src/crtexe.c:552
25 	kernel32.dll 	BaseThreadInitThunk 	
26 	ntdll.dll 	__RtlUserThreadStart 	
27 	ntdll.dll 	_RtlUserThreadStart 	
=============================================================
More reports: https://crash-stats.mozilla.com/report/list?product=Firefox&signature=KiUserCallbackDispatcher

This signature has been around for a really long time but something recently has caused this to spike. I first noticed it today when looking at the Firefox 31 explosiveness report. It went from between 10 to 120 reports a day for the previous 10 days to over 500 reports on 2014-05-22.

Source:
https://crash-analysis.mozilla.com/rkaiser/2014-05-22/2014-05-22.firefox.31.explosiveness.html

Current Volume:
Firefox 29 - #26 @ 0.48% with 3054 crashes
Firefox 30 - #28 @ 0.45% with 958 crashes
Firefox 31 - #8 @ 1.07% with 95 crashes (up 24 positions in a week)
Firefox 32 - no rank, only 3 crashes

I'm filing this report to investigate what might have caused this spike recently. Given the long history of this crash I'm wondering if there is some external event which might be triggering this, like a Windows update perhaps?

Comment 2

[u279076]

Here are the module correlations:
97% (525/539) vs.  58% (69563/120499) DWrite.dll
100% (539/539) vs.  74% (88603/120499) lpk.dll
100% (538/539) vs.  77% (92481/120499) winnsi.dll
100% (538/539) vs.  77% (92491/120499) IPHLPAPI.DLL
100% (538/539) vs.  78% (93521/120499) powrprof.dll
91% (493/539) vs.  70% (83828/120499) samcli.dll
91% (493/539) vs.  70% (83867/120499) wkscli.dll
91% (493/539) vs.  70% (83868/120499) netutils.dll
91% (493/539) vs.  70% (83869/120499) srvcli.dll
100% (538/539) vs.  78% (94006/120499) dwmapi.dll
100% (538/539) vs.  78% (94024/120499) nsi.dll
100% (539/539) vs.  78% (94260/120499) firefox.exe
100% (539/539) vs.  79% (94874/120499) dbghelp.dll
91% (493/539) vs.  71% (86126/120499) devobj.dll
91% (493/539) vs.  72% (86265/120499) CRYPTBASE.dll
91% (493/539) vs.  72% (86279/120499) sechost.dll
91% (493/539) vs.  72% (86279/120499) KERNELBASE.dll
91% (493/539) vs.  72% (87156/120499) cfgmgr32.dll
100% (539/539) vs.  84% (101390/120499) msctf.dll
6% (34/539) vs.   1% (1485/120499) sysfer.dll

Bsmedberg mentioned on IRC that this could be related to malware. Unfortunately, there is nothing really actionable in the comments. The majority are just complaints that they can't start Firefox.

Comment 3

[Lukas Blakk [:lsblakk] use ?needinfo]

Right now this looks mainly like a FF31 issue so tracking for that version but we've got nothing here that makes this a driver for a 29.0.2 this late in the cycle, and without a significant volume on FF30 there's no need to track/block there either. If there is any way to reach out to users who are experiencing this crash and get them to try safe mode or get lists of addons to look for correlations that might get us something to start with.

Comment 4

[Liz Henry (:lizzard) (privacy/fingerprinting team)]

I don't think this is a topcrash any more, but this crash signature is still steadily showing up in the top 50. Currently at 342/66523 crashes for  31.0b in the last 7 days. 

It hasn't show up at all for Firefox 32 or 33.

Comment 5

[Sylvestre Ledru [:Sylvestre]]

Not tracking anymore. Please resubmit to tracking if it spikes again.

Comment 6

[u279076]

There are 894 crashes in the last week on Beta which makes it #33 currently and has a Crashes per Install value of nearly 1:1. Given this is a startup crash it might be worth tracking as it may prevent people from running/updating Firefox, though I'm not sure what we can do about this.

Comment 7

[(Away)]

This is only happening on 32-bit versions of Windows 7 and Vista.

The stacks look like this:
<garbage code address>
user32!__fnINLPCREATESTRUCT+0x8b
ntdll!KiUserCallbackDispatcher+0x2e

The garbage address matches the call target (call dword ptr [esi+5Ch]), so we jumped directly there.

I was stepping through __fnINLPCREATESTRUCT on my own machine, and I noticed that the code used different field offsets even though I had exactly the same Windows branch and build timestamp as the dump [1]. The dump was native 32-bit, and my machine was 32-bit WOW64. So it seems that internal user32 data structures can be different between native 32-bit and emulated 32-bit.

I bet there's some external code poking around whose authors didn't bother testing native 32-bit.

[1] https://crash-stats.mozilla.com/report/index/b34f7c07-2a84-4943-8396-a9b682140626

Comment 8

[u279076]

So far this signature is not showing up for 31.0b99 or 31.0, bug 1035534 and bug 1035537 are still there in significant volume though.

Comment 9

[Wayne Mery (:wsmwk)]

still #2 crash

Comment 10

[Wayne Mery (:wsmwk)]

(In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment #9)
> still #2 crash

NOT. posted in wrong bug

Comment 11

[u279076]

Latest Firefox release has 12 crashes reported in the last week.
Latest Thunderbird has 2506 crashes reported in the last week.

Based on these stats I'm marking this as a topcrash for Thunderbird. This is basically a non-issue for Firefox but we should prioritize investigation for Thunderbird. Unfortunately none of the recent reports have yielded any more leads.

Comment 12

[Wayne Mery (:wsmwk)]

Yeah, I've been following both products and there is certainly a divergence.
Thus, Thunderbird related crashes are being tracked in bug 1111861.

Comment 13

[u279076]

(In reply to Wayne Mery (:wsmwk, use Needinfo for questions) from comment #12)
> Yeah, I've been following both products and there is certainly a divergence.
> Thus, Thunderbird related crashes are being tracked in bug 1111861.

Well I don't see a point in tracking this for Firefox based on volume. If there's nothing to be done in this bug report I suggest we just close it and focus investigation for Thunderbird in bug 1111861 in that case.