962163 - crash in __delayLoadHelper2 | ffi_closure_STDCALL

Status: VERIFIED FIXED

(Firefox :: General, defect)

Description

[Robert Kaiser]

This bug was filed from the Socorro interface and is 
report bp-266dbe5e-2d91-4d69-9a82-48ccb2140120.
=============================================================

This signature shows up as #4 on the Aurora to crash list right now and it's pretty weird in terms of stacks, most of them look similar to this one:

0 	KERNELBASE.dll 	RaiseException 	
1 	mozjs.dll 	__delayLoadHelper2 	f:/dd/vctools/delayimp/delayhlp.cpp
2 	mozjs.dll 	ffi_closure_STDCALL 	
3 	xul.dll 	NS_InitXPCOM2 	xpcom/build/nsXPComInit.cpp
4 	xul.dll 	ScopedXPCOMStartup::Initialize() 	toolkit/xre/nsAppRunner.cpp
5 	xul.dll 	XREMain::XRE_main(int,char * * const,nsXREAppData const *) 	toolkit/xre/nsAppRunner.cpp
6 	xul.dll 	XRE_main 	toolkit/xre/nsAppRunner.cpp
7 	yXpJpNW.TQO 	yXpJpNW.TQO@0x190c 	
8 		@0x40a2a8 	
9 		@0x74535f45 	
10 	msvcr100.dll 	msvcr100.dll@0x8b582 	
11 		@0x59330000 	
12 	yXpJpNW.TQO 	yXpJpNW.TQO@0x21d3 	

The "yXpJpNW.TQO" file name here is a different random name in other reports, e.g. bp-568ff717-c0e7-4322-8c11-891042140120 has "qtnouFI.yCR", bp-ed497ca6-2099-43e0-8c70-d9e4a2140120 has "tnouFIy.CRI", bp-704cddeb-feb3-457c-a0e7-94c982140120 has "UmuvyXp.JPp" etc.
Those randomly-named files also show up on the module lists, without any debug ID etc.

I found one report in bp-09e91f4c-0130-45b7-9103-e1c3d2140120 that does have a better stack and no such weird module name, but then that one is not at startup.

The crash happens across all Windows versions from XP to 8.1 and on both 28.0a2 and 29.0a1.

Comment 1

[Benjamin Smedberg]

This is similar or identical to bug 958108, where apps installed via webapprt wouldn't start. It's because the Firefox runtime is being started from a location where we can't find the ICU libs, which are delayloaded and therefore crash.

I'm not actually sure whether or how webapprt crashes would be reported, so it's possible this is all webapprt stuff and this is a straight duplicate, or some other similar kind of client is experiencing the same basic problem. The random module ID below XRE_main is definitely weird though.

This shouldn't affect beta because IIRC we config ICU off for beta still.

Comment 2

[Benjamin Kerensa [:bkerensa]]

I think we will track this considering it is a top crash on Nightly.

Comment 3

[(Away)]

TL;DR I think this is a duplicate of bug 958108 but in a very interesting way. 

These strange modules are firefox.exe with a disguised path.
e.g. O:\tPEnDjN WTXpJ\PpSoxtQ\IigRpTO.qtn
is:  C:\Program Files\Nightly\firefox.exe
(Who does this or why, I don't know)

I took one of the dumps with an intact stack [1] and compared its offsets to those in firefox.exe from that BuildID.

IigRpTO.qtn@0x19c2
IigRpTO.qtn@0x2079
IigRpTO.qtn@0x2183
IigRpTO.qtn@0x2ad3

0:000> u firefox+0x19c2 -6 L1
004019bc ff153c604000    call    dword ptr [firefox!XRE_main (0040603c)]
0:000> u firefox+0x2079 -5 L1
00402074 e89cf4ffff      call    firefox!do_main (00401515)
0:000> u firefox+0x2183 -5 L1
0040217e e8d8fdffff      call    firefox!NS_internal_main (00401f5b)
0:000> u firefox+0x2ad3 -5 L1
00402ace e8c0f5ffff      call    firefox!wmain (00402093)

Prior to the fix for bug 958108 we were relying on Windows auto-finding the ICU modules in the same directory as the executable. I think this strange disguise trick must have broken that search.

The crash reports on nightly stop after the day that the ICU bug was fixed.

[1] https://crash-stats.mozilla.com/report/index/16cf917b-83be-4814-8e44-048dc2140120

Comment 4

[Benjamin Smedberg]

To clarify, this crash (unlike bug 958108) is not with webapps. Instead it appears to be caused by certain Bandoo DLLs (mgrldr.dll or safetyldr.dll) which corrupt the main Firefox executable path in memory. This causes the Firefox directory to no longer be in the default DLL search path, which causes the ICU dlls to fail to load.

I'm going to mark this FIXED based on the other bug landing, although we should verify via crash-stats. We're going to file a followup to track this bad behavior and either block those DLLs or get Bandoo to fix them.

Comment 5

[u279076]

Bug 958108 was "fixed" on 2014-01-21 in Nightly (down 8 positions to #29 @ 0.41%) but I'm still seeing a significant volume of crashes.

2014-01-26:  7
2014-01-25: 13
2014-01-24: 13
2014-01-23: 46
2014-01-22: 25 <-- first allegedly fixed build
2014-01-21: 50 
2014-01-20: 30
2014-01-19: 14
2014-01-18: 49

Comment 6

[u279076]

(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #5)

Sorry, ignore this -- I don't see any crashes past 2014-01-21 on Nightly. I must have inadvertently been looking at Aurora numbers in comment 5.

Here's the Nightly data:
https://crash-stats.mozilla.com/report/list?product=Firefox&range_value=7&range_unit=days&date=2014-01-28&signature=__delayLoadHelper2+|+ffi_closure_STDCALL&version=Firefox%3A29.0a1#tab-reports

Comment 7

[u279076]

Calling this verified fixed in Nightly based on crashstats data.

Comment 8

[Lukas Blakk [:lsblakk] use ?needinfo]

What's the deal with this one on 28?  Is this still an issue on that branch? I see bug 958108 is marked affected on 28 - is that patch safe to uplift?

Comment 9

[Benjamin Smedberg]

My understanding is that we disabled ICU for 28 beta and so neither of these bugs will affect 28 beta or release.

Comment 10

[u279076]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #9)
> My understanding is that we disabled ICU for 28 beta and so neither of these
> bugs will affect 28 beta or release.

This would indeed seem to be the case -- I see no reports of this in 28 Beta on the top-300 crash report.