823077 - crash in gfxWindowsPlatform::UpdateRenderMode @ nvwgf2um.dll@0xf5ddf (Updating Nvidia driver 310.70)

Status: RESOLVED INCOMPLETE

(Core :: Graphics, defect)

Description

[Marcia Knous [:marcia]]

This bug was filed from the Socorro interface and is 
report bp-1bf3e70d-46b5-4272-b335-0e2eb2121218 .
============================================================= 

Seen while looking at crash stats and the most recent explosive report. Comments indicating at the time of the crash people were updating there Nvidia graphics driver 310.70 and that they are using hardware acceleration.

Link to crashes: https://crash-stats.mozilla.com/report/list?signature=nvwgf2um.dll@0xf5ddf

Frame 	Module 	Signature 	Source
0 	nvwgf2um.dll 	nvwgf2um.dll@0xf5ddf 	
1 	nvwgf2um.dll 	nvwgf2um.dll@0xf4194 	
2 	nvwgf2um.dll 	nvwgf2um.dll@0x1422aa 	
3 	nvwgf2um.dll 	nvwgf2um.dll@0x1417dc 	
4 	nvwgf2um.dll 	nvwgf2um.dll@0x16dbff 	
5 	nvwgf2um.dll 	nvwgf2um.dll@0x16e0d2 	
6 	nvwgf2um.dll 	nvwgf2um.dll@0x141964 	
7 	nvwgf2um.dll 	nvwgf2um.dll@0x16d69a 	
8 	nvwgf2um.dll 	nvwgf2um.dll@0x141a7d 	
9 	nvwgf2um.dll 	nvwgf2um.dll@0x16d69a 	
10 	nvwgf2um.dll 	nvwgf2um.dll@0x16cd8b 	
11 	nvwgf2um.dll 	nvwgf2um.dll@0x16d120 	
12 	nvwgf2um.dll 	nvwgf2um.dll@0x16d680 	
13 	nvwgf2um.dll 	nvwgf2um.dll@0x16cd8b 	
14 	nvwgf2um.dll 	nvwgf2um.dll@0x16d120 	
15 	nvwgf2um.dll 	nvwgf2um.dll@0x16d680 	
16 	nvwgf2um.dll 	nvwgf2um.dll@0x16cd8b 	
17 	nvwgf2um.dll 	nvwgf2um.dll@0x16d120 	
18 	nvwgf2um.dll 	nvwgf2um.dll@0x16d680 	
19 	nvwgf2um.dll 	nvwgf2um.dll@0xf3f00 	
20 	nvwgf2um.dll 	nvwgf2um.dll@0x14007d 	
21 	nvwgf2um.dll 	nvwgf2um.dll@0x140304 	
22 	nvwgf2um.dll 	nvwgf2um.dll@0x1404ad 	
23 	nvwgf2um.dll 	nvwgf2um.dll@0x13e6d6 	
24 	nvwgf2um.dll 	nvwgf2um.dll@0x70f2 	
25 	nvwgf2um.dll 	nvwgf2um.dll@0xc68b0 	
26 	dlumd32.dll 	dlumd32.dll@0x26f00 	
27 	dlumd32.dll 	dlumd32.dll@0x41e83 	
28 	dxgi.dll 	CD3D10Device::DestroyDriverInstance 	
29 	d3d10_1core.dll 	CDevice::LLOBeginLayerDestruction 	
30 	d3d10_1core.dll 	CBridgeImpl<ILayeredLockOwner,IDXGILayeredDevice,CLayeredObject<CDevice> >::LLOB 	
31 	dxgi.dll 	CD3D10Device::LLOBeginLayerDestruction 	
32 	dxgi.dll 	CBridgeImpl<ILayeredLockOwner,IDXGILayeredDevice,CLayeredObject<CD3D10Device> >: 	
33 	d3d10_1core.dll 	NMultithread::CDevice::LLOBeginLayerDestruction 	
34 	d3d10_1core.dll 	CBridgeImpl<ILayeredLockOwner,IDXGILayeredDevice,CLayeredObject<NMultithread::CD 	
35 	dxgi.dll 	NOutermost::CDevice::LLOBeginLayerDestruction 	
36 	dxgi.dll 	NOutermost::CDevice::FinalRelease 	
37 	dxgi.dll 	TComObject<NOutermost::CDevice>::~TComObject<NOutermost::CDevice> 	
38 	dxgi.dll 	TComObject<NOutermost::CDevice>::`scalar deleting destructor' 	
39 	dxgi.dll 	NOutermost::CDevice::FinalRelease 	
40 	d3d10_1core.dll 	CLayeredObject<NMultithread::CDevice>::CContainedObject::Release 	
41 	xul.dll 	nsRefPtr<nsCSSStyleSheet>::assign_with_AddRef 	obj-firefox/dist/include/nsAutoPtr.h:846
42 	xul.dll 	gfxWindowsPlatform::UpdateRenderMode 	gfx/thebes/gfxWindowsPlatform.cpp:474
43 	xul.dll 	gfxWindowsPlatform::gfxWindowsPlatform 	gfx/thebes/gfxWindowsPlatform.cpp:365
44 	xul.dll 	gfxPlatform::Init 	gfx/thebes/gfxPlatform.cpp:290
45 	xul.dll 	gfxPlatform::GetPlatform 	gfx/thebes/gfxPlatform.cpp:237
46 	xul.dll 	nsDOMClassInfo::Init 	dom/base/nsDOMClassInfo.cpp:4606
47 	xul.dll 	NS_GetDOMClassInfoInstance 	dom/base/nsDOMClassInfo.cpp:5185
48 	xul.dll 	nsFrameMessageManager::QueryInterface 	content/base/src/nsFrameMessageManager.cpp:107
49 	xul.dll 	nsCOMPtr_base::assign_from_qi 	obj-firefox/xpcom/build/nsCOMPtr.cpp:56
50 	xul.dll 	xpcObjectHelper::GetClassInfo 	js/xpconnect/src/xpcObjectHelper.h:71
51 	xul.dll 	XPCWrappedNative::GetNewOrUsed 	js/xpconnect/src/XPCWrappedNative.cpp:493
52 	xul.dll 	XPCConvert::NativeInterface2JSObject 	js/xpconnect/src/XPCConvert.cpp:920
53 	xul.dll 	NativeInterface2JSObject 	js/xpconnect/src/nsXPConnect.cpp:1240
54 	xul.dll 	nsXPConnect::WrapNative 	js/xpconnect/src/nsXPConnect.cpp:1272
55 	xul.dll 	nsJSCID::GetService 	js/xpconnect/src/XPCJSID.cpp:788
56 	xul.dll 	NS_InvokeByIndex_P 	xpcom/reflect/xptcall/src/md/win32/xptcinvoke.cpp:70
57 	xul.dll 	XPCWrappedNative::CallMethod 	js/xpconnect/src/XPCWrappedNative.cpp:2406
58 	xul.dll 	XPC_WN_CallMethod 	js/xpconnect/src/XPCWrappedNativeJSOps.cpp:1478
59 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:352
60 	mozjs.dll 	js::Interpret 	js/src/jsinterp.cpp:2414
61 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:363
62 	mozjs.dll 	js::Invoke 	js/src/jsinterp.h:119
63 	mozjs.dll 	js_fun_call 	js/src/jsfun.cpp:830
64 	mozjs.dll 	js_fun_apply 	js/src/jsfun.cpp:848
65 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:352
66 	mozjs.dll 	js::Interpret 	js/src/jsinterp.cpp:2414
67 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:363
68 	mozjs.dll 	js::Invoke 	js/src/jsinterp.cpp:396
69 	mozjs.dll 	js::CrossCompartmentWrapper::call 	js/src/jswrapper.cpp:736
70 	mozjs.dll 	proxy_Call 	js/src/jsproxy.cpp:1888
71 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:345
72 	mozjs.dll 	js::Invoke 	js/src/jsinterp.cpp:396
73 	mozjs.dll 	js::Shape::get 	js/src/jsscopeinlines.h:296
74 	mozjs.dll 	js::NameOperation 	js/src/jsinterpinlines.h:417
75 	mozjs.dll 	js::Interpret 	js/src/jsinterp.cpp:2517
76 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:363
77 	mozjs.dll 	array_readonlyCommon<ArrayForEachBehavior> 	js/src/jsarray.cpp:3189
78 	mozjs.dll 	array_forEach 	js/src/jsarray.cpp:3226
79 	mozjs.dll 	js::InvokeKernel 	js/src/jsinterp.cpp:352
80 	mozjs.dll 	js::Interpret 	js/src/jsinterp.cpp:2414
81 	mozjs.dll 	js::RunScript 	js/src/jsinterp.cpp:301
82 	mozjs.dll 	js::ExecuteKernel 	js/src/jsinterp.cpp:494
83 	mozjs.dll 	js::XDRState<1>::codeScript 	js/src/vm/Xdr.cpp:147
84 	mozjs.dll 	js::Execute

Comment 1

[Scoobidiver (away)]

(In reply to Marcia Knous [:marcia] from comment #0)
> people were updating there Nvidia graphics driver 310.70
It's confirmed by correlations:
    100% (184/184) vs.   5% (7349/145097) nvwgf2um.dll
        100% (184/184) vs.   1% (734/145097) 9.18.13.1070

Comment 2

[Robert Kaiser]

bp-1b86bc5d-78d4-4c5c-bebe-de9a22121218
User Comments:
After installing Nvidia graphics driver 310.70, firefox keeps crashing when starting up because "hardware acceleration when possible". When I turn it off, Firefox works normally.

We should contact NVidia about this to make sure they know their update causes problems.

Comment 3

[Robert Kaiser]

Benoit, what info should we provide to NVidia when contacting them, to make sure what we tell them is helpful?

Comment 4

[Benoit Jacob [:bjacob] (mostly away)]

Does the bug reproduce consistently under well-undestood conditions (e.g. with a particular driver version)? If yes, giving Nvidia steps to reproduce is the most important thing. They do care to run our testcases, at least if you ask the right person. CC'ing Piers Daniell.

In any case, do give them this information:
 - precise NVIDIA driver version
 - precise Windows version (e.g. Win7 SP1)
 - Firefox version
 - 32bit or 64bit for each of the above
 - anything you know about what NVIDIA devices reproduce. Precise Nvidia device IDs if applicable. Check a few crash reports to see if the AdapterDeviceID's always fall in the same device family (you can use google search to map device ID's to actual device names). 

And do also give them a couple of good Mozilla crash links (the original one in comment 0 looks good, good stack etc). Say that it's a crash in nvwgf2um.dll under CD3D10Device::DestroyDriverInstance (the first resolved symbol in the stack). You can also say it looks like a null pointer dereference (see how the crash address 0x64 is very close to 0, so the 0x64 here is likely just an offset).

Comment 5

[Robert Kaiser]

(In reply to Benoit Jacob [:bjacob] from comment #4)
> In any case, do give them this information:

I'll be putting everything I know in here as well.

>  - precise NVIDIA driver version

nvwgf2um.dll 9.18.13.1070 - to whatever driver version that corresponds

>  - precise Windows version (e.g. Win7 SP1)

The crashes all seem to be on Win7 SP1 (Windows NT 6.1.7601 Service Pack 1), judging from the "Reports" tab in https://crash-stats.mozilla.com/report/list?signature=nvwgf2um.dll@0xf5ddf

>  - Firefox version

As the "Signature Summary" tab in the link above points out, this happens across multiple Firefox versions, on top are the current 17.0.1 release and 18.0b4 beta.

>  - 32bit or 64bit for each of the above

As we only ship 32bit versions of Firefox on beta and release, this has only been seen in 32bit Firefox (so far). We don't see in crash reports if the OS is 32bit or 64bit.

>  - anything you know about what NVIDIA devices reproduce. Precise Nvidia
> device IDs if applicable. Check a few crash reports to see if the
> AdapterDeviceID's always fall in the same device family (you can use google
> search to map device ID's to actual device names). 

Adapter Vendor ID: 0x10de
Adapter Device ID: 0x06c4, 0x0a20, 0x0a66, 0x0dc4, 0x0de1, 0x0df0, 0x0df4, 0x0e22, 0x0e3a, 0x0f00, 0x1040, 0x1056, 0x1080, 0x1183, 0x11c0, 0x1200, 0x1201, 0x1244, 0x1251

That's in the reports I checked manually (by looking through links in the "Reports" tab). There are probably more.

> And do also give them a couple of good Mozilla crash links (the original one
> in comment 0 looks good, good stack etc). Say that it's a crash in
> nvwgf2um.dll under CD3D10Device::DestroyDriverInstance (the first resolved
> symbol in the stack). You can also say it looks like a null pointer
> dereference (see how the crash address 0x64 is very close to 0, so the 0x64
> here is likely just an offset).

Looking through a number of reports, I saw three different stacks:
1bf3e70d-46b5-4272-b335-0e2eb2121218 (28 frames of nvwgf2um.dll)
282cd30c-0efa-49ba-89eb-99dfd2121220 (34 frames of nvwgf2um.dll)
f0107b3d-f778-4c50-8a3b-deeb92121220 (10 frames of nvwgf2um.dll)

All others are repeats of those stacks - and all those call from CD3D10Device::DestroyDriverInstance into that driver library.

Comment 6

[Robert Kaiser]

(In reply to Robert Kaiser (:[e-mail]) from comment #5)
> Looking through a number of reports, I saw three different stacks:
> 1bf3e70d-46b5-4272-b335-0e2eb2121218 (28 frames of nvwgf2um.dll)
> 282cd30c-0efa-49ba-89eb-99dfd2121220 (34 frames of nvwgf2um.dll)
> f0107b3d-f778-4c50-8a3b-deeb92121220 (10 frames of nvwgf2um.dll)

Actually, let's put it into a form that is getting linked here in Bugzilla:
bp-1bf3e70d-46b5-4272-b335-0e2eb2121218
bp-282cd30c-0efa-49ba-89eb-99dfd2121220
bp-f0107b3d-f778-4c50-8a3b-deeb92121220

Comment 7

[Robert Kaiser]

Those crash reports we have been seeing are mostly in the first two seconds of starting Firefox, those that crash later might be slow computers that just take longer to launch it. We don't get any URLs reported, which tell me that we crash even before we can load the home page.

Note that I've now even seen a crash report from Windows 8, see bp-5fc8428b-0725-445b-99a0-1b1272121221 and we're also seeing some on Win7 without SP1, e.g. bp-ace5c4c9-2e81-4dc2-92d4-74a832121221

What I find interesting is that so early in the startup, the last call from Windows libraries before entering nvwgf2um.dll is ::DestroyDriverInstance which sounds like the crash occurs when actually shutting down or deleting a driver instance.

The last frames in Mozilla code before entering Windows libraries are those:

41 	xul.dll 	nsRefPtr<nsCSSStyleSheet>::assign_with_AddRef 	obj-firefox/dist/include/nsAutoPtr.h:846
42 	xul.dll 	gfxWindowsPlatform::UpdateRenderMode 	gfx/thebes/gfxWindowsPlatform.cpp:474
43 	xul.dll 	gfxWindowsPlatform::gfxWindowsPlatform 	gfx/thebes/gfxWindowsPlatform.cpp:365
44 	xul.dll 	gfxPlatform::Init 	gfx/thebes/gfxPlatform.cpp:290

That line in gfxWindowsPlatform::UpdateRenderMode is a call to VerifyD2DDevice() so it looks to me like we crash when trying to verify that D2D works.


Benoit, it looks like the folks at NVidia are unable to reproduce and might have a hard time finding out what's going on there. Does that info from the call stacks or something else we have in the reports give us some info we can forward to them to help them here?

Comment 8

[Benoit Jacob [:bjacob] (mostly away)]

I'm not a D3D specialist, but Bas is: CC'ing him. Maybe there exists a tracing tool that would allow us to record a trace of what's happening here, and give it to Nvidia.

Looking more closely at the crash reports:

An interesting bit is that the AppNotes don't have any of our graphics features annotations (like "D2D? D2D+"...).

That means that we crash _before_ hitting any ScopedGfxFeatureReporter.

Now looking at the stack, we are crashing inside of VerifyD2DDevice that's been inlined into UpdateRenderMode at line 474. This VerifyD2DDevice starts as follows:



void
gfxWindowsPlatform::VerifyD2DDevice(bool aAttemptForce)
{
#ifdef CAIRO_HAS_D2D_SURFACE
    if (mD2DDevice) {
        ID3D10Device1 *device = cairo_d2d_device_get_device(mD2DDevice);

        if (SUCCEEDED(device->GetDeviceRemovedReason())) {
            return;
        }
        mD2DDevice = nullptr;
    }

    mozilla::ScopedGfxFeatureReporter reporter("D2D", aAttemptForce);

    ... (more code) ...



Again, since we don't have any "D2D" annotation in the crash report, we must be crashing right there above the ScopedGfxFeatureReporter line.

That leaves only the 3 lines of code in the if (mD2DDevice) { ... } block.

Bas, any ideas?

Comment 9

[Benoit Jacob [:bjacob] (mostly away)]

I'm told that Bas is currently on vacation which is also going to be my case starting tomorrow. We might need a less-than-optimal short-term fix for this.

Is this crash entirely specific to one driver version such as 310.70 ?

If yes, we can simply blacklist D2D on it.

Comment 10

[Robert Kaiser]

(In reply to Benoit Jacob [:bjacob] from comment #9)
> Is this crash entirely specific to one driver version such as 310.70 ?

It is only with this version from all we know, see also the correlations listed in this bug.

Comment 11

[Benoit Jacob [:bjacob] (mostly away)]

OK. Is this Windows-7 only or does it also affect other Windows versions?

Comment 12

[Benoit Jacob [:bjacob] (mostly away)]

Hm, reports look windows 7-only, however, Direct2D is specific to Windows Vista/7/8 and 1) there are few Vista users with recent drivers and 2) there are very few Win8 users, so we can't necessarily conclude much (unless you have a really large volume of crash reports, like over 1,000, and none of them is outside of Win7).

Filing block request for all Windows versions.

Comment 13

[Benoit Jacob [:bjacob] (mostly away)]

Filed bug 823966 to request a downloaded blocklist entry.

Comment 14

[Danial Horton]

People updating drivers while keeping applications that use them through indirect API's deserve the browser to crash when the device is suddenly lost.

There is no crash what so ever once the driver is installed so I'm calling it user error.

Comment 15

[Danial Horton]

For anyone else KB2488113 applies

Comment 16

[Danial Horton]

Just to copy from the related 310.90 thread

this issue is occuring for a handful of people who have submitted the bug multiple times, the majority of the list is duplicates from the same users.

There is more to this crash then a simple bug in the nvidia drivers, which i cannot reproduce under .70 or .90.

Most of the users on that list ARE infact using the version of the Direct3D 10.1 binary from the above KB article.

There is no hardware correlation, some are using, one is using a 210, another a 480, another a 650, 670 etc.  The last two affected cards would put mine in the affected list.

Now everyone there is claiming the crash happens directly at startup.  Here i am running multiple instances of Firefox 18 - 20, and all indicate under about:support that Direct2D and Write are enabled and working.

It would be negligent to block list the driver at this point. there is not enough individial sample data at this point.

Comment 17

[Danial Horton]

And just to point out, the crash report count does NOT increase as you go further back, it is the same 485 reports (mostly dupes) whether you go 2 or 4 weeks.

At this point, I'm more inclined to point the finger at one of hotfix tuesday's 14 patches.......

Comment 18

[Danial Horton]

Had a reply from someone on the evga forums experiencing this, i have asked him to roll back KB2778930 - Vulnerability in Windows kernel-mode driver and provide feedback on if this resolves the issue.

If so, Nvidia will have to diagnose the issue or work with Microsoft to improve their broken hotfix.

Comment 19

[Danial Horton]

one of the comments reads

"It's been crashing for the passed week. I had one tab open I think it was the steam website. And I clicked restart pc because it had to update but i didn't close the firefox window it just closed it for me when I hit restart. I've reinstalled firefox and restarted my pc. This message still pops up."

That makes me wonder about those hotfixes all the more.
However,
I've got 2 users with fully up to date windows installs, one using 310.70(9600GT), the other 310.90(GTX 480) not having any crashing with Firefox 17 through 21.

Theres the chance this is a case of machines suddenly becoming corrupted by the hotfixes, a number of users have already reported being completely unable to boot with the new patches.

Comment 20

[Benoit Jacob [:bjacob] (mostly away)]

(In reply to Danial Horton from comment #16)
> Just to copy from the related 310.90 thread
> 
> this issue is occuring for a handful of people who have submitted the bug
> multiple times, the majority of the list is duplicates from the same users.

At least bug 828911 is happening for many users, otherwise it wouldn't have become the #15 topcrasher. Since bug 828911 alone is enough to warrant the blocklisting, discussion in the present bug is moot.

Comment 21

[Scoobidiver (away)]

There are only 11 crashes in 18.0.

Comment 22

[Robert Kaiser]

(In reply to Scoobidiver from comment #21)
> There are only 11 crashes in 18.0.

Yes, pretty low volume in both 18 and 19.0b1, now.

Comment 23

[u279076]

We still see these crashes though at very low volume, 2 reports against 46.0.1 currently. This is 100% isolated to NVIDIA 310.70 which was current when this bug was filed but is several versions out of date now. If it'll help we should probably just block these drivers (assuming it's not already). Otherwise I don't see there's much more we can do.