612103 - Crash in SetShaderResource [@ mozilla::layers::ImageLayerD3D10::RenderLayer ] [@ nvwgf2um.dll@0xe1edf ][@ nvwgf2um.dll@0x131def ][@ nvwgf2um.dll@0x9de32 ][@ nvwgf2um.dll@0x9d9b2 ][@ nvwgf2um.dll@0x131fe9 ][@ nvwgf2um.dll@0xe212b ][@ nvwgf2um.dll@0x9de92 ]

Status: RESOLVED FIXED

(Core :: Graphics, defect)

Description

[Scoobidiver (away)]

It is a new crash signature. Crashes first appeared in 4.0b8pre/20101111 build.
It is #165 top crasher in 4.0b8pre for the last 3 days.

Signature	nvwgf2um.dll@0xe1edf
UUID	8c1ca0e0-31b8-416b-9534-4210f2101113
Time 	2010-11-13 15:40:57.168406
Uptime	1359
Last Crash	112971 seconds (1.3 days) before submission
Install Age	8592 seconds (2.4 hours) since version was first installed.
Product	Firefox
Version	4.0b8pre
Build ID	20101113042433
Branch	2.0
OS	Windows NT
OS Version	6.1.7600
CPU	x86
CPU Info	GenuineIntel family 6 model 30 stepping 5
Crash Reason	EXCEPTION_ACCESS_VIOLATION_READ
Crash Address	0x8
App Notes 	AdapterVendorID: 10de, AdapterDeviceID: 0a3c

Frame 	Module 	Signature [Expand] 	Source
0 	nvwgf2um.dll 	nvwgf2um.dll@0xe1edf 	
1 	nvwgf2um.dll 	nvwgf2um.dll@0xb0f5d 	
2 	nvwgf2um.dll 	nvwgf2um.dll@0x16c5dd 	
3 	nvwgf2um.dll 	nvwgf2um.dll@0x3b716 	
4 	d3d10_1core.dll 	d3d10_1core.dll@0x2a280 	
5 	d3d10_1core.dll 	d3d10_1core.dll@0x30ca5 	
6 	d3d10_1.dll 	d3d10_1.dll@0x1b927 	
7 	d3d10_1.dll 	d3d10_1.dll@0x228f3 	
8 	d3d10_1.dll 	d3d10_1.dll@0x1baa0 	
9 	d3d10_1core.dll 	d3d10_1core.dll@0x1cd99 	
10 	d3d10_1.dll 	d3d10_1.dll@0x19987 	
11 	xul.dll 	mozilla::layers::ImageLayerD3D10::RenderLayer 	gfx/layers/d3d10/ImageLayerD3D10.cpp:209
12 	xul.dll 	mozilla::layers::ContainerLayerD3D10::RenderLayer 	gfx/layers/d3d10/ContainerLayerD3D10.cpp:242

The regression range is:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=df1d1ff6b489&tochange=0f17e5f1eb01

More reports at:
http://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=exact&query=&range_value=4&range_unit=weeks&hang_type=any&process_type=any&plugin_field=&plugin_query_type=&plugin_query=&do_query=1&admin=&signature=nvwgf2um.dll%400xe1edf

Comment 1

[Scoobidiver (away)]

For signatures that start with nvwgf2um.dll, there are about 100 crashes/week on 4.0b8pre, so it is #19 top crasher.

Here are some reports:
http://crash-stats.mozilla.com/query/query?product=Firefox&version=Firefox%3A4.0b8pre&platform=windows&branch=2.0&range_value=1&range_unit=weeks&date=12%2F13%2F2010+07%3A53%3A19&query_search=signature&query_type=startswith&query=nvwgf2um.dll&build_id=&process_type=any&hang_type=any&do_query=1

Comment 2

[Joe Drew (not getting mail)]

This might be a regression from asynchronous plugin drawing, which appears in the regression window. 

Does anyone have a way to reproduce this?

Comment 3

[Bas Schouten (:bas.schouten)]

Hrm, this looks like the technique broke somehow, I have no quick idea on what might cause this.

Comment 4

[Marcia Knous [:marcia]]

Here are a few of the comments in the reports:

Crashes lots even though I got latest NVidia driver installed.

It crashed while I was watching a WebM/HTML5 video on YouTube.

I have no idea how/what happened (sg). Minefield restarted OK.

Comment 5

[Bas Schouten (:bas.schouten)]

If we could get the d3d10_1core.dll symbols this would most likely be -much- easier to diagnose. Since we don't have a reproducable scenario.

bjacob: Any chance you could work your aggregation magic and figure out if this is limited to some device/driver?

Comment 6

[timeless]

bas: have you tried grabbing a .dmp and seeing if the ms symbol server has d3d10_1core? it probably does...

Comment 7

[Bas Schouten (:bas.schouten)]

Joe: Can you arrange that? I do not have the required permissions.

Comment 8

[Benoit Jacob [:bjacob] (mostly away)]

It's not limited to some particular device/driver:

The top signature nvwgf2um.dll@0x131def, which is characteristic of driver 260.99, already has lots of different devices: the first 5 are:
0427  G86 [GeForce 8400M GS] 
05e6  GT200b [GeForce GTX 275]
05e3  GT200b [GeForce GTX 285]
0a34  GT216 [GeForce GT 240M]
0622  G94 [GeForce 9600 GT]

That's already 3 generations of NVIDIA cards.

Then other signatures correspond to other driver versions:
nvwgf2um.dll@0xe1edf is driver version 258.96
nvwgf2um.dll@0x9de82 is driver version 188.86, which is old...

Comment 9

[Benjamin Smedberg]

Stack of https://crash-stats.mozilla.com/report/index/22b8e07d-c724-4a17-8573-6f1242101213

>	xul.dll!nsDisplayWrapper::WrapListsInPlace(aBuilder=0x008ba5d8, aFrame=0x00000000, aLists={...})  Line 1411	C++
 	d3d10_1core.dll!CDevice::ID3D10Device1_SetShaderResources_<3>() 	
 	d3d10_1core.dll!NMultithread::CDevice::PSSetShaderResources() 	
 	d3d10_1.dll!D3D10Effects::CEffect::ApplyShaderBlock() 	
 	d3d10_1.dll!D3D10Effects::CEffect::ApplyPassBlock() 	
 	d3d10_1.dll!D3D10Effects::SRuntimePassBlock::Apply() 	
 	xul.dll!mozilla::layers::ImageLayerD3D10::RenderLayer()  Line 210	C++
 	xul.dll!mozilla::layers::ContainerLayerD3D10::RenderLayer()  Line 265	C++
 	xul.dll!mozilla::layers::ContainerLayerD3D10::RenderLayer()  Line 265	C++
 	xul.dll!mozilla::layers::LayerManagerD3D10::Render()  Line 473	C++
 	xul.dll!mozilla::layers::LayerManagerD3D10::EndTransaction(aCallback=0x62ed6cd0, aCallbackData=0x001eca40)  Line 245	C++

Comment 10

[Scott A.]

Just got this crash for the first time, crashed going back in a tab, was on pages I usually am on and my normal browsing behavior, the only thing different was I had a tab that was sitting on imdb.com homepage for a long time.

http://crash-stats.mozilla.com/report/index/46e1aeef-04a3-49c6-9179-fa9752101217

Comment 11

[Ashu]

I have this crash since yesterday's nightly. 2010-12-17

My laptop is Optimus enabled (GT420M) but I don't think that's the problem. I've seen it occur when trying to view YT vids or video streams.

http://crash-stats.mozilla.com/report/index/bp-5c032834-cee7-4790-845d-2bf8f2101218

Comment 12

[Scoobidiver (away)]

Aashish,
Don't add a crash id that is specific to 64-bit build. It is not related.

Comment 13

[Ashu]

It looks like the same crash signature. Why would it make a difference if it's a 64Bit or 32Bit build considering the circumstances?

Comment 14

[Scoobidiver (away)]

> It looks like the same crash signature. Why would it make a difference if it's
> a 64Bit or 32Bit build considering the circumstances?
First your crash signature is not in the bug title even if it is a NVIDIA driver crash, then the stack traces are not similar (mozilla::layers::ImageLayerD3D10::RenderLayer is not inside the stack traces).
You can file a new bug, but read bug 609252 comment 13 first.

Comment 15

[Ashu]

Ok thanks will file a new bug.

Comment 16

[Marcia Knous [:marcia]]

I was able to crash several times in this stack on Windows 7 using this URL:

http://www.winamp.com/?src=toolbar

http://crash-stats.mozilla.com/report/index/b978ff01-75b9-41ee-9954-bb98d2101227
http://crash-stats.mozilla.com/report/index/bp-6a59ddf7-ae63-4061-b3bd-61c6b2101227

Comment 17

[Bas Schouten (:bas.schouten)]

Hrm, that URL doesn't seem to cause any trouble for me (NVidia GT230M) were you doing anything specific?

Comment 18

[Marcia Knous [:marcia]]

My recollection is that crash happened when I loaded the site.

I just crashed on my Win 7 laptop in the same stack. I had 3 tabs open and switched back to the first tab, which  had NY Times loaded in it.

http://crash-stats.mozilla.com/report/index/bp-9a2bcd7f-43f6-4547-bf2d-9ec372101229 is that report

(In reply to comment #17)
> Hrm, that URL doesn't seem to cause any trouble for me (NVidia GT230M) were you
> doing anything specific?

Comment 19

[NVD]

Try installing KB2028560 update, it does fix some bugs in Direct2D and DirectWrite. From the KB article, D2d1.dll, Dwrite.dll and D3d10_1core.dll were all updated to 6.1.7600.20781.

http://support.microsoft.com/kb/2028560

32bit
http://download.microsoft.com/download/9/7/6/976634BE-DC4D-4B7B-9567-209811D7AF14/Windows6.1-KB2028560-v2-x86.msu

64bit
http://download.microsoft.com/download/F/1/A/F1A02F0C-0F64-4E87-9BC1-EFA8CDE464FE/Windows6.1-KB2028560-v2-x64.msu

Comment 20

[Bas Schouten (:bas.schouten)]

I've managed to get this crash, sadly in a release nightly without a debugger attached. I've got that update though, I'm trying to reproduce it in a debug build now, but it seems to be tricky.

Comment 21

[NVD]

Nvidia has released new GeForce 266.35 beta drivers, might want to test them and see if they fix this.

Desktop drivers
http://www.nvidia.com/object/win7-winvista-32bit-266.35-beta-driver.html

http://www.nvidia.com/object/win7-winvista-64bit-266.35-beta-driver.html

Notebook drivers
http://www.nvidia.com/object/notebook-win7-winvista-266.35-beta-driver.html

http://www.nvidia.com/object/notebook-win7-winvista-64bit-266.35-beta-driver.html

Comment 23

[Scoobidiver (away)]

Here are crash reports for 4.0b8:
http://crash-stats.mozilla.com/query/query?product=Firefox&version=Firefox%3A4.0b8&platform=windows&branch=2.0&range_value=1&range_unit=weeks&query_search=signature&query_type=startswith&query=nvwgf2um.dll&build_id=&process_type=any&hang_type=any&do_query=1

There are 7150 crashes/week, so it is #1 top crasher in 4.0b8 for the last week.
It must be a beta9 blocker.

Comment 24

[Scoobidiver (away)]

It is responsible of 3.6% of every 4.0b8 crashes.

Assuming each crash signature is caused by a single driver version, I sorted them out by top crasher:
nvwgf2um.dll@0xe1edf 8.17.12.5896
nvwgf2um.dll@0x131def 8.17.12.6099
nvwgf2um.dll@0x9de32 8.16.11.8829
nvwgf2um.dll@0x9d9b2 8.16.11.8766
nvwgf2um.dll@0xe212b 8.17.12.5896
nvwgf2um.dll@0x9de92 8.16.11.8914
nvwgf2um.dll@0x131fe9 8.17.12.6099
nvwgf2um.dll@0x9de82 8.16.11.8958
nvwgf2um.dll@0x9bec2 8.15.11.8644
nvwgf2um.dll@0x9bf22 8.15.11.8642
nvwgf2um.dll@0x9de22 8.16.11.8911
nvwgf2um.dll@0x9df92 8.16.11.8864
nvwgf2um.dll@0x96d52 8.15.11.8593
nvwgf2um.dll@0x15a9c3 	8.17.12.6099
nvwgf2um.dll@0x9df02 8.16.11.8992
nvwgf2um.dll@0xc600f 8.17.11.9621
nvwgf2um.dll@0x9dd82 8.16.11.8782
nvwgf2um.dll@0x9def2 8.16.11.8988
...
It is well distributed over versions. Even recent versions (release 260) are impacted.

Comment 25

[Jeff Muizelaar [:jrmuizel]]

Bas, can you ask NVidia about this?

Comment 26

[Bas Schouten (:bas.schouten)]

(In reply to comment #18)
> My recollection is that crash happened when I loaded the site.
> 
> I just crashed on my Win 7 laptop in the same stack. I had 3 tabs open and
> switched back to the first tab, which  had NY Times loaded in it.
> 
> http://crash-stats.mozilla.com/report/index/bp-9a2bcd7f-43f6-4547-bf2d-9ec372101229
> is that report
> 
> (In reply to comment #17)
> > Hrm, that URL doesn't seem to cause any trouble for me (NVidia GT230M) were you
> > doing anything specific?

Considering you can reproduce this, is there any chance you could send me a dump with heap included for this crash?

Comment 27

[Jeff Muizelaar [:jrmuizel]]

It seems like the function we're calling in the drivers is probably PsSetShaderResources:
http://msdn.microsoft.com/en-us/library/ff569210%28VS.85%29.aspx

I wonder if we can call PSGetShaderResources and perhaps ID3D10ShaderResourceView::GetDesc ourselves to perhaps trigger the crash earlier.

Comment 28

[Bas Schouten (:bas.schouten)]

(In reply to comment #27)
> It seems like the function we're calling in the drivers is probably
> PsSetShaderResources:
> http://msdn.microsoft.com/en-us/library/ff569210%28VS.85%29.aspx
> 
> I wonder if we can call PSGetShaderResources and perhaps
> ID3D10ShaderResourceView::GetDesc ourselves to perhaps trigger the crash
> earlier.

If I could reproduce that, it's definitely the test I'd run. But we'd still need to catch that crash in a debugger to be able to find out what caused it.

Comment 29

[Bas Schouten (:bas.schouten)]

Attachment: Part 1: Do not draw when texture creation fails.

This will cause us not to draw, or bind NULL as the shader resource (also drawing nothing) when texture creation fails. I don't think this is the cause since ShaderResource creation should return NULL if the texture is NULL, but some additional safety (and a debug warning) can't hurt.

Comment 30

[Bas Schouten (:bas.schouten)]

Attachment: Part 2: Don't draw when ShaderResource belongs to the wrong device.

I suspect this is the real problem here. When our image belongs to an old device that crashed, and we created a new one, we'll still attempt to use that old device's object with our new device, this is bad and we shouldn't do this.

Comment 31

[Jeff Muizelaar [:jrmuizel]]

Comment on attachment 502639 [details] [diff] [review]
Part 1: Do not draw when texture creation fails.

Have you tested this? (By simulating the failure perhaps?)

Patches like this always scare me a bit because they can create more state to worry about and more paths through code. This can end up moving failures to places that are harder to diagnose. Perhaps this situation can also be logged with ReportFailure().

Comment 32

[Bas Schouten (:bas.schouten)]

(In reply to comment #31)
> Comment on attachment 502639 [details] [diff] [review]
> Part 1: Do not draw when texture creation fails.
> 
> Have you tested this? (By simulating the failure perhaps?)
> 
> Patches like this always scare me a bit because they can create more state to
> worry about and more paths through code. This can end up moving failures to
> places that are harder to diagnose. Perhaps this situation can also be logged
> with ReportFailure().

Yes, I've tested this. It makes us draw something bogus in some cases in the plugin area but it doesn't crash. I could also use ReportFailure for this.

Comment 33

[Jeff Muizelaar [:jrmuizel]]

Comment on attachment 502639 [details] [diff] [review]
Part 1: Do not draw when texture creation fails.

Looks fine. Perhaps add the ReportFailure at the call to CreateTexture.

Comment 34

[Jeff Muizelaar [:jrmuizel]]

Comment on attachment 502640 [details] [diff] [review]
Part 2: Don't draw when ShaderResource belongs to the wrong device.

This looks fine. Is there anything we could have done/can still do to prevent this mistake?

Comment 35

[Bas Schouten (:bas.schouten)]

Attachment: Part 0.5: Add ReportFailure function for D3D10 layers.

Comment 36

[Bas Schouten (:bas.schouten)]

Attachment: Part 1: Do not draw when texture creation fails. v2

Comment 37

[Bas Schouten (:bas.schouten)]

Attachment: Part 2: Don't draw when ShaderResource belongs to the wrong device. v2

This adds a similar check for yuvImage.

Comment 38

[Bas Schouten (:bas.schouten)]

http://hg.mozilla.org/mozilla-central/rev/5a1e9d51d095
http://hg.mozilla.org/mozilla-central/rev/04ae4e309307
http://hg.mozilla.org/mozilla-central/rev/c8e72631d44b

Leaving open until crash-stats confirm tomorrow if this has been fixed.

Comment 39

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

Comment on attachment 502973 [details] [diff] [review]
>+void
>+LayerManagerD3D10::ReportFailure(const nsACString &aMsg, HRESULT aCode)
>+{
>+  // We could choose to abort here when hr == E_OUTOFMEMORY.
>+  nsCString msg;
>+  msg.Append(aMsg);
>+  msg.AppendLiteral(" Error code: ");
>+  msg.AppendInt(PRUint32(aCode));
>+  NS_WARNING(msg.BeginReading());
>+}

Should the whole implementation here be ifdef DEBUG?

Comment on attachment 502975 [details] [diff] [review]
>+    if (yuvImage->mDevice != device()) {
>+	// These shader resources were created for an old device! Can't draw
>+	// that here.
>+	return;
>+    }
>+

Tabs!

Comment 40

[NVD]

Microsoft has released KB2454826 update, it replaces the older KB2028560 update and installs newer Direct2D and DirectWrite dlls which fixes more bugs.

http://support.microsoft.com/kb/2454826

32bit
http://download.microsoft.com/download/5/E/4/5E404378-9A5D-41AB-AFBA-1AC04F3B2A13/Windows6.1-KB2454826-x86.msu

64bit
http://download.microsoft.com/download/D/B/D/DBD62263-2627-49CB-B675-AA1601EBE0BD/Windows6.1-KB2454826-x64.msu

Comment 41

[Bas Schouten (:bas.schouten)]

(In reply to comment #39)
> Comment on attachment 502973 [details] [diff] [review]
> >+void
> >+LayerManagerD3D10::ReportFailure(const nsACString &aMsg, HRESULT aCode)
> >+{
> >+  // We could choose to abort here when hr == E_OUTOFMEMORY.
> >+  nsCString msg;
> >+  msg.Append(aMsg);
> >+  msg.AppendLiteral(" Error code: ");
> >+  msg.AppendInt(PRUint32(aCode));
> >+  NS_WARNING(msg.BeginReading());
> >+}
> 
> Should the whole implementation here be ifdef DEBUG?

I don't think so, longer run the intention is to connect this to some kind of runtime recording/logging system. In any case the optimizer should figure this out.

> 
> Comment on attachment 502975 [details] [diff] [review]
> >+    if (yuvImage->mDevice != device()) {
> >+	// These shader resources were created for an old device! Can't draw
> >+	// that here.
> >+	return;
> >+    }
> >+
> 
> Tabs!

Argh, that's what I get for working in cairo code. will fix.

Comment 42

[Bas Schouten (:bas.schouten)]

After fixing the tabs and examining crash data, I'm going to optimistically mark this fixed!

Comment 44

[Bas Schouten (:bas.schouten)]

Attachment: Followup: Only do a warning in the case of failure

A little oops here introduced excessive NS_WARNINGs for no reason. We should fix that.

Comment 46

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Pushed followup for spurious warnings

http://hg.mozilla.org/mozilla-central/rev/fa1a4b6abff0