Closed Bug 1294748 Opened 8 years ago Closed 8 years ago

Crash in nvwgf2umx.dll & nvd3dumx.dll

Categories

(Core :: Audio/Video: Playback, defect, P3)

Product:
Core
Component:
Audio/Video: Playback
Version:
50 Branch
Platform:
x86_64
Windows 7
Type:
defect

Tracking

()

Status:
VERIFIED FIXED
Milestone:
mozilla52
Tracking Flags:
Tracking Status
firefox49 --- unaffected
firefox-esr45 --- unaffected
firefox50 --- verified
firefox51 --- verified
firefox52 --- verified

People

(Reporter: Virtual, Assigned: kinetik)

References

Details

(6 keywords, Whiteboard: [gfx-noted] [fixed by patches from bug #1308418])

Crash Data

Attachments

(3 files, 1 obsolete file)

Graphics section from about:config.txt
1.36 KB, text/plain
Details
Graphics win 7 x64.txt
861 bytes, text/plain
Details
about:support.rtf
4.82 KB, text/rtf
Details 
STR:
1. Open some movies (in full HD or better) in about 2-5 tabs in one window
2. Watch one movie for some amount of time and let others buffer for later watching
3. In meantime after some movie watching go to about:support to see that some errors are pilling up like:
> [D2D1.1] 4CreateBitmap failure...
4. In about 10-30min after over 20-40 errors like this crash will happen

Crashlog reports:
https://crash-stats.mozilla.com/report/index/0f02c2f7-2977-48b8-b4ed-2123f2160812
https://crash-stats.mozilla.com/report/index/f279a73b-47e6-422f-8b8a-a8a2c2160812
Whiteboard: [gfx-noted]
Same STR but different signature [@ nvd3dumx.dll@0x7f5154 ]
https://crash-stats.mozilla.com/report/index/c03c94e3-8f21-43b9-ba45-7c1312160818
Crash Signature: [@ nvwgf2umx.dll@0x6be8eb ] → [@ nvwgf2umx.dll@0x6be8eb ] [@ nvd3dumx.dll@0x7f5154 ]
Summary: Crash in [@ nvwgf2umx.dll@0x6be8eb ] → Crash in [@ nvwgf2umx.dll@0x6be8eb ] & [@ nvd3dumx.dll@0x7f5154 ]
Now with [@ nvd3dumx.dll@0x89085a ]
https://crash-stats.mozilla.com/signature/?product=Firefox&signature=nvd3dumx.dll%400x89085a
Crash Signature: [@ nvwgf2umx.dll@0x6be8eb ] [@ nvd3dumx.dll@0x7f5154 ] → [@ nvwgf2umx.dll@0x6be8eb ] [@ nvd3dumx.dll@0x7f5154 ] [@ nvd3dumx.dll@0x89085a ]
Summary: Crash in [@ nvwgf2umx.dll@0x6be8eb ] & [@ nvd3dumx.dll@0x7f5154 ] → Crash in [@ nvwgf2umx.dll@0x6be8eb ] & [@ nvd3dumx.dll@0x7f5154 ] & [@ nvd3dumx.dll@0x89085a ]
Crashed with [@ nvwgf2umx.dll@0x6c9ddb ]
https://crash-stats.mozilla.com/report/index/98b04298-77a0-4b6f-9b42-4ab862160821
when I was closing the tab with movie (only one).
Crash Signature: [@ nvwgf2umx.dll@0x6be8eb ] [@ nvd3dumx.dll@0x7f5154 ] [@ nvd3dumx.dll@0x89085a ] → [@ nvwgf2umx.dll@0x6be8eb ] [@ nvd3dumx.dll@0x7f5154 ] [@ nvd3dumx.dll@0x89085a ] [@ nvwgf2umx.dll@0x6c9ddb ]
Summary: Crash in [@ nvwgf2umx.dll@0x6be8eb ] & [@ nvd3dumx.dll@0x7f5154 ] & [@ nvd3dumx.dll@0x89085a ] → Crash in [@ nvwgf2umx.dll@0x6be8eb ] & [@ nvd3dumx.dll@0x7f5154 ] & [@ nvd3dumx.dll@0x89085a ] & [@ nvwgf2umx.dll@0x6c9ddb ]
Crash with [[@ nvwgf2umx.dll@0x6cb08b ]
https://crash-stats.mozilla.com/report/index/543bd15a-5344-4062-809f-adf412160901
while I was watching embedded YouTube movie.
Crash Signature: [@ nvwgf2umx.dll@0x6be8eb ] [@ nvd3dumx.dll@0x7f5154 ] [@ nvd3dumx.dll@0x89085a ] [@ nvwgf2umx.dll@0x6c9ddb ] → [@ nvd3dumx.dll@0x7f5154 ] [@ nvd3dumx.dll@0x89085a ] [@ nvwgf2umx.dll@0x6be8eb ] [@ nvwgf2umx.dll@0x6c9ddb ] [@ nvwgf2umx.dll@0x6cb08b ]
Summary: Crash in [@ nvwgf2umx.dll@0x6be8eb ] & [@ nvd3dumx.dll@0x7f5154 ] & [@ nvd3dumx.dll@0x89085a ] & [@ nvwgf2umx.dll@0x6c9ddb ] → Crash in nvwgf2umx.dll & nvd3dumx.dll
Hi Peter,
Do you think you can help take a look at this?
Flags: needinfo?(howareyou322)
Bulk move of gfx-noted bugs without priority to P3 for tracking.
Priority: -- → P3
Crash [@ nvwgf2umx.dll@0x6cb08b ] when watching YouTube movie, only one window with one tab.

https://crash-stats.mozilla.com/report/index/e7a4fbd1-0a95-44ec-ae8c-1a0bb2160922
Component: Graphics → Audio/Video: Playback
Crash at [@ nvwgf2umx.dll@0x6cb61b ] when watching movie and some on them was in the background.

https://crash-stats.mozilla.com/report/index/672f394a-8c61-424f-8930-0440c2160928
Crash Signature: [@ nvd3dumx.dll@0x7f5154 ] [@ nvd3dumx.dll@0x89085a ] [@ nvwgf2umx.dll@0x6be8eb ] [@ nvwgf2umx.dll@0x6c9ddb ] [@ nvwgf2umx.dll@0x6cb08b ] → [@ nvd3dumx.dll@0x7f5154 ] [@ nvd3dumx.dll@0x89085a ] [@ nvwgf2umx.dll@0x6be8eb ] [@ nvwgf2umx.dll@0x6c9ddb ] [@ nvwgf2umx.dll@0x6cb08b ] [@ nvwgf2umx.dll@0x6cb61b ]
Next one [@ nvd3dumx.dll@0x890bba ]

https://crash-stats.mozilla.com/report/index/b1e01a8a-9283-4853-8608-ef7d32160930
Crash Signature: [@ nvd3dumx.dll@0x7f5154 ] [@ nvd3dumx.dll@0x89085a ] [@ nvwgf2umx.dll@0x6be8eb ] [@ nvwgf2umx.dll@0x6c9ddb ] [@ nvwgf2umx.dll@0x6cb08b ] [@ nvwgf2umx.dll@0x6cb61b ] → [@ nvd3dumx.dll@0x7f5154 ] [@ nvd3dumx.dll@0x89085a ] [@ nvd3dumx.dll@0x890bba ] [@ nvwgf2umx.dll@0x6be8eb ] [@ nvwgf2umx.dll@0x6c9ddb ] [@ nvwgf2umx.dll@0x6cb08b ] [@ nvwgf2umx.dll@0x6cb61b ]
Adding also [@ nvwgf2umx.dll@0x966790 ] (top35) and [@ nvwgf2umx.dll@0x20ec12 ] (top9) top crashes for Firefox 52, looking on crashlog reports same thing as this one mine here.
Crash Signature: [@ nvd3dumx.dll@0x7f5154 ] [@ nvd3dumx.dll@0x89085a ] [@ nvd3dumx.dll@0x890bba ] [@ nvwgf2umx.dll@0x6be8eb ] [@ nvwgf2umx.dll@0x6c9ddb ] [@ nvwgf2umx.dll@0x6cb08b ] [@ nvwgf2umx.dll@0x6cb61b ] → [@ nvwgf2umx.dll@0x966790 ] [@ nvwgf2umx.dll@0x20ec12 ] [@ nvd3dumx.dll@0x7f5154 ] [@ nvd3dumx.dll@0x89085a ] [@ nvd3dumx.dll@0x890bba ] [@ nvwgf2umx.dll@0x6be8eb ] [@ nvwgf2umx.dll@0x6c9ddb ] [@ nvwgf2umx.dll@0x6cb08b ] [@ nvwgf2umx.dll@0x6cb61b ]
Moreover in the last crashing threads the msmpeg2vdec.dll@0xe59cd is crashing, so it could be related to bug #1254389.
Keywords: top50, topcrash-win
(90.41% in signature vs 60.08% overall) Addon "skia-windows-nightly@experiments.mozilla.org" = true

More likely with Skia?
Could be, but in my case it happens without Skia (see attachment).
(In reply to Virtual_ManPL [:Virtual] - (ni? me) from comment #15)
> Could be, but in my case it happens without Skia (see attachment).

The signatures in the bug might be related to multiple bugs.
There's a spike around Sep 24, which might be when we launched
the Skia experiment.
(In reply to Marco Castelluccio [:marco] from comment #16)
> (In reply to Virtual_ManPL [:Virtual] - (ni? me) from comment #15)
> > Could be, but in my case it happens without Skia (see attachment).
> 
> The signatures in the bug might be related to multiple bugs.
> There's a spike around Sep 24, which might be when we launched
> the Skia experiment.

The spike looks more matching with us changing trains and getting more people with different hardware, rather than the Skia experiment.
I believe the Nvidia drivers have "x" at the end for the 64-bit version, so searching for those will always give you the 64-bit version.  If you get rid of the trailing x in the dll names in the search, you get only the 32-bit flavour: https://crash-stats.mozilla.com/search/?signature=~nvwgf2um.dll&signature=~nvd3dum.dll&product=Firefox&_sort=-date&_facets=signature&_facets=cpu_arch&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-cpu_arch
(In reply to Milan Sreckovic [:milan] from comment #19)
> I believe the Nvidia drivers have "x" at the end for the 64-bit version, so
> searching for those will always give you the 64-bit version.  If you get rid
> of the trailing x in the dll names in the search, you get only the 32-bit
> flavour:
> https://crash-stats.mozilla.com/search/?signature=~nvwgf2um.
> dll&signature=~nvd3dum.dll&product=Firefox&_sort=-
> date&_facets=signature&_facets=cpu_arch&_columns=date&_columns=signature&_col
> umns=product&_columns=version&_columns=build_id&_columns=platform#facet-
> cpu_arch

Oh, that is good to know. I assumed they would add '64' at the end of the name,
like many other DLLs do.
It's still interesting though that Virtual can reproduce the crash in a 64-bit
build, but not in a 32-bit one.
(In reply to Virtual_ManPL [:Virtual] - (ni? me) from comment #0)
> ...
> 3. In meantime after some movie watching go to about:support to see that
> some errors are pilling up like:
> > [D2D1.1] 4CreateBitmap failure...

Not surprising, given that we're asking for hardware textures with dimensions like 64800x90 and 42840x90, and we fail with the invalid parameter errors.  I don't know where those requests would be coming from, but they all seem to be 90 pixels high.

Is this on a clean profile?
Marco, what's the correlation with DXVA2D3D9+ in the AppNotes?
For nvwgf2umx.dll@0x20ec12 on Beta, ~89% have "DXVA2D3D9?", ~43% have "DXVA2D3D9-", ~46% have "DXVA2D3D9+", ~41% have ("DXVA2D3D9-" in app_notes and "DXVA2D3D9 video decoding is disabled due to a previous crash." in graphics_critical_error).

If you want I can run the analysis considering several signatures together, but
we need to define which signatures we want to consider.
I've done it in bug 1307543 with all signatures containing "UpdateSubresource", with similar results.
See Also: → 1308863
Depends on: 1309225
See Also: → 1274882
I've added the new signatures from Virtual's crash reports. There might be different
causes for those crashes.
Crash Signature: ] → ] [@ nvwgf2umx.dll | NDXGI::CDevice::NTStatusToHResult ] [@ nvd3dumx.dll | WerpValidatePebHeader ] [@ nvwgf2umx.dll | RtlAllocateHeap | nvwgf2umx.dll | CCLSAllocator::Free ] [@ nvwgf2umx.dll | HeapFree | nvwgf2umx.dll | BaseThreadInitThunk ] [@ nvwg…
Virtual, any chance you could find a regression range with mozregression?
Flags: needinfo?(virtual)
(In reply to Milan Sreckovic [:milan] from comment #21)
> Is this on a clean profile?

No.
But nothing oddly changed in about:profile and I don't think these extensions (Classic Theme Restorer, Decentraleyes, Don't track me Google, DownThemAll!, h264ify, Session Manager and uBlock Origin) are causing these crashes or these odd errors in about:support, but who knows. I will redo test on Nightly Portable.



(In reply to Marco Castelluccio [:marco] from comment #25)
> Virtual, any chance you could find a regression range with mozregression?

I will try to find regression range.
Flags: needinfo?(virtual)
See Also: → 1310600
Hey Virtual, have you had time to find the regression range? I'm hoping that it will help us finding why NVIDIA is so crashy on 50.
Is this bug related to bug 1310600?
Flags: needinfo?(mcastelluccio)
It could be, some of the signatures of Virtual's crashes are the same or similar as bug 1310600.
Flags: needinfo?(mcastelluccio)
Flags: needinfo?(virtual)
Note that we have a lot of Win64 media tests hitting nvwgf2umx.dll crashes at the moment. They started around August 10th, which is around the time that bug 1289525 and bug 1291163 landed. Conveniently, both of those were uplifted to Fx50 as well.
https://bugzilla.mozilla.org/buglist.cgi?keywords=intermittent-failure%2C%20&keywords_type=allwords&list_id=13273800&short_desc_type=allwordssubstr&short_desc=nvwgf2umx.dll&resolution=---&query_format=advanced

Any chance there's a relationship?
George, Nical, what do you think of Ryan's guess in comment 30?
Flags: needinfo?(nical.bugzilla)
Flags: needinfo?(gwright)
I've spoken with Ryan about it. We're waiting on more data. I think it's highly unlikely to be my patch, but Nical's might be the culprit. Nical already has some theories as to what might be causing it if it does turn out to be his patch.
Flags: needinfo?(gwright)
(In reply to Marco Castelluccio [:marco] from comment #27)
> Hey Virtual, have you had time to find the regression range? I'm hoping that
> it will help us finding why NVIDIA is so crashy on 50.

I'm very sorry, but now I don't have that much time to spend on finding regression range as steps to reproduce are very time consuming, because I'm in the course of finding the new job.

I just want to add that it crashed with new signature - [@ nvwgf2umx.dll | kernel32.dll@0x21a09 ]

crashlog report:
https://crash-stats.mozilla.com/report/index/e6e3ad3f-d0b2-48d5-9d74-4fe902161020
Crash Signature: nvwgf2umx.dll ] [@ nvd3dumx.dll | HeapFree | nvd3dumx.dll | IcmSelectColorTransform ] → nvwgf2umx.dll ] [@ nvd3dumx.dll | HeapFree | nvd3dumx.dll | IcmSelectColorTransform ] [@ nvwgf2umx.dll | kernel32.dll@0x21a09 ]
Flags: needinfo?(virtual)
Ok, no problem. I guess you also can't try RyanVM builds from bug 1308863 comment 14?

Could we share your crash dumps with NVIDIA? It would make it easier to debug the problem, and would not require you to spend any time reproducing. They might contain some personal information (e.g. the URLs you were visiting), so I understand if you aren't comfortable sharing them.
Flags: needinfo?(virtual)
You already replied to my first question in that bug :)
Rares, can some of your people try to reproduce given the STR here? If anybody is successful, it would be great to see if either of the Try builds from bug 1308863 comment 14 help too.
Flags: needinfo?(rares.bologa)
(In reply to Ryan VanderMeulen [:RyanVM] from comment #36)
> Rares, can some of your people try to reproduce given the STR here? If
> anybody is successful, it would be great to see if either of the Try builds
> from bug 1308863 comment 14 help too.

Ovidiu will look into this.
Flags: needinfo?(rares.bologa) → needinfo?(ovidiu.boca)
(In reply to Marco Castelluccio [:marco] from comment #34)
> Ok, no problem. I guess you also can't try RyanVM builds from bug 1308863
> comment 14?

For now, the only thing I can do is the advise and the recommendation of backing out the suspected patches. There are available 3 test branches of Firefox which are affected by these issues (Beta [50], Aurora [51] and Nightly [52]) to test with at least 3 reverted packs of patches in each branch to diagnose what and which patches were the cause.


(In reply to Marco Castelluccio [:marco] from comment #34)
> Could we share your crash dumps with NVIDIA? It would make it easier to
> debug the problem, and would not require you to spend any time reproducing.
> They might contain some personal information (e.g. the URLs you were
> visiting), so I understand if you aren't comfortable sharing them.

Sure thing, I'm allowing to this. If not, I wouldn't be sending crash reports and URLs with them.
Flags: needinfo?(virtual)
Version: 51 Branch → 50 Branch
Attached file Graphics win 7 x64.txt — 
I tested this issue on Windows 7 x64 with FF Nightly 52.0a1, FF 51.0a2, FF beta 50.0b9, the graphic card: NVIDIA GeForce GTX 750 Ti but I couldn't reproduce the crashes. 

I followed the steps from description several times but with no success. I attached the section with my graphic card from about:support. The only think that is not the same with the graphic card from reporter is the hardware decoding section "Hardware H264 Decoding	No; Failed to create H264 decoder". Please see the attached file. 

Please feel free to contact me for further investigation if you think is necessary.
Flags: needinfo?(ovidiu.boca)
(In reply to ovidiu boca[:Ovidiu] from comment #39)
> The only think
> that is not the same with the graphic card from reporter is the hardware
> decoding section "Hardware H264 Decoding	No; Failed to create H264 decoder".
> Please see the attached file. 

This could be because you are not using latest Windows 7 version with Service Pack 1 and with latest patches (and I'm not talking about botnet ones [aka diagnostics, telemetry and data gathering]).
> 
> This could be because you are not using latest Windows 7 version with
> Service Pack 1 and with latest patches (and I'm not talking about botnet
> ones [aka diagnostics, telemetry and data gathering]).

I'm using the latest Win 7 with SP1 .
(In reply to ovidiu boca[:Ovidiu] from comment #39)
> I followed the steps from description several times but with no success. I
> attached the section with my graphic card from about:support. The only think
> that is not the same with the graphic card from reporter is the hardware
> decoding section "Hardware H264 Decoding	No; Failed to create H264 decoder".

This is probably important to be able to reproduce the crash.
Follow up - I'm pretty sure that patches from bug #1308418 fixed the issue, same like with bug #1308863 and bug #1310600, so probably bug #1290425 regressed all of these issues and also looking on crash rates I'm marking this as fixed.



(In reply to ovidiu boca[:Ovidiu] from comment #41)
> > 
> > This could be because you are not using latest Windows 7 version with
> > Service Pack 1 and with latest patches (and I'm not talking about botnet
> > ones [aka diagnostics, telemetry and data gathering]).
> 
> I'm using the latest Win 7 with SP1 .

But probably not with latest patches like I wrote and I'm basing on your DirectWrite version.
Blocks: 1290425
Status: NEW → RESOLVED
Has Regression Range: --- → yes
Has STR: --- → yes
Closed: 8 years ago
status-firefox50: affected → fixed
status-firefox51: affected → fixed
status-firefox52: affected → fixed
Depends on: 1310600
Resolution: --- → FIXED
Whiteboard: [gfx-noted] → [gfx-noted] [fixed by patches from bug #1308418]
Target Milestone: --- → mozilla52
droping ni
Flags: needinfo?(nical.bugzilla)
Flags: needinfo?(howareyou322)
(In reply to ovidiu boca[:Ovidiu] from comment #39)
> decoding section "Hardware H264 Decoding	No; Failed to create H264 decoder".
> Please see the attached file. 

this indicates that you haven't installed microsoft media feature pack.
https://www.microsoft.com/en-us/download/details.aspx?id=16546
Thanks for your suggestion Jean-Yves, also Ryan suggested to do that, but from unknown reason in the browser I see this message.
But did you install it ?

what's your about:support ? did you change anything in the config such as media.mp4.enabled preference?
Yes I have installed it but with no change in about:support under decoding section "Hardware H264 Decoding" 
Here is the link that I used for update https://www.microsoft.com/en-us/download/details.aspx?id=36805. I don't remember changing something in config such as media.mp4.enabled preference.
(In reply to ovidiu boca[:Ovidiu] from comment #48)
> Yes I have installed it but with no change in about:support under decoding
> section "Hardware H264 Decoding" 
> Here is the link that I used for update
> https://www.microsoft.com/en-us/download/details.aspx?id=36805. I don't
> remember changing something in config such as media.mp4.enabled preference.

Can you share the contents of your about:support page? This way we can see what are the values of your preferences.
Attached file about:support.rtf — 
In the attached file is the about:support content.
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: