Spike in crashes with Nvidia driver DLLs in Firefox 50

RESOLVED FIXED

Status

()

Core
Graphics
P3
critical
RESOLVED FIXED
a year ago
9 months ago

People

(Reporter: philipp, Unassigned)

Tracking

({crash, regression})

50 Branch
All
Windows
crash, regression
Points:
---

Firefox Tracking Flags

(platform-rel +, firefox49 unaffected, firefox50blocking fixed, firefox51 fixed, firefox52 fixed)

Details

(Whiteboard: [platform-rel-nVidia][gfx-noted], crash signature)

(Reporter)

Description

a year ago
[Tracking Requested - why for this release]:
in stability data we are seeing an increase of crash signatures relating to dlls from nvidia's graphics drivers after firefox 50 entered the beta stage.
those spiking crashes are accounting for at least 3% of browser crashes during the 50.0 beta cycle thus far.
See Also: → bug 1307543
(Reporter)

Updated

a year ago
Summary: Spike in crashes with Nvidia driver DLLs in FIrefox 50 → Spike in crashes with Nvidia driver DLLs in Firefox 50
I would consider this blocking for Fx50 release.
tracking-firefox50: ? → blocking

Updated

a year ago
See Also: → bug 1292273
See Also: → bug 1292311
(Reporter)

Comment 2

a year ago
bug 1292311 already spiked with 49, so i didn't include it in this list of signatures.
Right, but maybe as a last resort we could evaluate the same workaround we employed for bug 1292311 (blacklisting more).
platform-rel: --- → ?
This is being tracked by NVIDIA bug 1828302.
Whiteboard: [platform-rel-nVidia]
Marco, we were wondering if you could look into specific DLL versions or get other info to help make a blacklisting decision here.
Flags: needinfo?(mcastelluccio)
The 50b spike crashes show 0x10de and 0x8086 as adapter vendor IDs. There seems to be a range of adapter driver versions... from this link Marco showed me yesterday: https://crash-stats.mozilla.com/search/?signature=%5Envwgf2umx.dll&signature=%5Envwgf2um.dll&product=Firefox&version=50.0b&date=%3E%3D2016-10-11T17%3A09%3A00.000Z&date=%3C2016-10-18T17%3A09%3A00.000Z&_sort=-date&_facets=signature&_facets=adapter_driver_version&_facets=adapter_device_id&_facets=adapter_vendor_id&_facets=platform_version&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports
A fix for at least a subset of the crashes here is being worked on in bug 1292273. Let's see what happens after that lands.
This is looking very video-related, but reluctant to say for sure.

Many (most?) URLs are youtube, facebook, adult video, or other sites with inline autoplaying video.

Further, I see a lot of "DXVA2D3D9 video decoding is disabled due to a previous crash.". Which is interesting; not sure yet whether this is useful information.

Most of the stacks have some mention of BufferTextureHost or IMFYCbCrImage or similar, which AIUI are generally used for video more than anything.
Many of the signatures in the list are highly correlated with 'DXVA2D3D9?' in the app notes, some with 'DXVA2D3D9+'. If I remember correctly we're only adding these annotations when we start playing a video.

The user who's able to reproduce crashes from bug 1294748 and bug 1308863 is able to do so when playing videos.

I've also noticed many of the signatures are correlated with Windows 7 and many with "D3D11 Layers+" in app_notes.
Whiteboard: [platform-rel-nVidia] → [platform-rel-nVidia][gfx-noted]
This is an Nvidia version of bug 1282562 (which is a superset of bug 1282564), which go back to at least 47, if not earlier.  Since it's a D2D crash, it will always have D3D11+ - that's the only time we have D2D in play.

It may spike more on Nvidia because we allowed more Nvidia cards to be accelerated in 50, under bug 1284322, which got backed out in bug 1292311, and uplifted to 50 and 51.
Priority: -- → P3
(In reply to Milan Sreckovic [:milan] from comment #10)
> This is an Nvidia version of bug 1282562 (which is a superset of bug
> 1282564), which go back to at least 47, if not earlier.  Since it's a D2D
> crash, it will always have D3D11+ - that's the only time we have D2D in play.

The signatures containing "UpdateSubresource" regressed between 49 Beta and 50 Beta (both with NVIDIA and Intel). The Intel signatures went back to normal (see bug 1307543 comment 2) after bug 1291084 was uplifted. The NVIDIA signatures were unaffected by that uplift.

> It may spike more on Nvidia because we allowed more Nvidia cards to be
> accelerated in 50, under bug 1284322, which got backed out in bug 1292311,
> and uplifted to 50 and 51.

The backout in bug 1292311 didn't improve the situation here.
Bug 1284322 landed in 49, so it can't be responsible for the increase in crash volume between 49 Beta and 50 Beta.
That said, it appears that for some reason things are going back to 49.0b levels in 50.0b10 (e.g. the signature from bug 1308863 disappeared completely in 50.0b10).

The pushlog doesn't contain anything obviously related: https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=FIREFOX_50_0b9_RELEASE&tochange=FIREFOX_50_0b10_RELEASE.
(In reply to Milan Sreckovic [:milan] from comment #10)
> It may spike more on Nvidia because we allowed more Nvidia cards to be
> accelerated in 50, under bug 1284322, which got backed out in bug 1292311,
> and uplifted to 50 and 51.

Bug 1292311 isn't enough to account for this. Most crashes were with relatively recent drivers and wouldn't have been affected by that blacklisting.
platform-rel: ? → +
(Reporter)

Comment 14

a year ago
all these nvidia related crashes are gone or reclining back to their pre-50 levels in 50.0b10 - this is probably due to the same underlying reason than we saw with bug 1308863 (see https://bugzilla.mozilla.org/show_bug.cgi?id=1308863#c19).

i'm going to look out if the crash pattern is gone in beta 11 as well and mark the bug as fixed if this is the case...
Flags: needinfo?(madperson)
See Also: → bug 1308863
(Reporter)

Comment 15

a year ago
still looking good in 50.0b11, so this was probably fixed by bug 1308418 too.
Status: NEW → RESOLVED
Last Resolved: a year ago
status-firefox50: affected → fixed
status-firefox51: affected → fixed
status-firefox52: affected → fixed
Depends on: 1308418
Flags: needinfo?(mcastelluccio)
Flags: needinfo?(madperson)
Resolution: --- → FIXED
Blocks: 1294748
No longer blocks: 1294748
Not sure this is fixed, we're still getting ~300 crashes/week, but the spike is certainly gone.  If you look at different vendors, it's more like ~900 crashes/week.  See also bug 1207993.
See Also: → bug 1207993
You need to log in before you can comment on or make changes to this bug.