1310600 - Spike in crashes with Nvidia driver DLLs in Firefox 50

Status: RESOLVED FIXED

(Core :: Graphics, defect, P3)

Description

[[:philipp]]

[Tracking Requested - why for this release]:
in stability data we are seeing an increase of crash signatures relating to dlls from nvidia's graphics drivers after firefox 50 entered the beta stage.
those spiking crashes are accounting for at least 3% of browser crashes during the 50.0 beta cycle thus far.

Comment 1

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

I would consider this blocking for Fx50 release.

Comment 2

[[:philipp]]

bug 1292311 already spiked with 49, so i didn't include it in this list of signatures.

Comment 3

[Marco Castelluccio [:marco]]

Right, but maybe as a last resort we could evaluate the same workaround we employed for bug 1292311 (blacklisting more).

Comment 4

[David Bolter [:davidb] (NeedInfo me for attention)]

This is being tracked by NVIDIA bug 1828302.

Comment 5

[Andrew Overholt [:overholt]]

Marco, we were wondering if you could look into specific DLL versions or get other info to help make a blacklisting decision here.

Comment 6

[David Bolter [:davidb] (NeedInfo me for attention)]

The 50b spike crashes show 0x10de and 0x8086 as adapter vendor IDs. There seems to be a range of adapter driver versions... from this link Marco showed me yesterday: https://crash-stats.mozilla.com/search/?signature=%5Envwgf2umx.dll&signature=%5Envwgf2um.dll&product=Firefox&version=50.0b&date=%3E%3D2016-10-11T17%3A09%3A00.000Z&date=%3C2016-10-18T17%3A09%3A00.000Z&_sort=-date&_facets=signature&_facets=adapter_driver_version&_facets=adapter_device_id&_facets=adapter_vendor_id&_facets=platform_version&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports

Comment 7

[Marco Castelluccio [:marco]]

A fix for at least a subset of the crashes here is being worked on in bug 1292273. Let's see what happens after that lands.

Comment 8

[Edwin Flores [inactive from 2016-12-01] [:eflores] [:edwin]]

This is looking very video-related, but reluctant to say for sure.

Many (most?) URLs are youtube, facebook, adult video, or other sites with inline autoplaying video.

Further, I see a lot of "DXVA2D3D9 video decoding is disabled due to a previous crash.". Which is interesting; not sure yet whether this is useful information.

Most of the stacks have some mention of BufferTextureHost or IMFYCbCrImage or similar, which AIUI are generally used for video more than anything.

Comment 9

[Marco Castelluccio [:marco]]

Many of the signatures in the list are highly correlated with 'DXVA2D3D9?' in the app notes, some with 'DXVA2D3D9+'. If I remember correctly we're only adding these annotations when we start playing a video.

The user who's able to reproduce crashes from bug 1294748 and bug 1308863 is able to do so when playing videos.

I've also noticed many of the signatures are correlated with Windows 7 and many with "D3D11 Layers+" in app_notes.

Comment 10

[Milan Sreckovic [:milan] (needinfo for best results)]

This is an Nvidia version of bug 1282562 (which is a superset of bug 1282564), which go back to at least 47, if not earlier.  Since it's a D2D crash, it will always have D3D11+ - that's the only time we have D2D in play.

It may spike more on Nvidia because we allowed more Nvidia cards to be accelerated in 50, under bug 1284322, which got backed out in bug 1292311, and uplifted to 50 and 51.

Comment 11

[Marco Castelluccio [:marco]]

(In reply to Milan Sreckovic [:milan] from comment #10)
> This is an Nvidia version of bug 1282562 (which is a superset of bug
> 1282564), which go back to at least 47, if not earlier.  Since it's a D2D
> crash, it will always have D3D11+ - that's the only time we have D2D in play.

The signatures containing "UpdateSubresource" regressed between 49 Beta and 50 Beta (both with NVIDIA and Intel). The Intel signatures went back to normal (see bug 1307543 comment 2) after bug 1291084 was uplifted. The NVIDIA signatures were unaffected by that uplift.

> It may spike more on Nvidia because we allowed more Nvidia cards to be
> accelerated in 50, under bug 1284322, which got backed out in bug 1292311,
> and uplifted to 50 and 51.

The backout in bug 1292311 didn't improve the situation here.
Bug 1284322 landed in 49, so it can't be responsible for the increase in crash volume between 49 Beta and 50 Beta.

Comment 12

[Marco Castelluccio [:marco]]

That said, it appears that for some reason things are going back to 49.0b levels in 50.0b10 (e.g. the signature from bug 1308863 disappeared completely in 50.0b10).

The pushlog doesn't contain anything obviously related: https://hg.mozilla.org/releases/mozilla-beta/pushloghtml?fromchange=FIREFOX_50_0b9_RELEASE&tochange=FIREFOX_50_0b10_RELEASE.

Comment 13

[Edwin Flores [inactive from 2016-12-01] [:eflores] [:edwin]]

(In reply to Milan Sreckovic [:milan] from comment #10)
> It may spike more on Nvidia because we allowed more Nvidia cards to be
> accelerated in 50, under bug 1284322, which got backed out in bug 1292311,
> and uplifted to 50 and 51.

Bug 1292311 isn't enough to account for this. Most crashes were with relatively recent drivers and wouldn't have been affected by that blacklisting.

Comment 14

[[:philipp]]

all these nvidia related crashes are gone or reclining back to their pre-50 levels in 50.0b10 - this is probably due to the same underlying reason than we saw with bug 1308863 (see https://bugzilla.mozilla.org/show_bug.cgi?id=1308863#c19).

i'm going to look out if the crash pattern is gone in beta 11 as well and mark the bug as fixed if this is the case...

Comment 15

[[:philipp]]

still looking good in 50.0b11, so this was probably fixed by bug 1308418 too.

Comment 16

[Milan Sreckovic [:milan] (needinfo for best results)]

Not sure this is fixed, we're still getting ~300 crashes/week, but the spike is certainly gone.  If you look at different vendors, it's more like ~900 crashes/week.  See also bug 1207993.