1298221 - Consistent crashing with HW Acceleration + WindowBlinds

Status: RESOLVED DUPLICATE of bug 1544435

(Core :: Graphics, defect, P3)

Description

[lyralei]

User Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:48.0) Gecko/20100101 Firefox/48.0
Build ID: 20160823121617

Steps to reproduce:

I can reproduce this issue in several ways:

Preconditions:
* HW Acceleration must be enabled
* Firefox must not be in safe mode
* Addons being enabled or disabled does not appear to matter
* Version needs to be Firefox 48, as 47 doesn't have this problem

Scenario A - Just updated
1. Updated Firefox to 48 (I have done this twice, downgrading to 47 in between - both times crashed on initial start)
2. Opened Firefox
(Firefox crashed)

Scenario B - Facebook
1. Opened Firefox 48
2. Went to www.facebook.com
3. Attempted to open notifications pane
(Firefox crashed)

Scenario C - Youtube
1. Opened Firefox 48
2. Went to www.youtube.com/feed/subscriptions
3. Attempted to watch a video
(Firefox crashed)

Scenario D - Support
1. Opened Firefox 48
2. Went to about:support
(Firefox crashed)

I cannot reproduce these on Firefox 47, which I used before updating to 48 and temporarily downgraded to before upgrading back to 48 again.

My computer specs are as follows:

* i7-4700MQ
* 32GB RAM
* Windows 8.1 x64
* Nvidia 770M (Driver version 372.54)
* Intel HD GRaphics 4600 (Driver version 10.18.14.4264) - I have not yet updated this as Firefox uses the Nvidia GPU driver


Actual results:

Firefox crashed and I received a message from Windows stating my Nvidia GPU kernel driver had crashed and successfully recovered.

I updated my Nvidia GPU drivers with a clean install, restarted my laptop and tried all of the scenarios again. Firefox continued to crash in these situations.

I have a number of bug reports which I can link:

Initial crash after updating to Firefox 48, this occured straight away and has a slightly different signature from the other crashes: (wbload.dll@0x1ad04)

https://crash-stats.mozilla.com/report/index/f024634a-5a33-4cca-bff4-310b72160825

Trying to watch Youtube video: (wbload.dll@0x1f384)

https://crash-stats.mozilla.com/report/index/67f13df5-9d86-4286-9f22-b2a732160825

Trying to open notifications on Facebook after updating my Nvidia GPU drivers: (wbload.dll@0x1f384)

https://crash-stats.mozilla.com/report/index/9004e6f9-3eba-4025-9f66-6f0552160825

This one may have been when I tried to visit about:support (wbload.dll@0x1f384)

https://crash-stats.mozilla.com/report/index/c743b5a0-f3f3-4606-add4-b043f2160825

Also had these crashes with different signatures, but are probably related at least tangentially:

CDevice::CheckFormatSupportImp
https://crash-stats.mozilla.com/report/index/6c06c04f-3753-4be6-b4d0-8ecd72160825

d3d11.dll@0xc48f
https://crash-stats.mozilla.com/report/index/db9fdf12-feb7-4873-a5d0-d85352160825

Caspy7 in #firefox advised me to disable HWA which resolved the issue (unsurprisingly) and to submit this bug report.


Expected results:

Firefox shouldn't have crashed :-)

Comment 1

[lyralei]

I was asked to copy / paste the Graphics section of About:Support, here it is below:

Graphics
Features
Compositing	Basic
Asynchronous Pan/Zoom	none
WebGL Renderer	Google Inc. -- ANGLE (Intel(R) HD Graphics 4600 Direct3D9Ex vs_3_0 ps_3_0)
Hardware H264 Decoding	No; Hardware video decoding disabled or blacklisted
DirectWrite	false (6.3.9600.18123)
GPU #1
Active	Yes
Description	Intel(R) HD Graphics 4600
Vendor ID	0x8086
Device ID	0x0416
Driver Version	10.18.14.4264
Driver Date	8-4-2015
Drivers	igdumdim64 igd10iumd64 igd10iumd64 igdumdim32 igd10iumd32 igd10iumd32
Subsys ID	10e81462
RAM	Unknown
GPU #2
Active	No
Description	NVIDIA GeForce GTX 770M
Vendor ID	0x10de
Device ID	0x11e0
Driver Version	21.21.13.7254
Driver Date	8-11-2016
Drivers	nvd3dumx,nvwgf2umx,nvwgf2umx nvd3dum,nvwgf2um,nvwgf2um
Subsys ID	10e81462
RAM	3072
Diagnostics
AzureCanvasAccelerated	0
AzureCanvasBackend	skia
AzureContentBackend	cairo
AzureFallbackCanvasBackend	cairo

Comment 2

[lyralei]

Addendum to preconditions of experiencing the bug - Windowblinds needs to be enabled. It appears that the bug doesn't happen when Windowblinds is off.

Comment 3

[lyralei]

I just updated WindowBlinds (turned out my licence expired some time ago). Latest version of WindowBlinds 8 still has the issue.

I have also submitted a support ticket to WindowBlinds (#YGT-759-83776) and made them aware of this bug report.

Version details for WindowBlinds:

Windows 8.1 (9600.winblue_ltsb.160328-1315)
WindowBlinds 8.12 (037 - Windows 8 Edition) - 64 bit OS

    Wblind.dll     	2015/04/22 03:40:ll
    Wblind64.dll     	2015/04/22 03:40:47
    Wbsrv.dll     	2015/04/02 06:36:18
    WB8Config.exe     	2015/04/22 03:40:09
    Wbload.dll          	2015/04/15 07:31:02
    Screen.exe     	2015/04/02 06:36:08

Comment 4

[lyralei]

Using mozregression I traced this back to a specific commit:

Changeset e355cacefc881ba360d412853b57e8e060e966f4
Bug 1248496 - Enable D3D11 DXVA. r=ajones

Comment 5

[stoozor]

I have seen the same thing happen. However I cannot reproduce it at will like you do. I have already filed a bug report here : Bug 1298212

Firefox has crashed on 2 separate occasions when attempting to display a download dialogue after clicking a link to download something. It has crashed once after attempting to display the Inspector. I could successfully perform these same actions after Firefox restarted after the crash. The crash reports for these crashes are linked below.

Switching to an unloaded tab caused the video driver to crash and recover. I then proceeded to click on more unloaded tabs to see if the same thing would happen and it did.

At the time of those issues, my Firefox profile had been refreshed no longer than 11 days earlier. Video drivers are current. I run WindowBlinds on a dual monitor setup.

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:48.0) Gecko/20100101 Firefox/48.0
Build ID: 20160823121617

---
Crash reports
---
https://crash-stats.mozilla.com/report/index/bp-fbd0c763-c660-4df9-a471-f1c152160824
https://crash-stats.mozilla.com/report/index/bp-2fb24ade-fb13-4759-8084-ae5ec2160824
https://crash-stats.mozilla.com/report/index/bp-445daa65-be9d-4b0b-afbc-7111b2160825

---
Graphics
---
Features
Compositing	Basic
Asynchronous Pan/Zoom	none
WebGL Renderer	Google Inc. -- ANGLE (NVIDIA GeForce GTX 960 Direct3D9Ex vs_3_0 ps_3_0)
Hardware H264 Decoding	Yes; Using D3D9 API
Direct2D	true
DirectWrite	true (10.0.10240.16430)
GPU #1
Active	Yes
Description	NVIDIA GeForce GTX 960
Vendor ID	0x10de
Device ID	0x13d8
Driver Version	21.21.13.7254
Driver Date	8-11-2016
Drivers	nvd3dumx,nvwgf2umx,nvwgf2umx,nvwgf2umx nvd3dum,nvwgf2um,nvwgf2um,nvwgf2um
Subsys ID	00000000
RAM	3072
Diagnostics
AzureCanvasAccelerated	0
AzureCanvasBackend	direct2d 1.1
AzureContentBackend	direct2d 1.1
AzureFallbackCanvasBackend	cairo
failures	[GFX1-]: Detected rendering device reset on refreshGraphics
Features
Compositing	Basic
Asynchronous Pan/Zoom	none
WebGL Renderer	Google Inc. -- ANGLE (NVIDIA GeForce GTX 960 Direct3D9Ex vs_3_0 ps_3_0)
Hardware H264 Decoding	Yes; Using D3D9 API
Direct2D	true
DirectWrite	true (10.0.10240.16430)
GPU #1
Active	Yes
Description	NVIDIA GeForce GTX 960
Vendor ID	0x10de
Device ID	0x13d8
Driver Version	21.21.13.7254
Driver Date	8-11-2016
Drivers	nvd3dumx,nvwgf2umx,nvwgf2umx,nvwgf2umx nvd3dum,nvwgf2um,nvwgf2um,nvwgf2um
Subsys ID	00000000
RAM	3072
Diagnostics
AzureCanvasAccelerated	0
AzureCanvasBackend	direct2d 1.1
AzureContentBackend	direct2d 1.1
AzureFallbackCanvasBackend	cairo

Comment 6

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

(In reply to lyralei from comment #1)
> I was asked to copy / paste the Graphics section of About:Support, here it
> is below:

The report shows basic layers, which is not consistent with it being the changeset in c4. Is the about:support information from the same point in time that generates a crash?

Comment 7

[lyralei]

(In reply to Anthony Jones (:kentuckyfriedtakahe, :k17e) from comment #6)
> The report shows basic layers, which is not consistent with it being the
> changeset in c4. Is the about:support information from the same point in
> time that generates a crash?


When in the configuration that crashes, opening about:support always results in a crash. I went to the about:support page to get that information while in safe mode.

Now that I have several other workarounds for the issue, I can get the information outside of safe mode. Here's the same information from my Firefox with the following configuration:

* Firefox 48
* media.windows-media-foundation.allow-d3d11-dxva at default setting of true
* WindowBlinds set to not skin Firefox (with these settings it'd crash on about:support if it was skinning Firefox - I verified this was still the case just now)

I notice my Intel GPU drivers are fairly old, I should perhaps update those in case it helps.

Features
Compositing	Direct3D 11
Asynchronous Pan/Zoom	none
WebGL Renderer	Google Inc. -- ANGLE (Intel(R) HD Graphics 4600 Direct3D11 vs_5_0 ps_5_0)
Hardware H264 Decoding	Yes; Using D3D11 API
Direct2D	true
DirectWrite	true (6.3.9600.18123)
GPU #1
Active	Yes
Description	Intel(R) HD Graphics 4600
Vendor ID	0x8086
Device ID	0x0416
Driver Version	10.18.14.4264
Driver Date	8-4-2015
Drivers	igdumdim64 igd10iumd64 igd10iumd64 igdumdim32 igd10iumd32 igd10iumd32
Subsys ID	10e81462
RAM	Unknown
GPU #2
Active	No
Description	NVIDIA GeForce GTX 770M
Vendor ID	0x10de
Device ID	0x11e0
Driver Version	21.21.13.7254
Driver Date	8-11-2016
Drivers	nvd3dumx,nvwgf2umx,nvwgf2umx nvd3dum,nvwgf2um,nvwgf2um
Subsys ID	10e81462
RAM	3072
Diagnostics
AzureCanvasAccelerated	0
AzureCanvasBackend	direct2d 1.1
AzureContentBackend	direct2d 1.1
AzureFallbackCanvasBackend	cairo

Comment 8

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

Benjamin - looks like Window Blinds is causing our compositor to crash. It is crashing in the compositor which looks like it is being intercepted. It doesn't look (to my uneducated eyes) like D3D11 DXVA is the real issue.

Comment 9

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

See also http://www.stardock.com/products/windowblinds/history.asp

Comment 10

[Milan Sreckovic [:milan] (needinfo for best results)]

Not nvidia only; I added a few more crashes to the list.  I'll connect with WB.

Comment 11

[Milan Sreckovic [:milan] (needinfo for best results)]

Most are D3D11 DXVA, but some are D3D9 DXVA (e.g., https://crash-stats.mozilla.com/report/index/407736f1-ff40-41a0-8788-12d3d2160829), so DXVA is probably irrelevant.

Comment 12

[Milan Sreckovic [:milan] (needinfo for best results)]

lyralei - if you find firefox.exe, rename it to something else, and run, do you get the same crash?

Comment 13

[lyralei]

(In reply to Milan Sreckovic [:milan] from comment #12)
> lyralei - if you find firefox.exe, rename it to something else, and run, do
> you get the same crash?

I renamed it to burningcat.exe and did not experience the same crash.

Comment 14

[Milan Sreckovic [:milan] (needinfo for best results)]

That confirms WindowBlinds is doing something "special" for Firefox, throwing an exception and crashing.  I assume that switching back to default "Aero" would also have the crash go away.  I'm not convinced we know which versions are safe, at least not until we hear back from Startdock (I have a ticket open), so we may have to block all versions of wbload.dll and wblind.dll

Comment 15

[Milan Sreckovic [:milan] (needinfo for best results)]

A random sampling of the crashes has shown 10.0.0.{1,2,3}, 8.1.1.1, 8.3.0.0.  Comment 3 also mentions 8.1.2.0.  Bug 1149761 had us waiting for 8.1.1.0 to fix a previous problem, and comment 3 mentions "previous version" (to 8.1.2.0) that also had a problem.
It's possible 8.1.1 is OK and nothing else is.

Comment 16

[Milan Sreckovic [:milan] (needinfo for best results)]

Attachment: Bug 1298221: Block all versions of wbload and wblind.

Review commit: https://reviewboard.mozilla.org/r/75816/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/75816/

Comment 17

[Milan Sreckovic [:milan] (needinfo for best results)]

1. It is a bug in WindowBlinds.  They have a fix.
2. For (their) version 10.*, the fix will be out in the next beta, planned for the week of September 5th.
3. For earlier version 8.*, they're considering an update with the fix, but only later in September.
4. Stardock does not advise we try and block the DLL load.  They believe it will keep trying to load.
5. On versions 10.0.7 and higher, there is a way to exclude Firefox.exe from WindowBlinds, by an explicit user action.
6. Stardock is considering adding the "exclusion" functionality to the 8.* stream as well.

Apparently this bug in WindowBlinds tickled with video, games, going to about:support, etc.

It sounds like this is not actionable on our part, outside of putting some kind of an advisory to the community, telling them to update to the latest version, 10.0.7+ and exclude Firefox.  Liz, can you help us figure out what to do?

Comment 18

[Liz Henry (:lizzard) (relman/hg->git project)]

Release Note Request (optional, but appreciated)
[Why is this notable]:
[Suggested wording]:
[Links (documentation, blog post, etc)]:

We could write up an article on SUMO or a blog post, and link to it from release notes as a known issue.
 
Joni, do we describe specific issues like this in SUMO when it isn't in response to a user question? The crash troubleshooting page advises users to check release notes, in general. https://support.mozilla.org/en-US/kb/firefox-keeps-crashing-startup#w_uninstall-software-known-to-cause-crashes   
But a release note isn't going to be long enough to explain the details as Milan does above.

Comment 19

[Milan Sreckovic [:milan] (needinfo for best results)]

I also have a more detailed explanation from Stardock, but I'm not sure how public those are, and if we can/should publish it verbatim.  Probably not, I guess.

Comment 20

[lyralei]

(In reply to Milan Sreckovic [:milan] from comment #17)
> 3. For earlier version 8.*, they're considering an update with the fix, but
> only later in September.
> 5. On versions 10.0.7 and higher, there is a way to exclude Firefox.exe from
> WindowBlinds, by an explicit user action.
> 6. Stardock is considering adding the "exclusion" functionality to the 8.* stream as well.


In WindowBlinds version 8.* there is already functionality to exclude Firefox.exe from WindowBlinds via explicit user action. It is one of the current workarounds for the crashing that I am aware of (the others being to disable HWA in Firefox or set media.windows-media-foundation.allow-d3d11-dxva to false in about:config)

Comment 21

[Joni Chan]

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #18)
> Release Note Request (optional, but appreciated)
> [Why is this notable]:
> [Suggested wording]:
> [Links (documentation, blog post, etc)]:
> 
> We could write up an article on SUMO or a blog post, and link to it from
> release notes as a known issue.
>  
> Joni, do we describe specific issues like this in SUMO when it isn't in
> response to a user question? The crash troubleshooting page advises users to
> check release notes, in general.
> https://support.mozilla.org/en-US/kb/firefox-keeps-crashing-
> startup#w_uninstall-software-known-to-cause-crashes   
> But a release note isn't going to be long enough to explain the details as
> Milan does above.

Hi Liz, for specific bugs, we usually write a canned response that our community members can use in the forums or in social, but if it needs to be linked from somewhere else (release notes, tweets, etc.), we'd write a SUMO article.

Can someone help me with a first draft? I'm not sure that I understand the issue well enough to write the article. Here's a working doc: https://docs.google.com/document/d/14Q-QmCzxzTXxQlWzTE6dD0RQuEyyZmI0O6oRyoX_1xE/edit?usp=sharing

Comment 22

[Milan Sreckovic [:milan] (needinfo for best results)]

I wrote down the two options we know of in the document.

Comment 23

[Joni Chan]

We've put up the article here: https://support.mozilla.org/kb/hardware-acceleration-and-windowblinds-crash

If there's a specific article we can link to that talks about disabling WindowBlinds, please let us know. I'm on a Mac so I can't try it.

Comment 24

[lyralei]

(In reply to Joni Savage ("need info" me) from comment #23)
> We've put up the article here:
> https://support.mozilla.org/kb/hardware-acceleration-and-windowblinds-crash
> 
> If there's a specific article we can link to that talks about disabling
> WindowBlinds, please let us know. I'm on a Mac so I can't try it.


Here's the (rather short) page on the WindowBlinds KB on how to add an exclusion for a specific program: (excluding Firefox from WindowBlinds skinning prevents the crashing)

http://esupport.stardock.com/index.php?/default_import/Knowledgebase/Article/View/59/124/how-can-i-temporarily-exclude-a-program-from-being-skinned

Comment 25

[Liz Henry (:lizzard) (relman/hg->git project)]

Looks good Joni! Thanks. I used your basic format for another KB page on a specific crash.

Comment 26

[Sylvestre Ledru [:Sylvestre]]

Small volume, not sure we want to relnote it.

Comment 27

[Chris Peterson [:cpeterson]]

We're still seeing a few dozen wbload.dll crashes in each Firefox release with these DLL versions:

8.1.1.1
8.3.0.0
10.0.0.3
10.5.9.0

Comment 28

[Firefox Bug Husbandry Bot]

https://wiki.mozilla.org/Bug_Triage/Projects/Bug_Handling/Bug_Husbandry#Move_fix-optionals