1149761 - Users reporting crash in gfxWindowsPlatform::InitD3D11Devices() after updating to Firefox 37

Status: RESOLVED FIXED

(Core :: Graphics, defect)

Description

[[:philipp]]

This bug was filed from the Socorro interface and is 
report bp-5c60f961-b644-4ad2-a32c-9810d2150331.
=============================================================

this seems to be a new startup crash and a regression in firefox 37 - numerous comments mention that it happened immediately after updating.
as per feedback of this sumo user the crash happens when you attempt to run firefox in safe mode as well: https://support.mozilla.org/en-US/questions/1054989

Comment 1

[Robert Kaiser]

[Tracking Requested - why for this release]:
I'm nominating this just so it shows up on the radar of release management, in case this ends up being a larger issue.

Comment 2

[[:philipp]]

a helpful user has narrowed it down to this regression range: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=8f6965ca&tochange=a4b4cd74

Comment 3

[Milan Sreckovic [:milan] (needinfo for best results)]

Lawrence, we may want to throttle while we're figuring this one out?

Comment 4

[Milan Sreckovic [:milan] (needinfo for best results)]

Bas, this is "we failed to initialize D3D11 WARP device" where we call MOZ_CRASH.

Comment 5

[Jeff Muizelaar [:jrmuizel]]

Attachment: An emergency solution to avoid crashing

Comment 6

[Jeff Muizelaar [:jrmuizel]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=280ee4065540

Comment 7

[Milan Sreckovic [:milan] (needinfo for best results)]

What's the "rank" of this crash?  How common is it?

Comment 8

[[:philipp]]

sorry about clearing the need info flags before (this may have been a bug in the BMO experimental modular interface). re-adding...

Comment 9

[Bas Schouten (:bas.schouten)]

This is almost certainly caused by the stardock malware intercepting Windows DLLs and messing with the calls. I wonder if this is fixed by blacklisting the wblind and wbload DLLs. But for now we should take Jeff's patch.

Comment 10

[Robert Kaiser]

(In reply to Milan Sreckovic [:milan] from comment #7)
> What's the "rank" of this crash?  How common is it?

Winthin the firs 24-48 hours, we do not have much in terms of ranks or anything, but early 37.0 release data suggests this is the #3 crash right now, with #1 being another gfx startup issue in bug 1137716, which actually was supposed to be fixed in 37 but still appears at the top with 12% of all crashes.

Comment 11

[Robert Kaiser]

See https://crash-stats.mozilla.com/search/?product=Firefox&version=37.0&release_channel=release&process_type=browser&process_type=content&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature for a current search on 37.0 release crashes.

Comment 12

[Jeff Muizelaar [:jrmuizel]]

I installed Stardock window blinds on my machine and 37 causes the whole machine to hang and require a reboot. It would be interesting to see what happens for other people when window blinds is installed.

Comment 13

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/58b1ce7f0629

Comment 14

[Jeff Muizelaar [:jrmuizel]]

Comment on attachment 8586405 [details] [diff] [review]
An emergency solution to avoid crashing

Approval Request Comment
[Feature/regressing bug #]: WARP (37)
[User impact if declined]: Startup crashes
[Risks and why]: We should not MOZ_CRASH on startup, cause that's bad.

Comment 15

[Jeff Muizelaar [:jrmuizel]]

I tried a build that should've included this fix on the machine that can reproduce it and it didn't seem to help. I wasn't able to get crash symbols so I'm not sure where the crash moved to.

Comment 16

[Milan Sreckovic [:milan] (needinfo for best results)]

The "infected" machine runs 36 fine?

Comment 17

[Jeff Muizelaar [:jrmuizel]]

(In reply to Milan Sreckovic [:milan] from comment #16)
> The "infected" machine runs 36 fine?

I believe so yes. My desktop (the machine that hangs) certainly ran 36 fine, I'm not as sure about that laptop that reproduces the issue.

Comment 18

[Milan Sreckovic [:milan] (needinfo for best results)]

What about nightly?  I installed this stuff on my desktop, nightly seems to start fine.

Comment 19

[Milan Sreckovic [:milan] (needinfo for best results)]

I take that last comment back; managed to get it to crash.

Comment 20

[Milan Sreckovic [:milan] (needinfo for best results)]

Possible workaround for the people that are stuck with this - perhaps post in sumo - if they do have windowblinds, and they switch back to Windows Aero theme, the crash should go away.  Well, at least it does for me.  Fails with others, but not with Aero.

Comment 21

[Milan Sreckovic [:milan] (needinfo for best results)]

Comment on attachment 8586405 [details] [diff] [review]
An emergency solution to avoid crashing

Review of attachment 8586405 [details] [diff] [review]:
-----------------------------------------------------------------

With this change, I don't crash on startup.

::: gfx/thebes/gfxWindowsPlatform.cpp
@@ +1972,5 @@
>  
>      if (FAILED(hr)) {
>        // This should always succeed... in theory.
>        gfxCriticalError() << "Failed to initialize WARP D3D11 device!" << hr;
> +    } else {

I would just replace MOZ_CRASH with return instead (in the original code.)  I'm not sure it's safe to continue.

Comment 22

[Milan Sreckovic [:milan] (needinfo for best results)]

(In reply to Milan Sreckovic [:milan] from comment #21)
> 
> With this change, I don't crash on startup.

I'm not sure I like where I end up, there are some artifacts (close, minimize, maximize buttons don't show up), but it doesn't crash.  I don't know how much of the problem is due to windowblinds changing the "substyle".

Comment 23

[Jim Morrison]

Since the regression is about a Windows specific bug, why the automatic update for Firefox/Thunderbird is also disabled for Linux & Mac?

Comment 24

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #12)
> I installed Stardock window blinds on my machine and 37 causes the whole
> machine to hang and require a reboot. It would be interesting to see what
> happens for other people when window blinds is installed.

I was able to easily reproduce the startup crash with Firefox 37.0 and Firefox 40 Nightly, on Windows 7 x64, Windows 8 x64 and Windows 8.1 x64, using the following scenario:
1. Download and install Windowblinds (http://www.stardock.com/products/windowblinds/).
2. Start Window Blinds and apply a custom style (e.g. Lantana).
3. Attempt to start Firefox.

Results: Firefox crashes before even showing the Profile Manager window (if set to show it). I tried this on two separate machines, and never did Windows freeze... I just got the crash.

(In reply to Milan Sreckovic [:milan] from comment #20)
> Possible workaround for the people that are stuck with this - perhaps post
> in sumo - if they do have windowblinds, and they switch back to Windows Aero
> theme, the crash should go away.  Well, at least it does for me.  Fails with
> others, but not with Aero.

Switching to Aero seems to indeed work around this issue on Windows 7, but doesn't seem to work on Windows 8 or 8.1 (I had to uninstall WindowBlinds here to get Firefox to start again).

Comment 25

[Jeff Muizelaar [:jrmuizel]]

Here's what's going wrong in the Window Blinds case.

Window Blinds hooks D3D11CreateDevice, the hook (in wbload.dll) then calls GetModuleHandle("FIREFOX.EXE") and if this returns a valid handle, Window Blinds immediately returns failure from D3D11CreateDevice and we proceed to MOZ_CRASH.

Comment 26

[Milan Sreckovic [:milan] (needinfo for best results)]

Florin, if you rename firefox.exe to something else, does the crash go away?

Comment 27

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

(In reply to Milan Sreckovic [:milan] from comment #26)
> Florin, if you rename firefox.exe to something else, does the crash go away?

Yes, renaming to something else like test.exe will start Firefox without crashing.

Comment 28

[Milan Sreckovic [:milan] (needinfo for best results)]

(In reply to Florin Mezei, QA (:FlorinMezei) from comment #27)
> (In reply to Milan Sreckovic [:milan] from comment #26)
> > Florin, if you rename firefox.exe to something else, does the crash go away?
> 
> Yes, renaming to something else like test.exe will start Firefox without
> crashing.

Thanks.  Not a good workaround in general, there is other code (not written by us, things like driver workarounds and such) that depend on this name, but good to know.  :lmandel is reaching out to Stardock.

Comment 29

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

Note that while Firefox starts without crashing, it does have some visual issues with different theme applied (missing titlebar) - see https://bugzilla.mozilla.org/attachment.cgi?id=8587404. Google Chrome has the same visual issues if that matters in any way.

Comment 30

[Milan Sreckovic [:milan] (needinfo for best results)]

:kairo, can you check if we have a 100% correlation between user having WindowBlinds (wbload.dll exists?) and crashes?  Or is there a better way to test if WindowBlinds is installed?

Comment 31

[Milan Sreckovic [:milan] (needinfo for best results)]

Bas will land the modified patch (comment 21), with implicit r+ from Jeff (over the phone) as inbound opens.

Comment 32

[Jeff Muizelaar [:jrmuizel]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/76739f7bfac7

Comment 33

[Jeff Muizelaar [:jrmuizel]]

Attachment: Don't MOZ_CRASH if WARP fails

Approval Request Comment
[Feature/regressing bug #]: WARP (36)
[User impact if declined]: Many crashes
[Describe test coverage new/current, TreeHerder]: None yet
[Risks and why]: should just fall back to the path we use on Vista/XP

Comment 34

[Jim Morrison]

Well, people at Mozilla is very busy fixing this problem, so for anyone finding an answer here, https://support.mozilla.org/en-US/questions/1055150 says OK to perform a manual update under Mac. I see machines using Ubuntu updating Firefox without problems to the latest version, so I will assume that the same is true for other Linux variants using the tarball. Many thanks to all the Mozillians for their great work.

Comment 35

[Milan Sreckovic [:milan] (needinfo for best results)]

Attachment: Aurora/beta patch if the original doesn't apply properly.

If the patch that's asking for the approval doesn't apply because of the previous one, here's a rollup.

Comment 36

[Lawrence Mandel [:lmandel] (use needinfo)]

Comment on attachment 8587558 [details] [diff] [review]
Don't MOZ_CRASH if WARP fails

I reviewed this fix with Milan and Jeff. This should solve all known cases of the crash even though not all cases are definitely known. There is some risk that the fix is not complete but without any additional information our best course of action is pushing an update with this fix. We'll take this in 37.0.1 Release+ Beta+ Aurora+

Comment 37

[Robert Kaiser]

(In reply to Milan Sreckovic [:milan] from comment #30)
> :kairo, can you check if we have a 100% correlation between user having
> WindowBlinds (wbload.dll exists?) and crashes?  Or is there a better way to
> test if WindowBlinds is installed?

https://crash-stats.mozilla.com/report/list?product=Firefox&range_value=7&range_unit=days&date=2015-04-02&signature=gfxWindowsPlatform%3A%3AInitD3D11Devices%28%29&version=Firefox%3A37.0#tab-correlations says there there is not a 100% correlation, it's merely a 45% one - but 45% vs. 0% is still very strong.

Comment 38

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/f1567b49630a

Comment 39

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/cf2036567077

Comment 40

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/6d12f36cb8fd

Comment 41

[Chris]

2 questions.

1 - there i snothing in the ff37 changes about changes to hardware acceleration so what was ommited from the changes page that is causing this issue? as clearly you guys changed something sinc eff36.
2 - why havent you pulled the ff37 security page? you have disclosed security bugs which end users cannot patch now due to the update been pulled.

Comment 42

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/76739f7bfac7

Comment 43

[Lawrence Mandel [:lmandel] (use needinfo)]

I tried to encompass the 3 start-up crash bugs that we fixed in 37.0.1 in a single release note: "Fixed: Start-up crash due to graphics hardware and third party software"

Comment 44

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

Verified today that Firefox 37.0.1 (BuildID=20150402191859) starts without crashing, both for a normal start and in safe mode (bug 1149864), on the same Windows 7 x64 and Windows 8 x64 machines where I reproduced the original issue with Firefox 37.0.

The visual issue mentioned in comment 29 (missing titlebar) is already tracked separately in bug 1077245.

The fix doesn't seem to have made the latest Nightly or Aurora yet.

Comment 45

[Lawrence Mandel [:lmandel] (use needinfo)]

I received a response from Stardock that indicated that there was an issue with WindowBlinds and the DirectX11 path in Firefox 37. They have fixed the issue in WindowBlinds 8.11 beta.

Comment 46

[[SV Manager] Florin Mezei, QA (:FlorinMezei)]

Also verified as fixed on Firefox 38 Beta 5 on Windows 7 x64 and Windows 8.1 x64.