1143806 - crash in mozilla::layers::SyncObjectD3D11::FinalizeFrame()

Status: RESOLVED FIXED

(Core :: Graphics, defect)

Description

[Kartikaya Gupta (email:[e-mail])]

+++ This bug was initially created as a clone of Bug #1131370 +++

The patches in bug 1131370 reduced the crash volume down to 25% of the original volume, but there is still something that needs tracking down and fixing. See bug 1131370 comment 17 and bug 1131370 comment 18 for some potential next-steps.

Comment 1

[Andreas Jung]

I got this crash after windows reset my graphics driver.

Comment 2

[Lorenso Gwine]

I get this bug when I use a discrete graphics card for Nightly.
Notebook: Asus K551L, 
Adapter Description: Intel(R) HD Graphics Family, 
Adapter Description (GPU #2): NVIDIA GeForce 840M.
Adapter Drivers: igdumdim64 igd10iumd64 igd10iumd64 igdumdim32 igd10iumd32 igd10iumd32.
Adapter Drivers (GPU #2): nvd3dumx,nvwgf2umx,nvwgf2umx nvd3dum,nvwgf2um,nvwgf2um.

Comment 3

[Tyler Downer [He/Him]]

[Tracking Requested - why for this release]:

[Tracking Requested - why for this release]:

I have a user with this crash at https://support.mozilla.org/en-US/questions/1056636 who is able to reproduce this fairly regularly. Is this something we can possibly track for a 38 fix?

Comment 4

[Nicolas Silva [:nical]]

There's a lot of 0x8899000c error codes (D2DERR_RECREATE_TARGET) reported with this in crash-stats. MSDN doc says "A presentation error has occurred that may be recoverable. The caller needs to re-create the render target then attempt to render the frame again.". Perhaps another "device lost"-style scenario that we are not handling yet?

Comment 5

[Milan Sreckovic [:milan] (needinfo for best results)]

gfxSurfaceDrawable::DrawWithSamplingRect protects against calling gfxSurfaceDrawable::DrawInternal with null mSourceSurface.  gfxSurfaceDrawable::Draw does not, so you can get through and that's likely the crash?

Comment 6

[Milan Sreckovic [:milan] (needinfo for best results)]

The previous comment was actually meant for bug 1154003, but perhaps it's useful here as well :)

Comment 7

[Liz Henry (:lizzard) (relman/hg->git project)]

Tracking for 38 since this looks like a bad crash.

Comment 8

[Lawrence Mandel [:lmandel] (use needinfo)]

I'm tracking across the board because of the volume. This isn't a commitment to ship the fix in a 37 point release but I would like it on the list of potential fixes to take if a point release is required.

Comment 9

[Milan Sreckovic [:milan] (needinfo for best results)]

(In reply to Lorenso Gwine from comment #2)
> I get this bug when I use a discrete graphics card for Nightly.
> Notebook: Asus K551L, 
> Adapter Description: Intel(R) HD Graphics Family, 
> Adapter Description (GPU #2): NVIDIA GeForce 840M.
> Adapter Drivers: igdumdim64 igd10iumd64 igd10iumd64 igdumdim32 igd10iumd32
> igd10iumd32.
> Adapter Drivers (GPU #2): nvd3dumx,nvwgf2umx,nvwgf2umx
> nvd3dum,nvwgf2um,nvwgf2um.

Lorens, what driver versions have you got?

Comment 10

[Lorenso Gwine]

(In reply to Milan Sreckovic [:milan] from comment #9)
> Lorens, what driver versions have you got?
Nvidia Driver: 9.18.13.5012
Intel HD: 10.18.10.3379

Nightly crashes in e10s mode with NVidia, with intel HD works normally.

Comment 11

[Milan Sreckovic [:milan] (needinfo for best results)]

With older drivers, Nvidia wasn't allowing Firefox to run on the discrete graphics, always forcing us onto integrated.  Even when the global settings were to use the discrete.  Somewhere along the way (so far, appears to be between 327.68 and 341.01), they started allowing Firefox to run on discrete.  That combined with us using different things is conspiring to cause these problems.  See bug 1145143 as well; the signatures are different, but that's just a question of timing.

I was hoping that a 350.* driver fixes these crashes (TDR related, most likely), as at least the reporter of bug 1145143 had the problem go away, but in this case we have the user with 350.12, and the problem remains.

Comment 12

[Sylvestre Ledru [:Sylvestre]]

This is #11 in 38. Milan, is there anything we could do here? blacklisting?

Comment 13

[Milan Sreckovic [:milan] (needinfo for best results)]

We may be able to find out when the discrete graphics is being used and blacklist all those scenarios, which will send those people into unaccelerated situation.  A bit drastic, but we don't have the driver range to use.  Old enough drivers are actually OK, because in those Nvidia forced Firefox to integrated graphics, but the newer drivers will lead to not using GPU at all.

Let me see if we can get a quick patch for this.

Comment 14

[Bas Schouten (:bas.schouten)]

It's still unlikely that this is a result of people force running acceleration though. That should be fairly rare, there must be some other situation somehow where this still occurs.

Comment 15

[Milan Sreckovic [:milan] (needinfo for best results)]

I think people force discrete graphics usage, not as a Firefox preference, but as a global system setting. With the older Nvidia drivers, that made no difference for Firefox, we always ended up on Intel; with the more recent drivers, we now end up on Nvidia, and I think that has us run into trouble.  I have some notes (back in TO) as to which older drivers were "safe".

Comment 16

[Lorenso Gwine]

(In reply to Milan Sreckovic [:milan] from comment #15)
> I think people force discrete graphics usage, not as a Firefox preference,
> but as a global system setting.
In my case I put Nvidia especially for Firefox.

Comment 17

[Milan Sreckovic [:milan] (needinfo for best results)]

Right - I more meant that it isn't a Firefox preference you set, it is at the system/driver level.  With old Nvidia drivers, you couldn't even set the Firefox preference, it would just show the integrated graphics and not let you make any changes.

Comment 18

[Bas Schouten (:bas.schouten)]

Attachment: Tolerate timeouts occurring if the device driver is in the process of resetting

Analysis of the crash reports indicates the timeout is likely to occur on a device driver reset. When this occurs it's probably best to just continue.

Comment 19

[Jeff Muizelaar [:jrmuizel]]

Comment on attachment 8596746 [details] [diff] [review]
Tolerate timeouts occurring if the device driver is in the process of resetting

Review of attachment 8596746 [details] [diff] [review]:
-----------------------------------------------------------------

Add a gfxCriticalError when this happens.

Comment 20

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/962f0378c4c8

Comment 21

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/962f0378c4c8

Comment 22

[Sylvestre Ledru [:Sylvestre]]

Jeff, Bas, is it possible to have an uplift request for 38 & 39? Thanks

Comment 23

[Bas Schouten (:bas.schouten)]

(In reply to Sylvestre Ledru [:sylvestre] from comment #22)
> Jeff, Bas, is it possible to have an uplift request for 38 & 39? Thanks

I'm not entirely certain that's useful for 38? I think 38 always crashes when a driver crash occurs anyway, right?

Comment 24

[Sylvestre Ledru [:Sylvestre]]

(In reply to Bas Schouten (:bas.schouten) from comment #23)
> (In reply to Sylvestre Ledru [:sylvestre] from comment #22)
> > Jeff, Bas, is it possible to have an uplift request for 38 & 39? Thanks
> 
> I'm not entirely certain that's useful for 38? I think 38 always crashes
> when a driver crash occurs anyway, right?

It might be unrelated but we landed a change for this:
https://bugzilla.mozilla.org/show_bug.cgi?id=1154003#c12
Does it change anything?

Comment 25

[Bas Schouten (:bas.schouten)]

(In reply to Sylvestre Ledru [:sylvestre] from comment #24)
> (In reply to Bas Schouten (:bas.schouten) from comment #23)
> > (In reply to Sylvestre Ledru [:sylvestre] from comment #22)
> > > Jeff, Bas, is it possible to have an uplift request for 38 & 39? Thanks
> > 
> > I'm not entirely certain that's useful for 38? I think 38 always crashes
> > when a driver crash occurs anyway, right?
> 
> It might be unrelated but we landed a change for this:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1154003#c12
> Does it change anything?

I don't really think so, I think there's other issues. 38 didn't really have any sort of systemic approach for surviving driver resets. Since HTML5 video wasn't as wide spread yet at the time we were unaware of the large increase in driver resets (to an unacceptable level) it would (apparently) cause.

Comment 26

[Lorenso Gwine]

Still crashes

Comment 27

[Milan Sreckovic [:milan] (needinfo for best results)]

Lorenso, do you have a pointer to a crash you experienced?

Comment 28

[GMA]

https://communities.intel.com/thread/63760

Lorenso, can you try updating to the latest Intel 10.18.14.4170 driver?

Comment 29

[Lorenso Gwine]

(In reply to GMA from comment #28)
> Lorenso, can you try updating to the latest Intel 10.18.14.4170 driver?

Done.
Driver Version	10.18.14.4170
Driver Version (GPU #2)	9.18.13.5012
In e10s mode works with Intel HD, still dont works with Nvidia.
In normal mode all fine.

Comment 30

[Lorenso Gwine]

(In reply to Milan Sreckovic [:milan] from comment #27)
> Lorenso, do you have a pointer to a crash you experienced?

https://crash-stats.mozilla.com/report/index/0518dfb3-bd13-4c08-8462-02c7f2150430

Do you mean this?

Comment 31

[Milan Sreckovic [:milan] (needinfo for best results)]

I'll spin off another bug for this - the crash above is not reported as following a device reset.

Comment 32

[Milan Sreckovic [:milan] (needinfo for best results)]

Continuing conversation in bug 1160157.

Comment 33

[Liz Henry (:lizzard) (relman/hg->git project)]

We could still uplift this to 39 in early beta.  Milan, what do you think?

Comment 34

[Milan Sreckovic [:milan] (needinfo for best results)]

I'm going to ask for the uplift, but can somebody examine the crashstats and tell us if the numbers went down after this fix on 40?

Comment 35

[Milan Sreckovic [:milan] (needinfo for best results)]

Comment on attachment 8596746 [details] [diff] [review]
Tolerate timeouts occurring if the device driver is in the process of resetting

Approval Request Comment
[Feature/regressing bug #]:
[User impact if declined]: There should be fewer TDR related crashes with this change.
[Describe test coverage new/current, TreeHerder]:
[Risks and why]: 
[String/UUID change made/needed]:

Comment 36

[Liz Henry (:lizzard) (relman/hg->git project)]

Kairo can you have a look at the stats for this? Something that got checked in after build 2015042913 appears to have helped the general crash rate, but I'm not sure if that has anything to do with 40.

Comment 37

[Robert Kaiser]

(In reply to Liz Henry (:lizzard) from comment #36)
> Kairo can you have a look at the stats for this?

IMHO, Nightly is just too noisy to make any clear conclusions for this. That said, from what we can see through the noise, it doesn't look like there was any really huge change in the volume of this signature on Nightly in the last 3 months. That said, we might have more intermittent builds without those crashes since this landed on 4/24. We probably only will really know if it has a positive effect when we land on Beta.

Comment 38

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 8596746 [details] [diff] [review]
Tolerate timeouts occurring if the device driver is in the process of resetting

Approved for uplift to beta (39). This is a somewhat speculative fix to see if it brings down the rate of driver reset crashes.

Comment 39

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/3bb36b3ef64e

Comment 40

[alexios]

1160157 
1131370 

have you determined that are related to nvidia card and drivers?
for my gt240 nvidia hasn't released a new driver since February.

is it going to be fixed in the next release?

Comment 41

[alexios]

version 39 now and it crashed my graphic driver.

Comment 42

[alexios]

and it still crashes the driver

Comment 43

[Milan Sreckovic [:milan] (needinfo for best results)]

alexios, do you have a pointer to one of your recent crashes (from about:crashes)?  Can you attach it here?

Comment 44

[alexios]

(In reply to Milan Sreckovic [:milan] from comment #43)
> alexios, do you have a pointer to one of your recent crashes (from
> about:crashes)?  Can you attach it here?

all my crashes are these, most of them are related to the browser crashing the nvidia driver when watching cam/chatrooms on the website cam4.com :D

bp-7e389358-488c-429a-a692-8bbd42150701
bp-315f5206-4c48-4012-872a-3ae742150630
bp-2070b320-5e07-4610-bf76-69b552150628
bp-6cfee853-cdd3-4f4d-8f4e-493db2150623
bp-c364130d-5439-44e2-89e5-942412150622
bp-3cc20261-425a-432f-8bdd-1687d2150621

Comment 45

[Wayne Mery (:wsmwk)]

(In reply to Liz Henry (:lizzard) from comment #38)
> Comment on attachment 8596746 [details] [diff] [review]
> Tolerate timeouts occurring if the device driver is in the process of
> resetting
> 
> Approved for uplift to beta (39). This is a somewhat speculative fix to see
> if it brings down the rate of driver reset crashes.

bug 1187764 comment 6 suggests this patch should be very helpful on ESR for Thunderbird, where D3D11 is our #4 crash

And Firefox statistics seem to suggest it too should benefit from uplifting this patch to ESR
D3D11 is #2 crash https://crash-stats.mozilla.com/topcrasher/products/Firefox/versions/38.0.1
D3D11 is #4 crash https://crash-stats.mozilla.com/topcrasher/products/Firefox/versions/39.0
D3D11 is #9 crash https://crash-stats.mozilla.com/topcrasher/products/Firefox/versions/40.0b8

Requestiong uplift to ESR. I'm not in a position to comment on the risks

Comment 46

[Wayne Mery (:wsmwk)]

Comment on attachment 8596746 [details] [diff] [review]
Tolerate timeouts occurring if the device driver is in the process of resetting

[Approval Request Comment]
If this is not a sec:{high,crit} bug, please state case for ESR consideration:
User impact if declined: Thunderbird topcrash continues
Fix Landed on Version: Firefox 39, Firefox 40 (still in beta), Thunderbird 40 beta
Risk to taking this patch (and alternatives if risky): ??
String or UUID changes made by this patch: none

See https://wiki.mozilla.org/Release_Management/ESR_Landing_Process for more info.

Comment 47

[Kate Glazko]

Tracking for ESR 38 because fix exists and ESR is still affected through crashes.

Comment 48

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Comment on attachment 8596746 [details] [diff] [review]
Tolerate timeouts occurring if the device driver is in the process of resetting

Approved. This patch seems simple and has been in FF39 and FF40 for a while. Seems safe to uplift to ESR38.

Comment 49

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr38/rev/5cf905794365

Comment 50

[Jerry Shih[:jerry] (UTC+8) (inactive)]

Attachment: check device-removed status when we have timeout. v1