1239188 - Timeout in D3D lock mutex timeout, followed by a crash

Status: RESOLVED WORKSFORME

(Core :: Graphics: Layers, defect, P3)

Description

[Milan Sreckovic [:milan] (needinfo for best results)]

Crash notes for this crash: https://crash-stats.mozilla.com/report/index/7905b94a-c177-4f6a-8c6b-5a3db2160112 point to a failed timeout with mutex->AcquireSync(0, 10000) in LockD3DTexture in TextureD3D11.cpp.  Coming from ImageHost::Lock.

Comment 1

[Milan Sreckovic [:milan] (needinfo for best results)]

Start with Jeff :)  Expected that this will sometimes take more than 10 seconds?  Anything to worry about?

Comment 2

[Jeff Muizelaar [:jrmuizel]]

This code should probably check for device lost and do something sane if that happened instead of crashing.

Comment 3

[Robert Kaiser]

Crashes like this come out as mozilla::layers::LockD3DTexture<T> now, see bp-32ef8ed3-9013-44ac-8833-72eb02160211 - a search like this can find all those within the last week (only that one right now): https://crash-stats.mozilla.com/search/?product=Firefox&proto_signature=~CrashStatsLogForwarder%3A%3ACrashAction&signature=%3Dmozilla%3A%3Alayers%3A%3ALockD3DTexture%3CT%3E&_facets=signature#facet-signature

Comment 4

[6lobe]

(In reply to Robert Kaiser (:[e-mail], on hiatus until May the 4th) from comment #3)
> Crashes like this come out as mozilla::layers::LockD3DTexture<T> now, see
> bp-32ef8ed3-9013-44ac-8833-72eb02160211 - a search like this can find all
> those within the last week (only that one right now):
> https://crash-stats.mozilla.com/search/
> ?product=Firefox&proto_signature=~CrashStatsLogForwarder%3A%3ACrashAction&sig
> nature=%3Dmozilla%3A%3Alayers%3A%3ALockD3DTexture%3CT%3E&_facets=signature#fa
> cet-signature

I am the person experiencing this crash in that report. This is still an issue.

This is caused by certain Flash videos crashing the AMD display driver.

Here is an example: https://www.twitch.tv/matumbaman/v/55944544

Viewing the above video in "Source" quality crashes the AMD display driver and subsequently causes Firefox to crash.

This is a long standing issue and I have reported this to Adobe and AMD way back but they havent fixed anything.

The AMD display driver does automatically recover after crashing but Firefox cannot recover from it and subsequently crashes.

Comment 5

[Milan Sreckovic [:milan] (needinfo for best results)]

That's.. Windows 8.1, vendor 0x1002 (AMD), device 0x6798, driver 15.300.1025.1001, subsys id 32141682? What's the actual computer you're using?	

Anthony - see comment 4 - let's add that to a sanity test for BB.

Comment 6

[6lobe]

(In reply to Milan Sreckovic [:milan] from comment #5)
> That's.. Windows 8.1, vendor 0x1002 (AMD), device 0x6798, driver
> 15.300.1025.1001, subsys id 32141682? What's the actual computer you're
> using?	
> 
> Anthony - see comment 4 - let's add that to a sanity test for BB.

Thats the computer I am experiencing this crash with.

I think its an issue with NPAPI Flash and AMD drivers since PPAPI Flash doesnt crash the display driver with the same videos. I have seen reports by other people experiencing the same crash with AMD hardware. Its a long standing issue and has been reported to AMD and Adobe but evidently they havent been interested in fixing this issue.

I know it isnt strictly related to this bug but I thought I'd chime in with the exact reason behind that instance of this Firefox crash.

Comment 7

[u279076]

(In reply to Milan Sreckovic [:milan] from comment #5)
> Anthony - see comment 4 - let's add that to a sanity test for BB.

Okay, I am on it.

Comment 8

[Milan Sreckovic [:milan] (needinfo for best results)]

Attachment: Bug 1239188: Get the information if the sync is ever abandoned, rather than just timed out.

Review commit: https://reviewboard.mozilla.org/r/60532/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/60532/

Comment 9

[Milan Sreckovic [:milan] (needinfo for best results)]

Is this https://crash-stats.mozilla.com/report/index/17afd195-b2b2-498b-810d-c62cb2160620 suggesting that we're timing out here: https://dxr.mozilla.org/mozilla-central/source/gfx/layers/d3d11/CompositorD3D11.cpp#1196 because we're holding the lock here: https://dxr.mozilla.org/mozilla-central/source/gfx/layers/d3d11/TextureD3D11.cpp#185

Comment 10

[Jeff Muizelaar [:jrmuizel]]

Comment on attachment 8764967 [details]
Bug 1239188: Get the information if the sync is  ever abandoned, rather than just timed out.

https://reviewboard.mozilla.org/r/60532/#review57444

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/b777b3e5f962
Get the information if the sync is  ever abandoned, rather than just timed out. r=jrmuizel

Comment 12

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/b777b3e5f962

Comment 13

[Milan Sreckovic [:milan] (needinfo for best results)]

We do hit the "abandoned" - https://crash-stats.mozilla.com/report/index/83a53226-83fc-44d4-8665-28a622160629

Comment 14

[u279076]

These crashes go back as far as Firefox 39. The crash rate has dropped by a factor of 10 over the last 6 months (~50/day => ~5/day) beginning on March 25th.

Comment 15

[Milan Sreckovic [:milan] (needinfo for best results)]

See also bug 1292923 comment 9 for an example of an "abandoned" message and a crash elsewhere.

Comment 16

[Milan Sreckovic [:milan] (needinfo for best results)]

After sitting in single digits for a while, this went back up in volume mid September, before we switched trains, and really spiked in late October.  There is a matching increase of telemetry GFX_CRASH (value 2) which is this crash but in beta/release.  Which means that it was in 50, as it went from aurora to beta.

Anthony, any more information we can extract out of the crash frequency and reports to get us to narrow down when the numbers when back up?

Comment 17

[u279076]

> [@ CrashStatsLogForwarder::CrashAction]
There are no reports of this signature in the last week.

> [@ mozilla::layers::LockD3DTexture<T>]
There are 133 reports of this signature in the last week.

There are a few notable spikes but the first spike clearly occurs on September 14, 2016 when the average crash rate increased from ~4/day to ~18/day. There aren't very strong correlations to a specific set of hardware/drivers although it does seem to affect AMD/NVIDIA users disproportionately to Intel users (see correlations below).

Based on build IDs this implicates the following changeset:
http://hg.mozilla.org/releases/mozilla-aurora/pushloghtml?fromchange=696a981b6d534a7015d6f2ba3d2379af620e9b26&tochange=7852ac7e74faf5a637189437ce0a9a7f39e6292c

Version correlations point to Firefox 50/51:
> September 7-13:
> ===============
> Firefox 48.0.*: 19%
> Firefox 49.0b*:  3%
> Firefox 50.0a2: 16%
> Firefox 51.0a1: 16%
> 
> September 14-20:
> ================
> Firefox 48.0.*:  8%
> Firefox 49.0b*:  1%
> Firefox 50.0a2: 49%
> Firefox 51.0a1: 23%

Aurora correlations seem to point to 'GFX_ERROR "[D3D11] " = true' as a potential indicator:
> September 7-13:
> ===============
> (80.00% in signature vs 01.11% overall) GFX_ERROR "(gfxWindowsPlatform) Detected device reset: " = true
> (75.00% in signature vs 01.04% overall) GFX_ERROR "(gfxWindowsPlatform) Finished device reset." = true
> (70.00% in signature vs 00.12% overall) GFX_ERROR "GFX: D3D11 lock mutex abandoned" = true
> (55.00% in signature vs 00.90% overall) GFX_ERROR "(nsWindow) Finished device reset." = true
> (55.00% in signature vs 00.92% overall) GFX_ERROR "(nsWindow) Detected device reset: " = true
> (45.00% in signature vs 00.48% overall) GFX_ERROR "GFX: D3D11 skip BeginFrame with device-removed." = true
> 
> September 14-20:
> ================
> (80.00% in signature vs 01.11% overall) GFX_ERROR "(gfxWindowsPlatform) Detected device reset: " = true
> (75.00% in signature vs 01.04% overall) GFX_ERROR "(gfxWindowsPlatform) Finished device reset." = true
> (70.00% in signature vs 00.12% overall) GFX_ERROR "GFX: D3D11 lock mutex abandoned" = true
> (55.00% in signature vs 00.90% overall) GFX_ERROR "(nsWindow) Finished device reset." = true
> (55.00% in signature vs 00.92% overall) GFX_ERROR "(nsWindow) Detected device reset: " = true
> (45.00% in signature vs 00.48% overall) GFX_ERROR "GFX: D3D11 skip BeginFrame with device-removed." = true
> (35.00% in signature vs 00.96% overall) GFX_ERROR "[D3D11] " = true
> (35.00% in signature vs 00.96% overall) GFX_ERROR "[D3D11] " = true

Aurora correlations for adapter_vendor:
> September 7-13:
> ===============
>  0% AMD in signature vs 20% overall
> 80% INTEL in signature vs 54% overall
> 20% NVIDIA in signature vs 23% overall
>
> September 14-20:
> ================
> 13% AMD in signature vs 18% overall
> 30% INTEL in signature vs 53% overall
> 46% NVIDIA in signature vs 26% overall

Aurora correlations for adapter_device:
> September 7-13:
> ===============
> 60% Intel Haswell GPU
> 20% Intel Broadwell GPU
> 20% NVIDIA Fermi GPU
>
> September 14-20:
> ================
> AMD
> ------------------------
> 14.80% AMD Evergreen GPU
>  3.70% AMD Kabini GPU
>  3.70% AMD R700 GPU
>  1.85% AMD Cayman GPU
>
> Intel
> ------------------------
> 9.26% Intel Ironlake GPU
> 5.55% Intel Ivybridge GPU
> 3.70% Intel Haswell GPU
> 3.70% Intel GMA3500 GPU
> 3.70% Intel GMA4500HD GPU
> 1.85% Intel Broadwell GPU
> 1.85% Intel Skylake GPU
>
> NVIDIA
> ------------------------
> 12.95% NVIDIA Kepler GPU
> 12.95% NVIDIA Fermi GPU
> 12.95% NVIDIA Tesla GPU
>  7.40% NVIDIA Maxwell GPU

App Notes
> DXVA2D3D9?/+ in app_notes
>> 22% after September 14 vs 0% prior to September 14

Platforms
> 65% Windows 7 after September 14 vs 75% prior
> 33% Windows 10 after September 14 vs 19% prior
>  2% Windows 8 after September 14 vs 6% prior

Aurora Build IDs:
> 20160916004005:  8 crashes
> 20160915004005: 11 crashes
> 20160914004005: 14 crashes
> 20160913004005:  3 crashes
> 20160912004004:  1 crashes
> 20160911004011:  2 crashes
> 20160910004004:  1 crashes

Comment 18

[Nicholas Nethercote [inactive]]

Over the past 7 days in Nightly there have been 172 occurrences of this crash signature across 155 installations.

Comment 19

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:bas.schouten, maybe it's time to close this bug?

Comment 20

[Bas Schouten (:bas.schouten)]

There's still issues like this, but with different signatures now, they have bugs, so let's close this one.