1109718 - Figure out why and when RenderTargetView's need to be recreated

Status: RESOLVED INACTIVE

(Core :: Graphics: CanvasWebGL, defect)

Description

[Vladan Djeric (:vladan)]

Google Maps only blinks when I try to zoom the map or get directions. It's likely broken by the patch that turned on D3D11 on WebGL.
The 3D cube on http://get.webgl.org/ shakes, but doesn't spin.

Tested on Nightly 20141210030207, Windows 7 x64, Optimus Quadro 1000M + Intel HD 3000

Comment 1

[Vladan Djeric (:vladan)]

*Tested on a non-e10s build

Comment 2

[Jeff Muizelaar [:jrmuizel]]

This works for me on a Intel HD 4000

Comment 3

[Vladan Djeric (:vladan)]

Graphics
--------
Adapter Description: Intel(R) HD Graphics 3000
Adapter Description (GPU #2): NVIDIA Quadro 1000M
Adapter Drivers: igdumd64 igd10umd64 igd10umd64 igdumd32 igd10umd32 igd10umd32
Adapter Drivers (GPU #2): nvd3dumx,nvwgf2umx,nvwgf2umx nvd3dum,nvwgf2um,nvwgf2um
Adapter RAM: Unknown
Adapter RAM (GPU #2): 2048
Device ID: 0x0126
Device ID (GPU #2): 0x0dfa
Direct2D Enabled: true
DirectWrite Enabled: true (6.2.9200.16571)
Driver Date: 3-20-2014
Driver Date (GPU #2): 9-12-2014
Driver Version: 9.17.10.3517
Driver Version (GPU #2): 9.18.13.4084
GPU #2 Active: false
GPU Accelerated Windows: 1/1 Direct3D 11 (OMTC)
Subsys ID: 21d117aa
Subsys ID (GPU #2): 21d117aa
Vendor ID: 0x8086
Vendor ID (GPU #2): 0x10de
WebGL Renderer: Google Inc. -- ANGLE (NVIDIA Quadro 1000M Direct3D11 vs_5_0 ps_5_0)
windowLayerManagerRemote: true
AzureCanvasBackend: direct2d 1.1
AzureContentBackend: direct2d 1.1
AzureFallbackCanvasBackend: cairo
AzureSkiaAccelerated: 0

Comment 4

[Jeff Muizelaar [:jrmuizel]]

This worked also worked on Sotaro's Nvidia machine.

Comment 5

[Ahmad Saleem]

Shaking here and no spin with nvidia 344.75 driver on Windows 8.1 Pro 64-bit with FX 37 Nightly 64-bit (e10s enabled):

Graphics:
Adapter Description: NVIDIA GeForce GTX 460
Adapter Drivers: nvd3dumx,nvwgf2umx,nvwgf2umx nvd3dum,nvwgf2um,nvwgf2um
Adapter RAM: 1024
ClearType Parameters: Gamma: 2200 Pixel Structure: R ClearType Level: 0 Enhanced Contrast: 100
Device ID: 0x0e22
Direct2D Enabled: true
DirectWrite Enabled: true (6.3.9600.17415)
Driver Date: 11-12-2014
Driver Version: 9.18.13.4475
GPU #2 Active: false
GPU Accelerated Windows: 1/1 Direct3D 11 (OMTC)
Subsys ID: 00000000
Vendor ID: 0x10de
WebGL Renderer: Google Inc. -- ANGLE (NVIDIA GeForce GTX 460 Direct3D11 vs_5_0 ps_5_0)
windowLayerManagerRemote: true
AzureCanvasBackend: direct2d 1.1
AzureContentBackend: direct2d 1.1
AzureFallbackCanvasBackend: cairo
AzureSkiaAccelerated: 0

Comment 6

[Jeff Muizelaar [:jrmuizel]]

(In reply to Zlip792 from comment #5)
> Shaking here and no spin with nvidia 344.75 driver on Windows 8.1 Pro 64-bit
> with FX 37 Nightly 64-bit (e10s enabled):

Do you have a dual GPU machine?

Comment 7

[Ahmad Saleem]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #6)
> (In reply to Zlip792 from comment #5)
> > Shaking here and no spin with nvidia 344.75 driver on Windows 8.1 Pro 64-bit
> > with FX 37 Nightly 64-bit (e10s enabled):
> 
> Do you have a dual GPU machine?

No desktop with single GPU.

Comment 8

[Jeff Muizelaar [:jrmuizel]]

I had Vladan try a version of the build that didn't use KeyedMutex's for synchronization and that fixed the problem for him. At this point it looks like an Nvidia driver bug with keyed mutexes.

Comment 9

[Jeff Muizelaar [:jrmuizel]]

If you run http://people.mozilla.org/~jmuizelaar/canvas2webgl/canvas2webgl.html we end up repeating the first 4 frames in a loop

Comment 10

[Ahmad Saleem]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #9)
> If you run
> http://people.mozilla.org/~jmuizelaar/canvas2webgl/canvas2webgl.html we end
> up repeating the first 4 frames in a loop

Left one does not align and "0" keeps on blinking rather than changing.

Comment 11

[Jeff Muizelaar [:jrmuizel]]

Zlip792, does the build here fix the problem?

http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-6b4f2f8c1d89/try-win64/

Comment 12

[Ahmad Saleem]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #11)
> Zlip792, does the build here fix the problem?
> 
> http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/jmuizelaar@mozilla.
> com-6b4f2f8c1d89/try-win64/

Confirmed, this try build fixed both Comment 1 and Comment 9 Testcases.

Comment 1 Testcase:
- No Shake and spinning like Chromium 41.

Comment 9 Testcase:
- Aligned and in sync with right side.

Feel free to ask about anything else.

Comment 13

[Jeff Muizelaar [:jrmuizel]]

I didn't have much luck debugging this. My current theory is that there might be something wrong with this pattern:

producer1->release();

producer2->acquire()
 consumer1->acquire()
 consumer1->release();
producer2->release();

producer1->acquire();
 consumer2->acquire();
 consumer2->release();


vs

producer1->release();

 consumer1->acquire()
 consumer1->release();
producer2->acquire()
producer2->release();

 consumer2->acquire();
 consumer2->release();
producer1->acquire();

Comment 14

[Jeff Muizelaar [:jrmuizel]]

I ran a test where I dumped the contents of the frames on the client side. This showed the same results as we were seeing in the compositor. This suggests that we're not able to successfully write to the framebuffers after a while and just keep cycling through the old content.

Comment 15

[Jeff Muizelaar [:jrmuizel]]

So I've narrowed down the problem some. It looks like when we AcquireSync on a surface for a second time we can no longer write to that surface. I don't yet know why. The other thing I've discovered is that ReleaseSync seems to clear the RenderTarget from the device context. This was surprising to me.

Comment 16

[Jeff Muizelaar [:jrmuizel]]

So it looks the problem is as follows:
ANGLE creates a RenderTargetView during texture creation. On most drivers we seem to be able to reuse this across AcquireSync/ReleaseSync operations, however on machines with this problem it seems like we can not.

The solution seems to be to recreate our RenderTargetView everytime we AcquireSync. How to convince ANGLE to do this is unclear to me at the moment.

Comment 17

[Elbart]

I can reproduce this on an IntelHD-Machine, so it's not nVidia-only.
The try-build of comment 11 works for me, too.

Comment 18

[Jeff Muizelaar [:jrmuizel]]

I've disabled KeyedMutexes on trunk until we can fix this issue properly.

Comment 19

[Jeff Muizelaar [:jrmuizel]]

 Zlip792 and Elbart can you post your d3d11 dll version?

Comment 20

[Jeff Muizelaar [:jrmuizel]]

Vladan has version 6.2.9200.16570

Comment 21

[Elbart]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #20)
> Vladan has version 6.2.9200.16570
Same for me, on both machines.

Comment 22

[Jeff Muizelaar [:jrmuizel]]

(In reply to Elbart from comment #21)
> (In reply to Jeff Muizelaar [:jrmuizel] from comment #20)
> > Vladan has version 6.2.9200.16570
> Same for me, on both machines.

Are you able to reproduce on both machines?

Comment 23

[Ahmad Saleem]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #19)
>  Zlip792 and Elbart can you post your d3d11 dll version?

Mine: 6.3.9600.17415

I'm now on Nvidia latest Beta driver.

Comment 24

[Elbart]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #22)
> (In reply to Elbart from comment #21)
> > (In reply to Jeff Muizelaar [:jrmuizel] from comment #20)
> > > Vladan has version 6.2.9200.16570
> > Same for me, on both machines.
> 
> Are you able to reproduce on both machines?

Yes, with affected Nightlies before today's.

Comment 25

[Jeff Muizelaar [:jrmuizel]]

Elbart and Zlip792 can you try running this program and report it's output and whether it crashes or not:

http://people.mozilla.org/~jmuizelaar/d3d11-tests/firefox.exe

(I've renamed it to firefox.exe because drivers sometimes change behaviour depending on the executable name)

Comment 26

[Ahmad Saleem]

@jrmuizel

This exe file is crashing as soon as I start it with "MS-DOS" like window and "firefox.exe has stopped working".

I tried compatibility mode as well, removed EMET as well if it could be the case. Restarted as well. Any idea how to run it? I am willing to help as much as you want, even willing to share profile or system through Remote connection.

Comment 27

[Jeff Muizelaar [:jrmuizel]]

(In reply to Zlip792 from comment #26)
> @jrmuizel
> 
> This exe file is crashing as soon as I start it with "MS-DOS" like window
> and "firefox.exe has stopped working".

This is somewhat expected. If you run it from a cmd window it should print something before crashing.

Comment 28

[Ahmad Saleem]

Do I need to look into Event Viewer for app crash reason?

Comment 29

[Jeff Muizelaar [:jrmuizel]]

(In reply to Zlip792 from comment #28)
> Do I need to look into Event Viewer for app crash reason?

No just run cmd.exe navigate to the path where you downloaded the executable and then run it by typing 'firefox'

Comment 30

[Ahmad Saleem]

It doesn't show any log etc, this does happen:

URL: http://i.imgur.com/py4APWb.png

Willing to help any way you want. Remote session?

Comment 31

[Jeff Muizelaar [:jrmuizel]]

(In reply to Zlip792 from comment #30)
> It doesn't show any log etc, this does happen:
> 
> URL: http://i.imgur.com/py4APWb.png
> 
> Willing to help any way you want. Remote session?

I've uploaded a new binary. This one should not crash.

Comment 32

[Ahmad Saleem]

Hashes - http://i.imgur.com/A8Jcg9w.png

It shows same result.

URL: http://i.imgur.com/TbuWuyI.png

Comment 33

[Jeff Muizelaar [:jrmuizel]]

(In reply to Zlip792 from comment #32)
> Hashes - http://i.imgur.com/A8Jcg9w.png
> 
> It shows same result.
> 
> URL: http://i.imgur.com/TbuWuyI.png

Try again. The previous builds were using the d3d debug layer which I expect you don't have. This was probably the cause of the crashes.

Comment 34

[Ahmad Saleem]

After trying again, this got printed:

URL: http://i.imgur.com/im9bA7Y.png

Ask me as much as you want for any help, I'm willing to help.

Comment 35

[Jeff Muizelaar [:jrmuizel]]

(In reply to Zlip792 from comment #34)
> After trying again, this got printed:
> 
> URL: http://i.imgur.com/im9bA7Y.png
> 
> Ask me as much as you want for any help, I'm willing to help.

Ok. That means what was broken on the machine that I have to test on is not broken for you. Which means you might have a different problem.

Just to confirm, does webgl work properly for you with the current nightly build?

Comment 36

[Ahmad Saleem]

Yes, with current Nightly build, WebGL works fine. Not witnessed any issue.

Comment 37

[Jeff Muizelaar [:jrmuizel]]

Attachment: Avoid recreating the rendertargetview to see what hardware is all affected.

I'm looking into this. I don't think it's as wide spread as I originally thought, so I'm intentionally breaking it on Nightly to see what reports of brokeness we get.

Comment 38

[Kelsey Gilbert [:jgilbert]]

Comment on attachment 8630177 [details] [diff] [review]
Avoid recreating the rendertargetview to see what hardware is all affected.

Review of attachment 8630177 [details] [diff] [review]:
-----------------------------------------------------------------

I don't think we should break users in the hopes that they'll tell us what hardware they're using.

We certainly should not purposefully break them and leave them stranded. Strongly consider a programatic solution (particularly since a machine in the office reproduces the issue), or at the very least include a pref to re-enable the current behavior.

Comment 39

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/139680a3393b

Comment 40

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/139680a3393b

Comment 41

[Jeff Muizelaar [:jrmuizel]]

Here's the super search that will find these:

https://crash-stats.mozilla.com/search/?graphics_critical_error=%3Drendertargetviewneedsrecreating&_facets=signature&_facets=build_id&_facets=graphics_critical_error&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports

Comment 42

[Jeff Muizelaar [:jrmuizel]]

And it looks like all of them have vendorid mismatch:
https://crash-stats.mozilla.com/search/?graphics_critical_error=%3Drendertargetviewneedsrecreating&build_id=%3E%3D20150716030208&_facets=signature&_facets=build_id&_facets=graphics_critical_error&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-graphics_critical_error

However, not all vendorid mismatches have the renderview recreation problem.