1460499 - Use DirectComposition for hardware decoded video on Windows

Status: RESOLVED FIXED

(Core :: Graphics: WebRender, enhancement, P3)

Description

[Sotaro Ikeda [:sotaro]]

We want a way to directly composite video frames if possible with DirectComposition.

Comment 1

[Sotaro Ikeda [:sotaro]]

Need to use Direct3D 11 Video APIs
  https://msdn.microsoft.com/en-us/library/windows/desktop/hh447677(v=vs.85).aspx

Comment 2

[Jeff Muizelaar [:jrmuizel]]

We don't currently have any evidence we need to do this for the MVP

Comment 3

[Sotaro Ikeda [:sotaro]]

Chromium uses "Direct3D 11 Video Interfaces" for handling video overlays. To use it, texture data need to be the following format. Then we could not use software decoded videos for video overlays for now. Software decoded video normally uses YUV420(YV12).

• DXGI_FORMAT_NV12
• DXGI_FORMAT_P010
• DXGI_FORMAT_P016

https://cs.chromium.org/chromium/src/ui/gl/swap_chain_presenter.cc?l=942

Comment 4

[Sotaro Ikeda [:sotaro]]

Attachment: temporal patch - Test video rendering to dc layer with D3D11VideoProcessor

The patch does video rendering to DCompositionVisual when video is decoded to DXGITextureHostD3D11 with NV12 format. It was confirmed by visiting http://www.html5videoplayer.net/html5video/mp4-h-264-video-test/. When a video was playback, a video was also rendered to dc layer.

• We could add software decoded video support. But it needs to be NV12 format. And we do not know yet about performance.
• Video rendering with IDXGIDecodeSwapChain seems to provide better performance than D3D11VideoProcessor . But it seems that we need to modify decoding side code for using it.

Comment 5

[Sotaro Ikeda [:sotaro]]

Attachment: temporal patch - Test video rendering to dc layer with D3D11VideoProcessor

Rebased.

Comment 6

[Sotaro Ikeda [:sotaro]]

Attachment: temporal patch - Test video rendering to dc layer with D3D11VideoProcessor and skip WR rendering

Comment 7

[Sotaro Ikeda [:sotaro]]

I roughly a video playback performance with Attachment 9090712 [details] [diff]). The following youtube video was used for h.264 video playback. Majority of recent youtube videos does not use h.264.
https://www.youtube.com/watch?v=L5WFSSFuufI

With the patch, gpu and cpu usages were lower. gpu usage was lower than "Direct3D 11 (Advanced Layers)".

Compositing: WebRender

• cpu 20%
• gpu 60%

Compositing: WebRender with video overlay(Attachment 9090712 [details] [diff])

• cpu 5%-10%
• gpu 7%

Compositing: Direct3D 11 (Advanced Layers)

• cpu: 5%
• gpu: 16%-17%

Comment 8

[Sotaro Ikeda [:sotaro]]

Attachment: wip - Video rendering to dc layer with D3D11VideoProcessor

Comment 9

[Sotaro Ikeda [:sotaro]]

Attachment: wip - Video rendering to dc layer with D3D11VideoProcessor

Rebased.

Comment 10

[Sotaro Ikeda [:sotaro]]

Attachment: wip - Video rendering to dc layer with D3D11VideoProcessor

Comment 11

[Sotaro Ikeda [:sotaro]]

Attachment: wip - Video rendering to dc layer with D3D11VideoProcessor

Comment 12

[Sotaro Ikeda [:sotaro]]

Attachment: wip - Video rendering to dc layer with D3D11VideoProcessor

Rebased.

Comment 13

[Sotaro Ikeda [:sotaro]]

Attachment: wip - Video rendering to dc layer with D3D11VideoProcessor

Comment 14

[Sotaro Ikeda [:sotaro]]

Attachment: wip - Video rendering to dc layer with D3D11VideoProcessor

Comment 15

[Sotaro Ikeda [:sotaro]]

Attachment: wip - Video rendering to dc layer with D3D11VideoProcessor

Comment 16

[Sotaro Ikeda [:sotaro]]

Attachment: wip - Video rendering to dc layer with D3D11VideoProcessor

Comment 17

[Sotaro Ikeda [:sotaro]]

Attachment 9097505 [details] [diff] set video dc layer under "default swap chain dc layer" and the default swap chain renders a transparent hole for the video dc layer. It is because video frame could have a ui element over the video dc layer.

From a discussion, we do not chooser the above option for video dc layer for now. Instead, video frame and UI elements above video is split to different dc layers. This does not need a transparent hole. There could be a risk that it could increase a number of dc layers.

Comment 18

[Markus Stange [:mstange]]

For my own benefit, I'm going to describe what I think this patch does:
It creates a DC visual for each video. These visuals are ordered below the visual for the main swap chain. In WebRender's rendering (which goes into the main swap chain), video images are replaced with ClearRect items. Video visuals get their content as follows: Their content is a SEQUENTIAL_FLIP swap chain that has been created for the visual. It uses an RGB color space. Video is converted from NV12 and copied into the RGB swap chain using ID3D11VideoContext::VideoProcessorBlt, with the input being the ID3D11Texture2D from the video frame's RenderDXGITextureHostOGL, and the output being the ID3D11Texture2D back buffer of the swap chain. The patch hints that IDXGIDecodeSwapChain might be a better solution but doesn't use it.

Comment 19

[Sotaro Ikeda [:sotaro]]

Bug 1595994 might affect to this bug.

Comment 20

[Sotaro Ikeda [:sotaro]]

Attachment: Bug 1460499 - wip patch

Comment 21

[Sotaro Ikeda [:sotaro]]

Attachment: Bug 1460499 - Use DirectComposition for hardware decoded video on Windows

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9c88255c845c
Use DirectComposition for hardware decoded video on Windows r=nical

Comment 23

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/9c88255c845c

Comment 24

[Jeff Muizelaar [:jrmuizel]]

Sotaro, is it correct that this never got enabled? If so why?

Comment 25

[Jeff Muizelaar [:jrmuizel]]

FWIW, I tried out this patch on a Intel HD 530 and overall usage seemed lower. One thing that was interesting is that it causes us to spend time in the "Video Processing" block of Task Manager. Neither, Chrome or Firefox normally spend anytime there.

Comment 26

[Sotaro Ikeda [:sotaro]]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #24)

Sotaro, is it correct that this never got enabled? If so why?

It was because I saw frame drop during STR of Bug 1667500. I noticed it just before trying to enable it. Current implementation uses RGB SwapChain, it seemed not provide good performance. We might need to use YUV SwapChain for performance.

When I created the current implementation, I did not have a Win PC that supported NV12 SwapChain. NV12 SwapChain support was not common. The my Win PCs only supported YUY2 SwapChain. Gecko did not have code to handle YUY2 video. Then I stopped to think about YUV SwapChain before.

I have 2 new Win PCs since then. Both supports NV12 SwapChain. NV12 SwapChain support seemed to become more common. It is better to look into NV12 SwapChain support for video now.

Comment 27

[Sotaro Ikeda [:sotaro]]

Created Bug 1722447 for YUV SwapChain support for hardware decoded video.

Comment 28

[Sotaro Ikeda [:sotaro]]

(In reply to Jeff Muizelaar [:jrmuizel] from comment #25)

FWIW, I tried out this patch on a Intel HD 530 and overall usage seemed lower. One thing that was interesting is that it causes us to spend time in the "Video Processing" block of Task Manager. Neither, Chrome or Firefox normally spend anytime there.

Current implementation does not support YUV SwapChain. It might affect to performance.

Comment 29

[Sotaro Ikeda [:sotaro]]

:jrmuizel, can you provide chrome://gpu of chrome browser?

Comment 30

[Jeff Muizelaar [:jrmuizel]]

chrome://gpu does not show support for overlays on this machine, I believe because of blocked directcomposition. Running Chrome with --disable-gpu-driver-bug-workarounds shows even better performance and does seem to use overlays.

Comment 31

[Jeff Muizelaar [:jrmuizel]]

Or if not overlays at least a separate swapchain for video.