1724949 - Consider letting MediaFoundation manage our DirectComposition swap chain

Status: NEW

(Core :: Audio/Video: Playback, enhancement)

Description

[Jeff Muizelaar [:jrmuizel]]

Currently we have some support for using a video swapchain with DirectComposition. Like Chrome, we manage it ourselves. However, we may have the option of off loading that to MediaFoundation and having it take care of overlay detection/fallback etc. This is what https://github.com/microsoft/media-foundation/blob/a17adcfcee7c92c57eb2257aed1d4fa3fa99fa67/samples/MediaEngineDCompWin32Sample/src/ui/DirectCompositionLayer.cpp#L17 does.

Comment 1

[Jeff Muizelaar [:jrmuizel]]

I believe this might also let us avoid the copy mentioned in bug 1723207. Sotaro/Alastor, does it seem like this approach might work?

Comment 2

[Alastor Wu [:alwu]]

Sorry, I don't get the point of why off loading image swapping to WMF could avoid a copy. Is that because we copy videos into swap chain like what this comment said?

Comment 3

[Jeff Muizelaar [:jrmuizel]]

Because we can open the swap chain with a handle, I believe it may be possible for the swap chain to be associated with a different device and so we don't need to worry about any cross device synchronization because that is handled during composition by the DWM.

i.e. currently we have:

• get a shared texture
• copy from MF texture into shared texture
• blit from shared texture into a swap chain that we've configured attached to a DirectComposition visual

this proposal:

• ask MF to draw into a swap chain that it manages
• get a handle to that swap chain and create a DirectComposition visual for it

Another potential alternative might be for us to create the video swap chain using the media D3D11 device and then share that to avoid needing to do copy to a shared texture.

Comment 4

[Alastor Wu [:alwu]]

Although I don't really understand the details of how we handle images via a swap chain, your proposal sounds reasonable to me. In addition, I wonder where that swap chain exists? In GPU process? If the decoder is in the RDD process, then is ask MF to draw into a swap chain possible? I assume that the swap chain would exist in the side of the compositor, so we still need to copy texture into a shard texture in this case?

Comment 5

[Jeff Muizelaar [:jrmuizel]]

Elaborating on the flow of the sample app:

• The IMFMediaEngineEx gets put in DCOMP mode and we call GetVideoSwapchainHandle() to get a handle for the VideoSwapChain that it creates:
  https://github.com/microsoft/media-foundation/blob/969f38b9fff9892f5d75bc353c72d213da807739/samples/MediaEngineDCompWin32Sample/src/media/MediaEngineWrapper.cpp#L272

• Using the created HANDLE , IDCompositionDevice::CreateSurfaceFromHandle() is used to get a IDCompositionSurface. A visual is created and the the surface is set as the content of the visual:
  https://github.com/microsoft/media-foundation/blob/a17adcfcee7c92c57eb2257aed1d4fa3fa99fa67/samples/MediaEngineDCompWin32Sample/src/ui/DirectCompositionLayer.cpp#L17

However, it looks like we don't use IMFMediaEngine so perhaps the approach suggested in comment 0 is not an option.

Comment 6

[Sotaro Ikeda [:sotaro]]

It seems that Microsoft is going to add Media Foundation(MF) based Content Decryption Modules (CDMs) to chromium. chromium source already has several changes for it. It seems only to be used with MF based CDM. If video uses this framework, video rendering to canvas2d/WebGL and screen shot seems not work. a/v sync mechanism becoms different than another media playback.

https://bugs.chromium.org/p/chromium/issues/detail?id=999747
https://github.com/MicrosoftEdge/MSEdgeExplainers/blob/main/Media/MFCdm/explainer.md
https://docs.google.com/document/d/1kXMdmfsSKU3tcwTxG-wFU-MaEA6sN3by45PCGBV8zJU/edit#heading=h.7nki9mck5t64

Comment 7

[Sotaro Ikeda [:sotaro]]

chromium already seems to support no-video copy video playback. When "Supports sharing the decoded NV12 textures" is enabled, video textures are not copied

• support_share_nv12_textures_

Number of output buffers seems to be set in DXVAVideoDecodeAccelerator::CheckDecoderDxvaSupport()

When DecodeSwapChain is used, it seems that D3D11 resource sharing needs to be disabled. Then decoding might needs to happen in GPU process.

• DXVAVideoDecodeAccelerator::SetDecoderInputMediaType()

chromium uses ANGLE device when "sharing the decoded NV12 textures" is enabled.

• DXVAVideoDecodeAccelerator::ShouldUseANGLEDevice()
• DXVAVideoDecodeAccelerator::GetPictureBufferMechanism()
• DXVAPictureBuffer::Create()

This no-video copy mechanism looks more similar to current gecko media framework. Though I am not sure how difficult to implement it to gecko media.

This seems to related to bug 1723207.