1893427 - (hw-ffvpx) [meta] using hw accelerated video decoding through ffvpx

Status: NEW

(Core :: Audio/Video: Playback, task, P2)

Description

[Paul Adenot (:padenot)]

See https://bugzilla.mozilla.org/show_bug.cgi?id=1890622#c23.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:jimm, could you have a look please?

For more information, please visit BugBot documentation.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:jimm, could you have a look please?

For more information, please visit BugBot documentation.

Comment 3

[Sotaro Ikeda [:sotaro]]

Information for adding D3D11 Fence handling See Bug 1820370 comment51.

Comment 4

[Sotaro Ikeda [:sotaro]]

Because of the followings, this bug should have high priority, I think. Since ffmpeg also uses D3D video API directly like chromium.

• [1] Firefox uses MFT API for hardware video decoding. But chromium does not use MFT API for hardware video decoding at all.

  • Firefox use IMFTransform like MFTDecoder::Create()
  • Chromium uses D3D video API directly like D3D11VideoDecoderWrapperImpl

• [2] MFT API does not support D3D11 Fence handling. Chromium adds D3D11 Fence handling. See Bug 1820370 comment51.

• [3] MFT API does not support D3D12 handling.

[1] seemed to cause several problems. By [1], MFT API seems not be well tested compared to direct D3D video API usage. And chromium's bug reporting could not be used to detect hardware video decoding bugs in Firefox. Firefox's hardware video decoding bugs were reported occasionally. Reporting frequency seems to be increasing recently like Bug 1888354, Bug 1887942. In future, the problem might be increased more. And there are several bugs that increase video memory usage during YouTube video playback. It might also be related to high level MFT API usage.

And when the problem happens, we do not have enough ways to workaround the problem because of high level API. For example, Bug 1896823 blocks reusing decoder device on Intel gen12 GPUs recently. It disables "zero video frame copy"on Intel gen12 GPUs and it reduces video decoding performance on them.

It is also possible that multiple GPUs are experiencing decoding problems, just not reported by users.

[2] affects to WebGPU's video handling. Though WebGPU does not implement importing video frame for now. WebGPU uses D3D12 backend on Winodws. It requests to provide D3D Fence for waiting GPU rendering complete asynchronously.

For video rendering to WebRender, when hardware video decoding and WebRender uses same D3D11 Device, it does not need synchronization by gecko. But when hardware video decoding and WebRender uses different D3D11 Device, current synchronization mechanism use "SyncObject->Synchronize()" like D3D11DXVA2Manager::CopyToImage() and RenderCompositorANGLE::BeginFrame().

Keyed mutex does not work well with hardware video decoding with MFT API. Then it is not used for hardware video decoding(Bug 1340398 and Bug 1257013). In stead, "SyncObject->Synchronize()" is used in current gecko.

This synchronization mechanism forces WebRender to wait too long. "SyncObject->Synchronize()" in D3D11DXVA2Manager::CopyToImage() forces WebRender to wait longer than necessary. And just calling "SyncObject->Synchronize()" in RenderCompositorANGLE::BeginFrame() slow downs WebRender rendering.

Bug 1847665 and Bug 1638819 made that RenderCompositorANGLE::BeginFrame() calls "SyncObject->Synchronize()" only when necessary. But hardware video decoding with different D3D11 Device forces to call "SyncObject->Synchronize()".

And current WebGL video frame rendering does not wait "video decoding complete" correctly, since D3D11 Fence is not supported for hardware decoded video.

For [3], chromium is preparing for D3D12 backend for video decoding when compositing happens with D3D12 for future. MFT API does not support D3D12 handling.
https://issues.chromium.org/issues/40233230

Comment 5

[Alastor Wu [:alwu]]

(In reply to Sotaro Ikeda [:sotaro] from comment #4)

Because of the followings, this bug should have high priority, I think. Since ffmpeg also uses D3D video API directly like chromium.

I'm aiming back to this bug once I finish the issues for MFCDM. The reason of setting this as P2 is I won't have enough bandwidth to finish this on current version. I haven't had enough time to check Bug 1820370 comment51 yet, but if we use ffmpeg, we still won't use d3d11 video api directly, we would use ffmpeg API like this.

In addition, now I'm still in the early stage of investigation, so my understanding might not be correct. If we use ffmpeg for d3d hardware decoding, what I imagine is that we would need to create a d3d texture pool (or reuse existing gfx one) and provide that to ffmpeg, then we would receive the decoded video stored in the given texture, and wrap it to an shared image sending to the compositor. From this perspective, this flow would be similar with what we've been doing via the MFT, and the difference is the framework controlling d3d api. If so, is using D3D11 Fence still possible?

In addition, we should probably verify whether the issue we encountered when using MFT would not happen if we switch to ffmpeg. The worst situation is that the problems still exist due to driver's issue even if using ffmpeg.

Comment 6

[Sotaro Ikeda [:sotaro]]

(In reply to Alastor Wu [:alwu] from comment #5)

(In reply to Sotaro Ikeda [:sotaro] from comment #4)

In addition, now I'm still in the early stage of investigation, so my understanding might not be correct. If we use ffmpeg for d3d hardware decoding, what I imagine is that we would need to create a d3d texture pool (or reuse existing gfx one) and provide that to ffmpeg, then we would receive the decoded video stored in the given texture, and wrap it to an shared image sending to the compositor. From this perspective, this flow would be similar with what we've been doing via the MFT, and the difference is the framework controlling d3d api. If so, is using D3D11 Fence still possible?

MFT does not expose low level API. Then with MFT, we do not know exact timing like chromium is doing. If we implement D3D11 Fence, we might wait longer than chromium.

In addition, we should probably verify whether the issue we encountered when using MFT would not happen if we switch to ffmpeg. The worst situation is that the problems still exist due to driver's issue even if using ffmpeg.

In this case, the problem can be minimize the problem to the same level as chromium and could share the problem with chromium.

It seems that risks of using MFT seems like increasing recently, I think. It seems better that gecko switch to using same level of API like chromium and ffmpeg in near future.

Comment 7

[k4sum1]

Any progress regarding this? I would be interested in an alternative ffmpeg player for videos including h.264 and such. I was actually thinking of implementing such a thing in my fork, but I'm not sure where to begin. However if you (upstream) do it, that would be very nice.

Comment 8

[Alastor Wu [:alwu]]

(In reply to k4sum1 from comment #7)

Any progress regarding this? I would be interested in an alternative ffmpeg player for videos including h.264 and such. I was actually thinking of implementing such a thing in my fork, but I'm not sure where to begin. However if you (upstream) do it, that would be very nice.

Sorry I'm still side tracks by other stuffs, so haven't started a real progress on it. If you're interested in helping this issue, bug 1660336 and bug 1833354 would be good references.

Comment 9

[Alastor Wu [:alwu]]

Quick update, I'm back to this issue. Will update my implementation plan when I finish investigation.

Comment 10

[Alastor Wu [:alwu]]

Attachment: Bug 1893427 - part1 : add d3d11va2 vp9 decode path from ffmpeg 7.0 to bundled ffvpx on Windows.

Comment 11

[Alastor Wu [:alwu]]

Attachment: WIP: Bug 1893427 - part2 : add d3d11va2 av1 decode path from ffmpeg 7.0 to bundled ffvpx on Windows.

Depends on D226067

Comment 12

[Alastor Wu [:alwu]]

Attachment: WIP: Bug 1893427 - part3 : check hw codec support for ffmpeg decoder module.

Comment 13

[Alastor Wu [:alwu]]

I will make this bug as a meta bug, and file related bugs to block this bug.

Comment 14

[Alastor Wu [:alwu]]

Attachment: WIP: Bug 1893427 - part4 : use ffvpx linker on the gpu process in order to support hw decoding.

Depends on D226084

Comment 15

[Alastor Wu [:alwu]]

Attachment: WIP: Bug 1893427 - part5 : adjust definitions for the preparation of using hw decoding on Windows.

As we will enable hw decoding on Windows, we should mark those GTK only
codes explicitly.

Depends on D227113

Comment 16

[Alastor Wu [:alwu]]

Attachment: WIP: Bug 1893427 - part6 : init d3d11va decoder context.

Depends on D227114

Comment 17

[Elle Solomina (she/her)]

The Bug 1927466 is also fixed by this code changes.

Comment 18

[Alastor Wu [:alwu]]

As I would like make this as a meta bug, I will file different follow up bugs to address various issues. First, I'm planing to enable vp9 hw decoding via FFmpeg, then AV1. And then we will consult with the legal team to see if we are able to include the files decoding h264 and hevc into our codebase, if that is fine, then we will make them do HW decoding via FFmpeg as well.

Comment 19

[Alastor Wu [:alwu]]

I will open follow up bugs to implement ffvpx hw decoding for different codecs and platforms. This bug should be used as a meta bug.

Comment 20

[Phabricator Automation]

Comment on attachment 9431710 [details]
Bug 1893427 - part1 : add d3d11va2 vp9 decode path from ffmpeg 7.0 to bundled ffvpx on Windows.

Revision D226067 was moved to bug 1932772. Setting attachment 9431710 [details] to obsolete.

Comment 21

[Phabricator Automation]

Comment on attachment 9431711 [details]
WIP: Bug 1893427 - part2 : add d3d11va2 av1 decode path from ffmpeg 7.0 to bundled ffvpx on Windows.

Revision D226068 was moved to bug 1932772. Setting attachment 9431711 [details] to obsolete.

Comment 22

[Phabricator Automation]

Comment on attachment 9431735 [details]
WIP: Bug 1893427 - part3 : check hw codec support for ffmpeg decoder module.

Revision D226084 was moved to bug 1932772. Setting attachment 9431735 [details] to obsolete.

Comment 23

[Phabricator Automation]

Comment on attachment 9433794 [details]
WIP: Bug 1893427 - part4 : use ffvpx linker on the gpu process in order to support hw decoding.

Revision D227113 was moved to bug 1932772. Setting attachment 9433794 [details] to obsolete.

Comment 24

[Phabricator Automation]

Comment on attachment 9433795 [details]
WIP: Bug 1893427 - part5 : adjust definitions for the preparation of using hw decoding on Windows.

Revision D227114 was moved to bug 1932772. Setting attachment 9433795 [details] to obsolete.

Comment 25

[Phabricator Automation]

Comment on attachment 9433796 [details]
WIP: Bug 1893427 - part6 : init d3d11va decoder context.

Revision D227115 was moved to bug 1932772. Setting attachment 9433796 [details] to obsolete.