1866265 - Support 10-bit HEVC and AV1 for WMF decoder

Status: VERIFIED FIXED

(Core :: Audio/Video: Playback, defect, P2)

Description

[Yannis Juglaret [:yannis]]

Attachment: about-support.json

STR:

• Open Firefox Nightly on Windows.
• Navigate to this page.
• A message appears: "Video can't be played because the file is corrupt."

Expected behavior:

• A play button appears and the video can be viewed by clicking on it.

mozregression brought me to bug 1857097 and indeed:

• having media.wmf.hevc.enabled=1 (default on Nightly) reproduces the issue in all release channels;
• having media.wmf.hevc.enabled=0 (default on Release) fixes the issue in all release channels.

Here is a Media playback profile from 120 release (after setting media.wmf.hevc.enabled=1). I am using Windows 11 23H2 22635.2771 (this is an insider preview build). Attachment is my about:support.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

:alwu, since you are the author of the regressor, bug 1857097, could you take a look? Also, could you set the severity field?

For more information, please visit BugBot documentation.

Comment 2

[Paul Adenot (:padenot)]

Alastor, this is the failure:

LogMessages — (PlatformDecoderModule) WMFDecoderModule::CreateVideoDecoder failed for manager with description wmf HEVC codec hardware video decoder - D3D11, unknown with result: NS_ERROR_DOM_MEDIA_FATAL_ERR (0x806e0005) - MediaResult __cdecl mozilla::WMFVideoMFTManager::InitInternal(void): Fail to set the decoder media types.

Something https://searchfox.org/mozilla-central/source/dom/media/platforms/wmf/WMFVideoMFTManager.cpp#374-453 fails. Yannis has a laptop that can decode HEVC (standard recent Mozilla-issues DELL XPS iirc).

Comment 3

[Yannis Juglaret [:yannis]]

(In reply to Paul Adenot (:padenot) from comment #2)

Yannis has a laptop that can decode HEVC (standard recent Mozilla-issues DELL XPS iirc).

I reported this bug from my desktop machine but the issue also reproduces on the laptop.

Comment 4

[Alastor Wu [:alwu][PTO 11/1-11/9]]

I will check it later. As HEVC is still disabled by default on Release channel, I will need to finish other higher priority tasks first, before going back this bug. Keep NI.

Comment 5

[Alastor Wu [:alwu][PTO 11/1-11/9]]

I will take a look on this recently.

Comment 7

[Alastor Wu [:alwu][PTO 11/1-11/9]]

After a preliminary investigation, this seems related with 10 bits HEVC, and our current implementation doesn't support that. When creating the HEVC decoder, we expect to it to return NV12, which is used for 8 bits. We might need more plumbing to make this work. As the media engine playback is able to play that video, we should be able to make it work for our pipeline, given that we use WMF framework as well.

Comment 8

[Alastor Wu [:alwu][PTO 11/1-11/9]]

Attachment: Bug 1866265 - part1 : use marcos to have better error logging and warning.

By using the new marcos, it would show error via logging or error for
the failure function, instead of a general ENSURE_TRUE check.

Comment 9

[Alastor Wu [:alwu][PTO 11/1-11/9]]

Attachment: Bug 1866265 - part2 : add logs.

Depends on D223444

Comment 10

[Alastor Wu [:alwu][PTO 11/1-11/9]]

Attachment: Bug 1866265 - part3 : use last available type as fallback if the compatible type is not found.

Based on [1], IMFTransform::GetOutputAvailableType() won't guarantee
to return a full list of support type. Some MFTs would even can not
provide an accurate list of output types until the MFT receives the
first input sample.

Therefore, we should set the last available type if we can't find the
compatible type, instead of throwing and error.

[1] https://learn.microsoft.com/en-us/windows/win32/api/mftransform/nf-mftransform-imftransform-getoutputavailabletype#remarks

Depends on D223445

Comment 11

[Alastor Wu [:alwu][PTO 11/1-11/9]]

Attachment: Bug 1866265 - part4 : add test cases for hevc 8-bit and 10-bit.

HEVC 8bit video (1 second long)

ffmpeg -ss 0 -t 1 -i gizmo.mp4 -c:v libx265 -pix_fmt yuv420p -crf 28 > -preset fast gizmo_hevc_8bit_420.mp4

HEVC 10bit video (1 second long)

ffmpeg -ss 0 -t 1 -i gizmo.mp4 -c:v libx265 -pix_fmt yuv420p10le -crf > 28 -x265-params profile=main10 gizmo_hevc_10bit_420.mp4

Depends on D223446

Comment 12

[Alastor Wu [:alwu][PTO 11/1-11/9]]

I will add tests for 10-bit AV1 as well.

Comment 13

[Alastor Wu [:alwu][PTO 11/1-11/9]]

Attachment: Bug 1866265 - part5 : add hw test cases for AV1 8-bit and 10-bit.

AV1 8-bit (1 second long)

ffmpeg -ss 0 -t 1 -i gizmo.mp4 -c:v libaom-av1 -pix_fmt yuv420p -crf 30 -b:v 0 -strict experimental gizmo_av1_8bit_420.webm

AV1 10-bit (1 second long)

ffmpeg -ss 0 -t 1 -i gizmo.mp4 -c:v libaom-av1 -pix_fmt yuv420p10le -crf 30 -b:v 0 -strict experimental gizmo_av1_8bit_420.webm

Depends on D223447

Comment 14

[Alastor Wu [:alwu][PTO 11/1-11/9]]

Attachment: Bug 1866265 - part6 : if no compatible type, check preferred type, then the last available type as final fallback.

When running hevc test on the try server, the MFT on the try server has
MFVideoFormat_NV12 and MFVideoFormat_IYUV on its list. If we choose last
available type, it would be MFVideoFormat_IYUV, causing 10-bit hevc
failed to play on the try server.

As we know what format is usually used for video and audio, we should
use the preferred type first if we can't find the compatible type. If
none of them are found, then the last available type should be used as
a final fallback option.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5aa8aa1746ea
part1 : use marcos to have better error logging and warning. r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/ed7d59402624
part2 : add logs. r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/1a2813cf2c44
part3 : use last available type as fallback if the compatible type is not found. r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/cd492f952446
part4 : add test cases for hevc 8-bit and 10-bit. r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/d1d339952b54
part5 : add hw test cases for AV1 8-bit and 10-bit. r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/8c15f96caa8f
part6 : if no compatible type, check preferred type, then the last available type as final fallback. r=media-playback-reviewers,jolin

Comment 16

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/5aa8aa1746ea
https://hg.mozilla.org/mozilla-central/rev/ed7d59402624
https://hg.mozilla.org/mozilla-central/rev/1a2813cf2c44
https://hg.mozilla.org/mozilla-central/rev/cd492f952446
https://hg.mozilla.org/mozilla-central/rev/d1d339952b54
https://hg.mozilla.org/mozilla-central/rev/8c15f96caa8f

Comment 17

[Ryan VanderMeulen [:RyanVM]]

Did you want to add a relnote for this?

Comment 18

[Alastor Wu [:alwu][PTO 11/1-11/9]]

No, as HEVC is not enabled by default, and we already support AV1 10-bit in software decoding. This bug allows us to use hardware decoding for 10-bit AV1, which doesn't seem important enough to add it in the release note.

Comment 19

[Ada C.]

I was able to reproduce the issue on Win10x64 using Firefox build 132.0a1 (2024-09-01).
Verified as fixed on Win10x64 using Firefox build 134.0a1 and 133.0b9.