1944991 - Enable H265 Decoder on MacOS for WebCodecs

Status: RESOLVED FIXED

(Core :: Audio/Video: Web Codecs, task, P2)

Description

[C.M.Chang[:chunmin]]

HEVC is enabled by default on MacOS, so WebCodecs should be able to use it.

Comment 1

[C.M.Chang[:chunmin]]

Attachment: Bug 1944991 - Enable H265 decoding support on macOS

Comment 2

[C.M.Chang[:chunmin]]

Attachment: Bug 1944991 - Add HVCCConfig::ToString()

Depends on D236651

Comment 3

[C.M.Chang[:chunmin]]

Attachment: Bug 1944991 - Add debugging logs

Depends on D236652

Comment 4

[C.M.Chang[:chunmin]]

Attachment: Bug 1944991 - Fake missing extra data in AnnexB-to-HVCC conversion

Depends on D236653

Comment 5

[C.M.Chang[:chunmin]]

While h265.mp4 (h265 in hevc format) works correctly by enabling H265 support in WebCodecs (D236651), the h265.annexb format failes because it has no extra-data, resulting in an Invalid HVCC content error.

I tried to replicate our approach for H.264 - where we fake an extra data if none is provided (D236654) then replace it by the one extracted from the sample - but the AppleVTDecoder created using these settings fails in InitializeSession(). Specifically, VTDecompressionSessionCreate returns -4, indicating it cannot initialize. I need to investigate the proper initialization steps for AppleVTDecoder when using the Annex B format.

Comment 6

[C.M.Chang[:chunmin]]

By HVCCConfig::ToString() added in D236652, the parsed HEVC extra data from h265.mp4 (h265 in hevc format), which passes the WPT, is

D/PlatformDecoderModule AppleVTDecoder[104f96e00] Parsed config: HVCCConfig - version=1, profile_space=0, tier=false, profile_idc=1, profile_compatibility_flags=0x60000000, constraint_indicator_flags=0x00900000000000, level_idc=60, min_spatial_segmentation_idc=0, parallelismType=0, chroma_format_idc=1, bit_depth_luma_minus8=0, bit_depth_chroma_minus8=0, avgFrameRate=0, constantFrameRate=0, numTemporalLayers=1, temporalIdNested=true, lengthSizeMinusOne=3, nalus=3, buffer=110(bytes), NaluSize=4, NumSPS=1

while the one we extract from h265.annexb is

D/PlatformDecoderModule AppleVTDecoder[105995000] Parsed config: HVCCConfig - version=1, profile_space=0, tier=false, profile_idc=1, profile_compatibility_flags=0x60000000, constraint_indicator_flags=0x00900000000000, level_idc=60, min_spatial_segmentation_idc=0, parallelismType=0, chroma_format_idc=1, bit_depth_luma_minus8=0, bit_depth_chroma_minus8=0, avgFrameRate=0, constantFrameRate=0, numTemporalLayers=0, temporalIdNested=false, lengthSizeMinusOne=3, nalus=1, buffer=69(bytes), NaluSize=4, NumSPS=1

The differences between h265.mp4 and h265.annexb are

• numTemporalLayers: 1 vs 0,
• temporalIdNested: true vs false
• buffer: 110 vs 69 (bytes)

When we extract the extra data (H265::ExtractHVCCExtraData) from h265 data, we ignore the avgFrameRate , constantFrameRate, numTemporalLayers and temporalIdNested. I am going to try extracting those data as well to see if it works.

Comment 7

[C.M.Chang[:chunmin]]

Attachment: Bug 1944991 - Alphabetize include headers in H265.h

Comment 8

[C.M.Chang[:chunmin]]

Attachment: Bug 1944991 - Append PPS and VPS to H265 extra data if any

Depends on D236654

Comment 9

[C.M.Chang[:chunmin]]

(In reply to C.M.Chang[:chunmin] from comment #6)

• buffer: 110 vs 69 (bytes)

:alwu told me that AppleVTDecoder requires sps, pps, and vps to setup, so the buffer size should be the main factor. Most tests passed after appending both pps and vps, but decoder crashes somewhere during the test execution. I'll look into it tomorrow.

Comment 10

[C.M.Chang[:chunmin]]

Attachment: Bug 1944991 - Adjust WPT

Depends on D236979

Comment 11

[C.M.Chang[:chunmin]]

Attachment: crashlog

(In reply to C.M.Chang[:chunmin] from comment #9)
> but decoder crashes somewhere during the test execution. I'll look into it tomorrow.


```
 0:13.62 pid:1705 [1722] Assertion failure: mPromise, at /Users/cm/Work/gecko-dev/obj-aarch64-apple-darwin21.4.0/dist/include/mozilla/MozPromise.h:1513
...
Mozilla crash reason: MOZ_ASSERT(Promise)
....
```

I hit an assertion in MozPromise after getting a decoded output from `AppleVTDecoder`. However, I could not reproduce it anymore so I can only upload the stack shown on my terminal. It must be a race somewhere. The fuzzer might help us to catch it.

Comment 12

[C.M.Chang[:chunmin]]

(In reply to C.M.Chang[:chunmin] from comment #11)

I hit an assertion in MozPromise after getting a decoded output from
AppleVTDecoder. However, I could not reproduce it anymore so I can only
upload the stack shown on my terminal. It must be a race somewhere. The
fuzzer might help us to catch it.

I ran into a crash this morning. Here is the stack.

 0:23.97 TEST_END: CRASH, expected OK
 0:23.97 INFO No more tests
 0:42.68 pid:56093 #01: mozilla::MozPromise<nsTArray<RefPtr<mozilla::MediaData> >, mozilla::MediaResult, true>::ThenValue<mozilla::MediaChangeMonitor::DecodeFirstSample(mozilla::MediaRawData*)::$_0, mozilla::MediaChangeMonitor::DecodeFirstSample(mozilla::MediaRawData*)::$_1>::DoResolveOrRejectInternal(mozilla::MozPromise<nsTArray<RefPtr<mozilla::MediaData> >, mozilla::MediaResult, true>::ResolveOrRejectValue&) (/Users/cm/Work/gecko-dev/obj-aarch64-apple-darwin21.4.0/dist/include/mozilla/MozPromise.h:857)
 0:42.68 pid:56093 #02: mozilla::MozPromise<nsTArray<RefPtr<mozilla::MediaData> >, mozilla::MediaResult, true>::ThenValueBase::ResolveOrRejectRunnable::Run() (/Users/cm/Work/gecko-dev/obj-aarch64-apple-darwin21.4.0/dist/include/mozilla/MozPromise.h:489)
 0:42.68 pid:56093 #03: mozilla::TaskQueue::Runner::Run() (/Users/cm/Work/gecko-dev/xpcom/threads/TaskQueue.cpp:268)
 0:42.68 pid:56093 #04: nsThreadPool::Run() (/Users/cm/Work/gecko-dev/xpcom/threads/nsThreadPool.cpp:458)
 0:42.68 pid:56093 #05: nsThread::ProcessNextEvent(bool, bool*) (/Users/cm/Work/gecko-dev/xpcom/threads/nsThread.cpp:1154)
 0:42.68 pid:56093 #06: NS_ProcessNextEvent(nsIThread*, bool) (/Users/cm/Work/gecko-dev/xpcom/threads/nsThreadUtils.cpp:480)
 0:42.68 pid:56093 #07: mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) (/Users/cm/Work/gecko-dev/ipc/glue/MessagePump.cpp:300)
 0:42.68 pid:56093 #08: MessageLoop::RunInternal() (/Users/cm/Work/gecko-dev/ipc/chromium/src/base/message_loop.cc:0)
 0:42.68 pid:56093 #09: MessageLoop::Run() (/Users/cm/Work/gecko-dev/ipc/chromium/src/base/message_loop.cc:345)
 0:42.68 pid:56093 #10: nsThread::ThreadFunc(void*) (/Users/cm/Work/gecko-dev/xpcom/threads/nsThread.cpp:368)
 0:42.83 pid:56093 #11: _pt_root (/Users/cm/Work/gecko-dev/nsprpub/pr/src/pthreads/ptthread.c:194)
 0:42.84 pid:56093 #12: _pthread_start (/usr/lib/system/libsystem_pthread.dylib + 0x726c)

No crash report was created for this in about:crashes, and I haven't reproduced it under lldb successfully yet.

Comment 13

[C.M.Chang[:chunmin]]

Attachment: Bug 1944991 - Fence H265 in Nightly

This patch temporarily fences WebCodecs H265 support in Nightly until
it's stablized. This ensures that any outstanding issues cane be
addressed before enabling it in release.

Depends on D237279

Comment 14

[C.M.Chang[:chunmin]]

Attachment: Bug 1944991 - Add test for ExtractHVCCExtraData

Depends on D237825

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8a734d36adb8
Alphabetize include headers in H265.h r=media-playback-reviewers,jolin
https://hg.mozilla.org/integration/autoland/rev/87d56dbf7ca8
Add HVCCConfig::ToString() r=media-playback-reviewers,jolin
https://hg.mozilla.org/integration/autoland/rev/38ada52b83bb
Add debugging logs r=media-playback-reviewers,jolin
https://hg.mozilla.org/integration/autoland/rev/30b4cf3f555b
Enable H265 decoding support on macOS r=media-playback-reviewers,jolin
https://hg.mozilla.org/integration/autoland/rev/da8bd90758b4
Fake missing extra data in AnnexB-to-HVCC conversion r=media-playback-reviewers,jolin
https://hg.mozilla.org/integration/autoland/rev/91e134f79cf1
Append PPS and VPS to H265 extra data if any r=media-playback-reviewers,jolin
https://hg.mozilla.org/integration/autoland/rev/cdef40747e90
Adjust WPT r=jolin
https://hg.mozilla.org/integration/autoland/rev/c34c9122c361
Fence H265 in Nightly r=media-playback-reviewers,alwu
https://hg.mozilla.org/integration/autoland/rev/60e09be887dd
Add test for ExtractHVCCExtraData r=media-playback-reviewers,alwu

Comment 16

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/8a734d36adb8
https://hg.mozilla.org/mozilla-central/rev/87d56dbf7ca8
https://hg.mozilla.org/mozilla-central/rev/38ada52b83bb
https://hg.mozilla.org/mozilla-central/rev/30b4cf3f555b
https://hg.mozilla.org/mozilla-central/rev/da8bd90758b4
https://hg.mozilla.org/mozilla-central/rev/91e134f79cf1
https://hg.mozilla.org/mozilla-central/rev/cdef40747e90
https://hg.mozilla.org/mozilla-central/rev/c34c9122c361
https://hg.mozilla.org/mozilla-central/rev/60e09be887dd

Comment 17

[Pascal Chevrel (relman team) -> :pascalc|PTO until Sept 2]

Removing from the Nightly release notes as this has been included for more than 3 cycles now. Feel free to nominate it for inclusion once it rides the trains.