1870216 - Set temporal scalability configuration for libvpx in FFmpegVideoEncoder

Status: RESOLVED FIXED

(Core :: Audio/Video, task, P1)

Description

[C.M.Chang[:chunmin]]

ScalabilityMode in MediaDataEncoder should be mapped to temporal layers settings in libvpx encoder via ts-parameters: https://ffmpeg.org/ffmpeg-codecs.html#Options-33

Comment 1

[C.M.Chang[:chunmin]]

https://phabricator.services.mozilla.com/D196533,
https://phabricator.services.mozilla.com/D196534

Comment 2

[C.M.Chang[:chunmin]]

Attachment: Bug 1870216 - Add ScalabilityMode tests

With ffmpeg internal logs enabled in bug 1870165, it's easy to check the
ts_number_layers value under temporal layering settings in the logs
[1] when running the command:

MOZ_LOG="FFmpegVideo:5" ./mach gtest \
    MediaDataEncoderTest.VP8EncodeWithTemporalLayers

[1] https://searchfox.org/mozilla-central/rev/ddde3bbcafabe0fc8a36c660b3b673507d3e3874/media/ffvpx/libavcodec/libvpxenc.c#259

Comment 3

[C.M.Chang[:chunmin]]

Attachment: Bug 1870216 - Apply SVC settings for libvpx

Depends on D196533

Comment 4

[C.M.Chang[:chunmin]]

Attachment: WIP: Bug 1870216 - Move H264 functinos to mozilla namespace

Comment 5

[C.M.Chang[:chunmin]]

Attachment: Bug 1870216 - enum class BitrateMode

Depends on D196978

Comment 6

[C.M.Chang[:chunmin]]

Attachment: Bug 1870216 - enum class ScalabilityMode

Depends on D196979

Comment 7

[C.M.Chang[:chunmin]]

Attachment: Bug 1870216 - Add ScalabilityMode parameter to Create*Encoder

Depends on D196980

Comment 8

[Phabricator Automation]

Comment on attachment 9369680 [details]
WIP: Bug 1870216 - Move H264 functinos to mozilla namespace

Revision D196978 was moved to bug 1871122. Setting attachment 9369680 [details] to obsolete.

Comment 9

[C.M.Chang[:chunmin]]

Attachment: Bug 1870216 - Track temporal layer id in FFmpegVideoEncoder

Depends on D196534

Comment 10

[C.M.Chang[:chunmin]]

Attachment: Bug 1870216 - Track temporal layer id in MediaRawData

Depends on D197450

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/702a48a230df
enum class BitrateMode r=media-playback-reviewers,alwu
https://hg.mozilla.org/integration/autoland/rev/70cbf201525f
enum class ScalabilityMode r=media-playback-reviewers,alwu
https://hg.mozilla.org/integration/autoland/rev/3dfe64a290b1
Add ScalabilityMode parameter to Create*Encoder r=media-playback-reviewers,alwu
https://hg.mozilla.org/integration/autoland/rev/9c45c85a8fdf
Apply SVC settings for libvpx r=media-playback-reviewers,alwu
https://hg.mozilla.org/integration/autoland/rev/53ecd2bb0a7b
Track temporal layer id in FFmpegVideoEncoder r=media-playback-reviewers,alwu
https://hg.mozilla.org/integration/autoland/rev/703380b927a7
Track temporal layer id in MediaRawData r=media-playback-reviewers,alwu
https://hg.mozilla.org/integration/autoland/rev/0bf921735e7d
Add ScalabilityMode tests r=media-playback-reviewers,alwu

Comment 12

[Iulian Moraru]

Backed out for causing build bustages on TestMediaDataEncoder.cpp.

• Push with failures

• Failure log

• Backout link

[task 2024-01-03T18:09:03.492Z] 18:09:03     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/dom/media/gtest'
[task 2024-01-03T18:09:03.496Z] 18:09:03     INFO -  /builds/worker/fetches/sccache/sccache /builds/worker/fetches/clang/bin/clang++ --sysroot /builds/worker/fetches/sysroot-x86_64-linux-gnu -o Unified_cpp_dom_media_gtest2.o -c  -I/builds/worker/workspace/obj-build/dist/stl_wrappers -I/builds/worker/workspace/obj-build/dist/system_wrappers -include /builds/worker/checkouts/gecko/config/gcc_hidden.h -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fstack-clash-protection -DNDEBUG=1 -DTRIMMED=1 -DHAVE_UINT64_T -DWEBRTC_MOZILLA_BUILD -DRTC_ENABLE_VP9 -DWEBRTC_POSIX -DWEBRTC_BUILD_LIBEVENT -DWEBRTC_LINUX -DWEBRTC_USE_PIPEWIRE -DWEBRTC_USE_X11 -DENABLE_SET_CUBEB_BACKEND -DVISIBLE_TIMEUNIT_INTERNALS -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -DSTATIC_EXPORTABLE_JS_API -I/builds/worker/checkouts/gecko/dom/media/gtest -I/builds/worker/workspace/obj-build/dom/media/gtest -I/builds/worker/checkouts/gecko/dom/media/driftcontrol -I/builds/worker/checkouts/gecko/dom/media/mediasink -I/builds/worker/checkouts/gecko/dom/media/systemservices -I/builds/worker/checkouts/gecko/dom/media/webrtc -I/builds/worker/checkouts/gecko/dom/media/webrtc/common -I/builds/worker/checkouts/gecko/third_party/libwebrtc -I/builds/worker/checkouts/gecko/third_party/libwebrtc/third_party/abseil-cpp -I/builds/worker/workspace/obj-build/ipc/ipdl/_ipdlheaders -I/builds/worker/checkouts/gecko/ipc/chromium/src -I/builds/worker/checkouts/gecko/dom/media -I/builds/worker/checkouts/gecko/dom/media/encoder -I/builds/worker/checkouts/gecko/dom/media/gmp -I/builds/worker/checkouts/gecko/dom/media/mp4 -I/builds/worker/checkouts/gecko/dom/media/platforms -I/builds/worker/checkouts/gecko/dom/media/platforms/agnostic -I/builds/worker/checkouts/gecko/dom/media/webrtc -I/builds/worker/checkouts/gecko/gfx/2d -I/builds/worker/checkouts/gecko/security/certverifier -I/builds/worker/workspace/obj-build/dist/include -I/builds/worker/workspace/obj-build/dist/include/nspr -I/builds/worker/workspace/obj-build/dist/include/nss -DMOZILLA_CLIENT -include /builds/worker/workspace/obj-build/mozilla-config.h -D_GLIBCXX_USE_CXX11_ABI=0 -fno-sized-deallocation -fno-aligned-new -fcrash-diagnostics-dir=/builds/worker/artifacts -fno-exceptions -fPIC -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -gdwarf-4 -Xclang -load -Xclang /builds/worker/workspace/obj-build/build/clang-plugin/libclang-plugin.so -Xclang -add-plugin -Xclang moz-check -O2 -fno-omit-frame-pointer -funwind-tables -Werror -Wall -Wbitfield-enum-conversion -Wdeprecated-this-capture -Wempty-body -Wformat-type-confusion -Wignored-qualifiers -Wpointer-arith -Wshadow-field-in-constructor-modified -Wsign-compare -Wtautological-constant-in-range-compare -Wtype-limits -Wno-error=tautological-type-limit-compare -Wunreachable-code -Wunreachable-code-return -Wunused-but-set-parameter -Wno-invalid-offsetof -Wclass-varargs -Wempty-init-stmt -Wfloat-overflow-conversion -Wfloat-zero-conversion -Wloop-analysis -Wno-range-loop-analysis -Wc++2a-compat -Wenum-compare-conditional -Wenum-float-conversion -Wno-error=deprecated -Wno-error=deprecated-anon-enum-enum-conversion -Wno-error=deprecated-enum-enum-conversion -Wno-error=deprecated-pragma -Wno-error=deprecated-this-capture -Wcomma -Wimplicit-fallthrough -Wstring-conversion -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=free-nonheap-object -Wno-error=atomic-alignment -Wno-error=deprecated-builtins -Wformat -Wformat-security -Wno-psabi -Wthread-safety -Wno-error=builtin-macro-redefined -Wno-unknown-warning-option -Wno-inconsistent-missing-override -Wno-unused-private-field -fno-strict-aliasing -ffp-contract=off  -MD -MP -MF .deps/Unified_cpp_dom_media_gtest2.o.pp   Unified_cpp_dom_media_gtest2.cpp
[task 2024-01-03T18:09:03.497Z] 18:09:03     INFO -  In file included from Unified_cpp_dom_media_gtest2.cpp:47:
[task 2024-01-03T18:09:03.497Z] 18:09:03    ERROR -  /builds/worker/checkouts/gecko/dom/media/gtest/TestMediaDataEncoder.cpp:390:34: error: no matching function for call to 'CreateH264Encoder'
[task 2024-01-03T18:09:03.498Z] 18:09:03     INFO -    390 |     RefPtr<MediaDataEncoder> e = CreateH264Encoder(
[task 2024-01-03T18:09:03.498Z] 18:09:03     INFO -        |                                  ^~~~~~~~~~~~~~~~~
[task 2024-01-03T18:09:03.502Z] 18:09:03     INFO -  /builds/worker/checkouts/gecko/dom/media/gtest/TestMediaDataEncoder.cpp:25:7: note: expanded from macro 'RUN_IF_SUPPORTED'
[task 2024-01-03T18:09:03.502Z] 18:09:03     INFO -     25 |       test();                               \
[task 2024-01-03T18:09:03.502Z] 18:09:03     INFO -        |       ^~~~
[task 2024-01-03T18:09:03.502Z] 18:09:03     INFO -  /builds/worker/checkouts/gecko/dom/media/gtest/TestMediaDataEncoder.cpp:192:43: note: candidate function not viable: no known conversion from 'Maybe<H264Specific>' to 'MediaDataEncoder::ScalabilityMode' for 5th argument
[task 2024-01-03T18:09:03.502Z] 18:09:03     INFO -    192 | static already_AddRefed<MediaDataEncoder> CreateH264Encoder(
[task 2024-01-03T18:09:03.502Z] 18:09:03     INFO -        |                                           ^
[task 2024-01-03T18:09:03.502Z] 18:09:03     INFO -    193 |     MediaDataEncoder::Usage aUsage = MediaDataEncoder::Usage::Realtime,
[task 2024-01-03T18:09:03.502Z] 18:09:03     INFO -    194 |     MediaDataEncoder::PixelFormat aPixelFormat =
[task 2024-01-03T18:09:03.502Z] 18:09:03     INFO -    195 |         MediaDataEncoder::PixelFormat::YUV420P,
[task 2024-01-03T18:09:03.502Z] 18:09:03     INFO -    196 |     int32_t aWidth = WIDTH, int32_t aHeight = HEIGHT,
[task 2024-01-03T18:09:03.502Z] 18:09:03     INFO -    197 |     MediaDataEncoder::ScalabilityMode aScalabilityMode =
[task 2024-01-03T18:09:03.502Z] 18:09:03     INFO -        |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[task 2024-01-03T18:09:03.502Z] 18:09:03     INFO -    198 |         MediaDataEncoder::ScalabilityMode::None,
[task 2024-01-03T18:09:03.502Z] 18:09:03     INFO -        |         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[task 2024-01-03T18:09:03.503Z] 18:09:03     INFO -  1 error generated.
[task 2024-01-03T18:09:03.503Z] 18:09:03    ERROR -  gmake[4]: *** [/builds/worker/checkouts/gecko/config/rules.mk:690: Unified_cpp_dom_media_gtest2.o] Error 1
[task 2024-01-03T18:09:03.503Z] 18:09:03     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/dom/media/gtest'

Comment 13

[C.M.Chang[:chunmin]]

It's a rebase error: H264AVCC's change in https://phabricator.services.mozilla.com/D196981?id=805039 was dropped in the current version. It's corrected now: https://phabricator.services.mozilla.com/D196981#C7215604NL392

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ef3dca593ae6
enum class BitrateMode r=media-playback-reviewers,alwu
https://hg.mozilla.org/integration/autoland/rev/9591ae9386c4
enum class ScalabilityMode r=media-playback-reviewers,alwu
https://hg.mozilla.org/integration/autoland/rev/8ae151dbbc69
Add ScalabilityMode parameter to Create*Encoder r=media-playback-reviewers,alwu
https://hg.mozilla.org/integration/autoland/rev/cd07cd612b51
Apply SVC settings for libvpx r=media-playback-reviewers,alwu
https://hg.mozilla.org/integration/autoland/rev/3cf6fe799a11
Track temporal layer id in FFmpegVideoEncoder r=media-playback-reviewers,alwu
https://hg.mozilla.org/integration/autoland/rev/6dd95075c651
Track temporal layer id in MediaRawData r=media-playback-reviewers,alwu
https://hg.mozilla.org/integration/autoland/rev/a625bc43c505
Add ScalabilityMode tests r=media-playback-reviewers,alwu

Comment 15

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/ef3dca593ae6
https://hg.mozilla.org/mozilla-central/rev/9591ae9386c4
https://hg.mozilla.org/mozilla-central/rev/8ae151dbbc69
https://hg.mozilla.org/mozilla-central/rev/cd07cd612b51
https://hg.mozilla.org/mozilla-central/rev/3cf6fe799a11
https://hg.mozilla.org/mozilla-central/rev/6dd95075c651
https://hg.mozilla.org/mozilla-central/rev/a625bc43c505