1860899 - Implement FFmpegVideoEncoder based on modern APIs for VPX

Status: RESOLVED FIXED

(Core :: Audio/Video, task, P1)

Description

[C.M.Chang[:chunmin]]

There is no MediaDataEncoder implementation on Linux right now. To implement WebCodec's video encoder properly, it's best to implement an FFmpegVideoEncoder with a MediaDataEncoder interface

Comment 1

[C.M.Chang[:chunmin]]

Attachment: WIP: Bug 1860899 - Rename FFVPXRuntimeLinker::Create to ::CreateDecoder

Depends on D193602

Comment 2

[C.M.Chang[:chunmin]]

Attachment: WIP: Bug 1860899 - Rename FFmpegRuntimeLinker::Create to ::CreateDecoder

Depends on D193936

Comment 3

[C.M.Chang[:chunmin]]

Attachment: WIP: Bug 1860899 - Create and bring FFVPXRuntimeLinker::CreateEncoder to PEMFactory

Depends on D193937

Comment 4

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Rename FFVPXRuntimeLinker::Create to ::CreateDecoder

Depends on D193602

Comment 5

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Rename FFmpegRuntimeLinker::Create to ::CreateDecoder

Depends on D193939

Comment 6

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Create and bring FFVPXRuntimeLinker::CreateEncoder to PEMFactory

Depends on D193940

Comment 7

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Create and bring FFmpegRuntimeLinker::CreateEncoder to PEMFactory

Depends on D193941

Comment 8

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Create a FFmpegEncoderModule

Note that FFmpegEncoderModule won't be built since it's not included in
any moz.build yet. The following patches will add this into related
moz.build to build it.

Depends on D193942

Comment 9

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Get FFmpegEncoderModule via FFVPXRuntimeLinker

Depends on D193943

Comment 10

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Get FFmpegEncoderModule via FFmpegRuntimeLinker

Depends on D193944

Comment 11

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Create a FFmpegVideoEncoder

This patch creates an (blank) ffmpeg video encoder and builds it within
ffvpx and ffmpeg libraries with different versions. This encoder can be
obtained via FFmpegEncoderModule::CreateVideoEncoder

Depends on D193945

Comment 12

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Add logs to FFmpegEncoderModule

Depends on D193946

Comment 13

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Implement FFmpegVideoEncoder::GetDescriptionName

Depends on D193947

Comment 14

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Implement FFmpegEncoderModule::SupportsMimeType

Depends on D193948

Comment 15

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Create video encoder only if type is supported

In FFmpegEncoderModule::CreateVideoEncoder, we should only create video
encoder when the mime type is supported by the underlying FFmpeg
encoder.

Depends on D193949

Comment 16

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Add logs to FFmpegVideoEncoder

Depends on D193950

Comment 17

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Init gfxVars in PEMFactory ctor

This patch addes a gfx::gfxVars::Initialize() in PEMFactory, as what
PDMFactory does.

gfx::gfxVars::CanUseHardwareVideoDecoding() will be checked in
FFmpegLibWrapper::LinkVAAPILibs() during FF{VPX,
mpeg}RuntimeLinker::CreateEncoder via ::Init(). This API required to be
called after gfxVars are initialized. PEMFactory ctor seems a good place
to initialize gfx vars.

Depends on D193951

Comment 18

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Add encoder-related APIs into FFmpegLibWrapper

Comment 19

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - clang-format FFmpegLibWrapper.cpp

./mach clang-format --path
dom/media/platforms/ffmpeg/FFmpegLibWrapper.cpp

Depends on D193601

Comment 20

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Alphabetize avcodec.symbols

Depends on D193601

Comment 21

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Expose encoder-related APIs from ffvpx/libavcodec

Depends on D194127

Comment 22

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Add more logs in FFmpegLibWrapper

Depends on D194120

Comment 23

[C.M.Chang[:chunmin]]

Attachment: WIP: Bug 1860899 - Logging FFmpegLibWrapper link results

Depends on D193945

Comment 24

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Logging FFmpegLibWrapper link results

Depends on D193945

Comment 25

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Add a function to get AVCodecID by mime type.

This patch implements a function to get AVCodecID by mime type.

Comment 26

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Allocate/Free AVCodecContext in Init()/Shutdown()

This patch partially implements Init() and Shutdown() to allocate and
release the AVCodecContext respectively. These two functions will be
exectued on the platform's encoder task queue specified in the given
CreateEncoderParams. Thus, the allocated AVCodecContext is a task-queue
only member variable.

Depends on D193952

Comment 27

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Bring template-typed config in FFmpegVideoEncoder

This patch add a VideoConfig member into FFmpegVideoEncoder. Since
VideoConfig is a template, its associated type is introduced into
FFmpegVideoEncoder's template.

Depends on D194580

Comment 28

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Add conversions from encoder's pixel format to avcodec's pixel format

Depends on D194580

Comment 29

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Set AVCodecContext by VideoConfig values

This patch maps the VideoConfig's member values to AVCodecContext's
member values.

Depends on D194581

Comment 30

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Open/Close AVCodecContext in Init()/Shutdown()

avcodec_open2 is not thread-safe until version 60 [1]. Besides, even it
claims to be thread-safe now, data race can appear in situations [2, 3].
To accommodate order FFmpeg versions, opening/closing the AVCodecContext
in a critical section is the simplest way to avoid data race issues.

The following related tests:

1. MediaDataEncoderTest.VP8Create
2. MediaDataEncoderTest.VP8Inits
3. MediaDataEncoderTest.VP9Create
4. MediaDataEncoderTest.VP9Inits

can be run via below command:

MOZ_LOG="FFmpegVideo:5" /mach gtest <MediaDataEncoderTest 1-4>

[1] https://github.com/FFmpeg/FFmpeg/commit/ebf1d0f4bb7e849a4a4481eeba3a96c1df674308
[2] https://ffmpeg.org/pipermail/ffmpeg-devel/2022-January/290914.html
[3] https://ffmpeg.org/pipermail/ffmpeg-devel/2022-January/thread.html#290914

Depends on D194913

Comment 31

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Add a blank ProcessEncode

Depends on D194914

Comment 32

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Add a function to allocate/free AVFrame

Depends on D195056

Comment 33

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Implement Encode based on modern APIs

This patch implement an encoder function based on avcodec_send_frame and
avcodec_receive_packet, which were introduced in FFmpeg 58.

Depends on D195057

Comment 34

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Implement AVPacket-to-MediaRawData conversion

This patch completes the encoding functionality. It implements a
function copying AVPacket's data to a MediaRawData. Accordingly, the
main encode function can return the encoded data properly.

Depends on D195328

Comment 35

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Add a blank ProcessDrain

Depends on D195377

Comment 36

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Implement Drain based on modern APIs

This patch implements drain function based on avcodec_send_frame and
avcodec_receive_packet, similar to what Encode function did.
Thus, it only works when ffmpeg version is at least 58.

It's worth mentioning that the encoder cannot continue encoding once it
enters draining mode, due to API limitation. Users need to encode data
after draining are encouraged to create a new encoder.

Still, the alternative does exist. If AV_CODEC_CAP_ENCODER_FLUSH is set
in the AVCodec's capabilities of the AVCodecContext, then
avcodec_flush_buffers can be used to flush or drain all the encoded
data without explicitly entering draining mode, instead of sending a
NULL to the avcodec_send_frame. However, since this flag is not set
when using the libvpx encoder (at least on my machine), this approach is
put to the backlog.

With the Drain implementation, the patches so far can be tested with the
additional tests below:

1. MediaDataEncoderTest.VP8Encodes
2. MediaDataEncoderTest.VP9Encodes

via the following command:

MOZ_LOG="FFmpegVideo:5" ./mach gtest MediaDataEncoderTest.*

Those tests can check the sanity of Encode() and Drain().

Depends on D195378

Comment 37

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Add a function to get error message

This patch re-impelements the av_err2str macro to get the error
message from the error code, since av_strerror needs to be called via
the FFmpegLibWrapper.

Depends on D195379

Comment 38

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Allow encoder only when ffmpeg >= 58

Since the current Encode API is implemented by the modern APIs
introduced in ffmpeg 58, the encoder functionalities should be limited
by the version accordingly.

Depends on D195380

Comment 39

[C.M.Chang[:chunmin]]

Since the encode(/decode) APIs were changed in ffmpeg 58, I am going to rename this bug and push the implementation based on the older APIs in another bug.

Comment 40

[C.M.Chang[:chunmin]]

The current implementation is only for libvpx, so I am going to reflect this on the bug. Other format support will be tracked in bug 1868402

Comment 41

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Handle RGB-type images in Encode

This patch enables the encoding of RGB-type images.

Depends on D195328

Comment 42

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Init FFVPX lib on any thread

This patch allows linking FFVPX library on any thread.

Depends on D195381

Comment 43

[C.M.Chang[:chunmin]]

Attachment: Bug 1860899 - Activate FFmpegVideoEncoder only if its preference is set

This patch creates a media.ffmpeg.encoder.enabled pref for enabling
FFmpegVideoEncoder. As a result, it won't be activated in the webrtc
case.

Depends on D195739

Comment 44

[Paul Adenot (:padenot)]

Attachment: Bug 1860899 - Update the new ffmpeg PEM with the new PEM interface. r?chunmin

Comment 45

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f80b95e89dc3
Alphabetize avcodec.symbols r=padenot
https://hg.mozilla.org/integration/autoland/rev/87a375e3cf8b
Expose encoder-related APIs from ffvpx/libavcodec r=padenot
https://hg.mozilla.org/integration/autoland/rev/f373dc87bf48
Add more logs in FFmpegLibWrapper r=media-playback-reviewers,alwu
https://hg.mozilla.org/integration/autoland/rev/7acf94ed98f4
Add encoder-related APIs into FFmpegLibWrapper r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/effde2256c28
Rename FFVPXRuntimeLinker::Create to ::CreateDecoder r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/b3160edece06
Rename FFmpegRuntimeLinker::Create to ::CreateDecoder r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/86138ee1179f
Create and bring FFVPXRuntimeLinker::CreateEncoder to PEMFactory r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/fa0141a69328
Create and bring FFmpegRuntimeLinker::CreateEncoder to PEMFactory r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/64a46c99eaec
Create a FFmpegEncoderModule r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/c96b85ca5f90
Get FFmpegEncoderModule via FFVPXRuntimeLinker r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/fa0d4633cbc9
Get FFmpegEncoderModule via FFmpegRuntimeLinker r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/5be4e6ab78f1
Logging FFmpegLibWrapper link results r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/c93459f6cdbe
Create a FFmpegVideoEncoder r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/1c5001fba1a9
Add logs to FFmpegEncoderModule r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/dcc0d26d1be0
Implement FFmpegVideoEncoder::GetDescriptionName r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/b831b0a96d8f
Add a function to get AVCodecID by mime type. r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/e4fc220d0923
Implement FFmpegEncoderModule::SupportsMimeType r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/7b938872c0d0
Create video encoder only if type is supported r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/01e99865ca02
Add logs to FFmpegVideoEncoder r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/774a4e3053b3
Init gfxVars in PEMFactory ctor r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/3eb4213f5e34
Allocate/Free AVCodecContext in Init()/Shutdown() r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/ccbb7e5e48ae
Bring template-typed config in FFmpegVideoEncoder r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/2628ce993127
Add conversions from encoder's pixel format to avcodec's pixel format r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/377ff93825e3
Set AVCodecContext by VideoConfig values r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/012bcf903fa8
Open/Close AVCodecContext in Init()/Shutdown() r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/5ef06b306a36
Add a blank ProcessEncode r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/9095886717c2
Add a function to allocate/free AVFrame r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/5b152c42a6a7
Implement Encode based on modern APIs r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/901e3b9a8e11
Handle RGB-type images in Encode r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/f44e4a298b94
Implement AVPacket-to-MediaRawData conversion r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/0b7884423c94
Add a blank ProcessDrain r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/153024a89a78
Implement Drain based on modern APIs r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/514a900b5ef2
Add a function to get error message r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/fa5a88486f5a
Allow encoder only when ffmpeg >= 58 r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/f24499e2e7af
Init FFVPX lib on any thread r=media-playback-reviewers,karlt,padenot
https://hg.mozilla.org/integration/autoland/rev/935b39fd15fd
Activate FFmpegVideoEncoder only if its preference is set r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/dad6d260d8b9
apply code formatting via Lando

Comment 46

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/f80b95e89dc3
https://hg.mozilla.org/mozilla-central/rev/87a375e3cf8b
https://hg.mozilla.org/mozilla-central/rev/f373dc87bf48
https://hg.mozilla.org/mozilla-central/rev/7acf94ed98f4
https://hg.mozilla.org/mozilla-central/rev/effde2256c28
https://hg.mozilla.org/mozilla-central/rev/b3160edece06
https://hg.mozilla.org/mozilla-central/rev/86138ee1179f
https://hg.mozilla.org/mozilla-central/rev/fa0141a69328
https://hg.mozilla.org/mozilla-central/rev/64a46c99eaec
https://hg.mozilla.org/mozilla-central/rev/c96b85ca5f90
https://hg.mozilla.org/mozilla-central/rev/fa0d4633cbc9
https://hg.mozilla.org/mozilla-central/rev/5be4e6ab78f1
https://hg.mozilla.org/mozilla-central/rev/c93459f6cdbe
https://hg.mozilla.org/mozilla-central/rev/1c5001fba1a9
https://hg.mozilla.org/mozilla-central/rev/dcc0d26d1be0
https://hg.mozilla.org/mozilla-central/rev/b831b0a96d8f
https://hg.mozilla.org/mozilla-central/rev/e4fc220d0923
https://hg.mozilla.org/mozilla-central/rev/7b938872c0d0
https://hg.mozilla.org/mozilla-central/rev/01e99865ca02
https://hg.mozilla.org/mozilla-central/rev/774a4e3053b3
https://hg.mozilla.org/mozilla-central/rev/3eb4213f5e34
https://hg.mozilla.org/mozilla-central/rev/ccbb7e5e48ae
https://hg.mozilla.org/mozilla-central/rev/2628ce993127
https://hg.mozilla.org/mozilla-central/rev/377ff93825e3
https://hg.mozilla.org/mozilla-central/rev/012bcf903fa8
https://hg.mozilla.org/mozilla-central/rev/5ef06b306a36
https://hg.mozilla.org/mozilla-central/rev/9095886717c2
https://hg.mozilla.org/mozilla-central/rev/5b152c42a6a7
https://hg.mozilla.org/mozilla-central/rev/901e3b9a8e11
https://hg.mozilla.org/mozilla-central/rev/f44e4a298b94
https://hg.mozilla.org/mozilla-central/rev/0b7884423c94
https://hg.mozilla.org/mozilla-central/rev/153024a89a78
https://hg.mozilla.org/mozilla-central/rev/514a900b5ef2
https://hg.mozilla.org/mozilla-central/rev/fa5a88486f5a
https://hg.mozilla.org/mozilla-central/rev/f24499e2e7af
https://hg.mozilla.org/mozilla-central/rev/935b39fd15fd
https://hg.mozilla.org/mozilla-central/rev/dad6d260d8b9

Comment 47

[Phabricator Automation]

Comment on attachment 9367506 [details]
Bug 1860899 - Update the new ffmpeg PEM with the new PEM interface. r?chunmin

Revision D195805 was moved to bug 1867542. Setting attachment 9367506 [details] to obsolete.