1749047 - [WebCodecs] Implement VideoEncoder on Linux

Status: RESOLVED FIXED

(Core :: Audio/Video: Web Codecs, task, P1)

Description

[C.M.Chang[:chunmin] | PTO]

Implement https://w3c.github.io/webcodecs/#videoencoder-interface

Comment 2

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Add IsClosed on VideoFrame. r?chunmin

Comment 3

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Add VideoFrame::ToString. r?chunmin

This is debugging-only.

Depends on D195696

Comment 4

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Make Clone const on VideoFrame. r?chunmin

Depends on D195697

Comment 5

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Move and deduplicate some utility functions to WebCodecUtils. r?chunmin

They will be reused for the encoding side.

Depends on D195698

Comment 6

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Add a way to print VideoData object for debugging. r?chunmin

Depends on D195699

Comment 7

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Allow comparing video config and colorspace objects. r?chunmin

Depends on D195700

Comment 8

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Add BitrateMode to MediaRecorder.webidl. r?#webidl

https://w3c.github.io/mediacapture-record/#bitratemode, we're not implementing
this on MediaRecorder, but are needed this for Web Codecs.

Depends on D195701

Comment 9

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - WebCodecsUtils.cpp clang-format. r?chunmin

Depends on D195702

Comment 10

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Introduce the encoder types for the video side. r?chunmin

Depends on D195703

Comment 11

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Implement the boilerplate for VideoEncoder. r?chunmin

Depends on D195704

Comment 12

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Allow converting VideoEncoderConfigInternal to something a PEM can understand. r?chunmin

It intentionally more or less maps 1:1 since my refactoring.

Depends on D195705

Comment 13

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Allow diffing and producing a configuration change list between two encoder configuration. r?chunmin

Depends on D195706

Comment 14

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Add an EncoderTemplate. r?chunmin

It's a lot of code but there is no complexity. Messages are refptrs on this side
because it simplifies the code, remove lots of moves and the messages are const
anyways so uniqueptr don't bring anything.

There are two methods for encoding, because they don't have the same
signature: the video side can take options. Because I didn't want to refactor
yet, the audio side passes in empty options for now.

I'll reflect the changes bug 1865376 in a further patch, but they are similarly
needed.

Depends on D195707

Comment 15

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Implement encoder reconfiguration. r?chunmin

If an EncoderAgent is already present, it means this is a reconfigure, and not
e.g. the first configure call after a reset, or the first configure call ever.

This attempts to set new parameters on a PEM. This can fail, it depends on what
the PEM can do. If it succeed, all is good and this is a success.

If it fails, or if we know that the reconfigure can't succeed (codec change, or
e.g. change between hw and software encoder), the encoder is flushed, destroyed,
and a new one is created with the new params.

Depends on D195708

Comment 16

[Paul Adenot (:padenot)]

Attachment: WIP: Bug 1749047 - NOT READY - AudioEncoder implementation.

No complexity there, just lots of glue.

Depends on D195709

Comment 17

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Add EncoderTraits. r?chunmin

I don't think it's terribly useful but it's there to call into the PEM to check
support. We'll see if we keep it.

Depends on D195710

Comment 18

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Implement an EncoderAgent to bridge Web Codecs and the PEM. r?chunmin

Very similar to the decode side, except simpler because flushing's semantic
match.

Depends on D195711

Comment 19

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Convert from MediaRawData to EncodedVideoChunk one by one. r?chunmin

This simplifies the rest of the code, in remaining patches.

Depends on D195712

Comment 20

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Allow selecting the h264 bitstream format using EncoderConfig. r?jolin

There is a bit of overlap now conceptually between latency and bitstream format,
that will be cleaned up eventually. Web Codecs requires that latency and
bitstream format to be decoupled.

Depends on D195713

Comment 21

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Fix video-encoder-flush.https.any.js. r?chunmin

Depends on D195714

Comment 22

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Switch EncoderAgent::StateChange to Verbose logging. r?chunmin

Depends on D195715

Comment 23

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Allow forcing AVCC format in the AppleVTEncoder. r?jolin

Similarly, this will be decoupled in the future, but this allows running and
passing tests for now. We don't ship it anyways.

Depends on D195716

Comment 24

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Handle h264 bitstream format in WebCodecs. r?chunmin

Depends on D195717

Comment 25

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Teach the AppleEncoderModule to encode in h264 High profile. r?jolin

Useful, supported, and required by Web Codecs.

Depends on D195718

Comment 26

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Fix enable constant bitrate mode on an AppleVTEncoder. r?jolin

Depends on D195719

Comment 27

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - sRGB and BT709 use the same primaries. r?chunmin

From https://w3c.github.io/webcodecs/#enumdef-videocolorprimaries

bt709
Color primaries used by BT.709 and sRGB, as described by [H.273] section 8.1 table 2 value 1.

Depends on D195720

Comment 28

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Set the color spaces in more cases on VideoFrame. r?chunmin

Depends on D195721

Comment 29

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Wire up more codecs from VideoEncoder to the new FFmpeg PEM. r?chunmin

Comment 30

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - The encoders behind the new FFmpegVideoEncoder only support YUV420 input, convert. r?chunmin

Depends on D196108

Comment 31

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Simply use microseconds in FFmpegVideoEncoder.cpp. r?chunmin

The previous code was causing issues when the frame rate wasn't available.

I've opened https://bugzilla.mozilla.org/show_bug.cgi?id=1869560 to revisit
this if feel we need to.

Depends on D196109

Comment 32

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Allow printing a VideoDecoderConfig and a VideoDecoderConfigInternal. r?chunmin

This is very useful to understand in the log what config is output in what
encoder output callback, with respect to configure(...) calls.

Depends on D196209

Comment 33

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Improve logging around encoder output callbacks. r?chunmin

This logs when new configuration is output, and also logs the timestamp, to be
able to better match encode calls and output callbacks when logging.

Depends on D196210

Comment 34

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Properly handle reconfiguration and configuration changes. r?chunmin

When we a regular configure call happens, either it's the first configure call
after a reset (or at all), or we have the guarantee that the previous decoder
has been flushed, and so there are no packets in flight.

When we allow the FFmpegVideoEncoder to reconfigure on the fly, we should keep
track of what config each output should use.

Additionally, it's important to destroy / recreate the new EncoderAgent after
the packets have been output, so that ordering is correct. For this, we can
simply dispatch to the current thread, so that the runnable is after all the
output runnables. This was triggered by the WPT
reconfiguring-encoder.https.any.worker.html?vp9_p0, that doesn't output any
packet for the 16 first encode message, and the outputs 16 packets when the
encoder is flushed while reconfiguring.

Depends on D196211

Comment 35

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Don't leak AVPacket. r?chunmin

Per FFmepg API's it has to be allocated on the heap and freed manually.
MakeScopeExit instead of uniqueptr + free policy is nicer I think because we
need to capture the FFmpegLibWrapper and all that.

Depends on D196212

Comment 36

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Allow some decoder to take their color info from the codec string. r?chunmin

https://searchfox.org/mozilla-central/rev/da48f565f70a57ac28862090828fbaa7fd8556f6/dom/media/VideoUtils.cpp#294
is one example of this.

Depends on D196213

Comment 37

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Add the ability to print a VideoInfo and a VideoDecoderConfigInternal. r?chunmin,#media-playback-reviewers

Comment 38

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - BT709 is a better guess in any case today. r?chunmin,#media-playback-reviewers

Depends on D196336

Comment 39

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Attach color space information to decoded video frame going over IPC. r?chunmin,#media-playback-reviewers

Depends on D196337

Comment 40

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Set colorspace information on VideoFrame constructed from a GPUImage. r?chunmin,#media-playback-reviewers

There is a bit of a guess there as well, but a lot less than before.

https://bugzilla.mozilla.org/show_bug.cgi?id=1869825 has been filed to
investigate this, but there's various bits to it, in gfx and media.

Depends on D196338

Comment 41

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Add a logging statement when outputing video frames, with the timestamp. r?chunmin,#media-playback-reviewers

Depends on D196339

Comment 42

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Guess more colorspace information when missing, based on the information present. r?pehrsons

Depends on D196214

Comment 43

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Tell FFmpegVideoEncoder that it can encode AV1. r?chunmin

Depends on D196340

Comment 44

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Implement plumbing for SVC. r?chunmin

Depends on D196472

Comment 45

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Handle encoders having been reset while outputing encoded data. r?chunmin

Depends on D196473

Comment 46

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Consolidate some codec-specific checks from PEMs and add more checks for VP8 and VP9.r r?chunmin,#media-playback-reviewers

Depends on D196474

Comment 47

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Ask ffmpeg to pass duration through. r?chunmin,#media-playback-reviewers

This is ffmpeg 60 only, we might have to do it ourselves for other version. A
simple queue will work, since for all codecs we intend to support, it's one
packet per video frame.

Depends on D196475

Comment 48

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Resize at the input of the decoder if the source video frame is different from the resolution given to the encoder config. r?chunmin,#media-playback-reviewers

Depends on D196476

Comment 49

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Clear out the colorspace member of a VideoFrame when closing it. r?chunmin

Depends on D196477

Comment 50

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Adjust WPT expectations. r?chunmin,#media-playback-reviewers

Depends on D196478

Comment 51

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Remove unused function to avoid breaking the build. r?chunmin

Depends on D196456

Comment 52

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Use a typical time base in FFmpegVideoEncoder if the framerate isn't known. r?chunmin

Depends on D196480

Comment 53

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Clearly mark the Reconfigure method for FFmpegVideoEncoder as not implemented. r?chunmin

Comment 54

[Paul Adenot (:padenot)]

Attachment: Bug 1749047 - Validate H264 profile right after parsing. r?chunmin

This would error out later anyways, but causes an assertion in ASAN build
because the integer isn't part of the enum.

Depends on D196871

Comment 55

[C.M.Chang[:chunmin] | PTO]

Attachment: Bug 1749047 - Fix TestMediaDataEncoder merge error

Depends on D196872

Comment 56

[C.M.Chang[:chunmin] | PTO]

Attachment: Bug 1749047 - Use FFMPEG_PIX_FMT_YUV420P for D196109 pixel format changes

AV_PIX_FMT_YUV420P is only defined for ffmpeg version >= 54. For
version < 54, it's PIX_FMT_YUV420P.

ffmpeg::FFMPEG_PIX_FMT_YUV420P is the type automatically mapping those
pixel types for us.

Depends on D197019

Comment 57

[C.M.Chang[:chunmin] | PTO]

Attachment: Bug 1749047 - Use FFMPEG_PIX_FMT_YUV420P for D196477 pixel-format changes

AV_PIX_FMT_YUV420P is only defined for ffmpeg version >= 54. For
version < 54, it's PIX_FMT_YUV420P.

ffmpeg::FFMPEG_PIX_FMT_YUV420P is the type automatically mapping those
pixel types for us.

Depends on D197020

Comment 58

[C.M.Chang[:chunmin] | PTO]

Attachment: Bug 1749047 - Include ImageContainer.h in VideoEncoder.cpp

mozilla::layers::Image is incomplete type since it's forward declared
in MediaData.h, unless ImageContainer.h is included.

Depends on D197021

Comment 59

[C.M.Chang[:chunmin] | PTO]

Attachment: Bug 1749047 - Take EncoderConfig in EncoderAgent::Configure

Using a template type, ConfigType, in EncoderAgent::Configure requires a
template implementation. Instead of defining the implementation
somewehre, taking a EncoderConfig in EncoderAgent makes it easier to
add a EncoderTemplate implementation in the future, as long as the
ConfigTypeInternal comes with a ToEncoderConfig() method.

Depends on D197022

Comment 60

[C.M.Chang[:chunmin] | PTO]

Attachment: Bug 1749047 - Remove extra empty line in EncoderTraits.cpp

Depends on D197023

Comment 61

[Jim Mathies [:jimm]]

Latest try push for those following along - https://treeherder.mozilla.org/jobs?repo=try&revision=0abe1d8aa051ff9ba7472f7571e52d1fd6c14f68

Comment 62

[C.M.Chang[:chunmin] | PTO]

Attachment: Bug 1749047 - Explicitly make WPT fail in quantizer bitrate mode

Depends on D197024

Comment 63

[C.M.Chang[:chunmin] | PTO]

Attachment: Bug 1749047 - Explicitly disable scalability mode

Depends on D197095

Comment 64

[C.M.Chang[:chunmin] | PTO]

Attachment: Bug 1749047 - Bring GetExtraData back to ToMediaRawData

GetExtraData, which is removed in D196209, must be added back to
generate the extra data for H264 AVCC format, so the VideoEncoder can
forward those info with the encoded data.

Depends on D197137

Comment 65

[C.M.Chang[:chunmin] | PTO]

Attachment: Bug 1749047 - Disable VideoEncoder WPTs on non-linux platforms

This patch disables the WebCodecs encoding support on non-linux
platforms and adds some checks in WPTs to ensure that the execution
occurs only when the encoding is supported on the platform.

• Add checkEncoderSupport-like code to
  video-encoder-content-hint.https.any.js
• Add checkEncoderSupport to video-encoder-flush.https.any.js
• Add checkEncoderSupport to video-encoder.https.any.js
  and update video-encoder.https.any.js.ini

Depends on D197138

Comment 66

[C.M.Chang[:chunmin] | PTO]

Attachment: Bug 1749047 - Apply tryserver customized settings

• For H264 encoding, FFmpegVideoEncoder requires a dynamically loaded
  ffmpeg with version > 57. However, the ffmpeg verions is 57 in ubuntu
  18.04, so video encoder module returns an "unsupported" for h264
• The av1 encoding of FFmpegVideoEncoder relies on the in-tree libaom,
  but the libaom is out-of-date. It's vendered in 2018, and it's pretty
  slow, so it's likely av1 encoding will get timeout in some situations.

Depends on D197139

Comment 67

[C.M.Chang[:chunmin] | PTO]

Attachment: Bug 1749047 - clang-format FFmpegVideoEncoder.cpp

Run:

./mach clang-format --path \
  dom/media/platforms/ffmpeg/FFmpegVideoEncoder.cpp

Depends on D196872

Comment 68

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3695862104e5
Add IsClosed on VideoFrame. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/9430aed8659c
Make Clone const on VideoFrame. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/029f5470314e
Add VideoFrame::ToString. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/e903ac081227
Move and deduplicate some utility functions to WebCodecUtils. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/2d5bc871fb9e
Add a way to print VideoData object for debugging. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/700cb49e7c8a
Allow comparing video config and colorspace objects. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/b6d97ebd3dda
Add BitrateMode to MediaRecorder.webidl. r=webidl,smaug
https://hg.mozilla.org/integration/autoland/rev/ee31e639b046
Introduce the encoder types for the video side. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/bc4f18f14bd3
Implement the boilerplate for VideoEncoder. r=chunmin,webidl,smaug,saschanaz
https://hg.mozilla.org/integration/autoland/rev/e3395cf67e5c
Allow converting VideoEncoderConfigInternal to something a PEM can understand. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/00a94805df47
Allow diffing and producing a configuration change list between two encoder configuration. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/5a2cef0729cc
Add an EncoderTemplate. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/3ce5091256fd
Implement encoder reconfiguration. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/a65dfea63fc7
Add EncoderTraits. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/e59a8dc5add5
Implement an EncoderAgent to bridge Web Codecs and the PEM. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/da42eb3f9664
Convert from MediaRawData to EncodedVideoChunk one by one. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/23a3b0995f32
Switch EncoderAgent::StateChange to Verbose logging. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/d8501e23246c
Allow forcing AVCC format in the AppleVTEncoder. r=jolin
https://hg.mozilla.org/integration/autoland/rev/272ecdefa3db
Handle h264 bitstream format in WebCodecs. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/b32edc2659a6
Teach the AppleEncoderModule to encode in h264 High profile. r=jolin
https://hg.mozilla.org/integration/autoland/rev/88d92a0509cd
Fix enable constant bitrate mode on an AppleVTEncoder. r=jolin
https://hg.mozilla.org/integration/autoland/rev/0595d51073c6
sRGB and BT709 use the same primaries. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/a78bb8799ef4
Set the color spaces in more cases on VideoFrame. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/bcb45500ba22
Wire up more codecs from VideoEncoder to the new FFmpeg PEM. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/b9234a7fb938
The encoders behind the new FFmpegVideoEncoder only support YUV420 input, convert. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/be191f9ac293
Simply use microseconds in FFmpegVideoEncoder.cpp. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/2e334c8a1160
Allow printing a VideoDecoderConfig and a VideoDecoderConfigInternal. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/2214654fd856
Improve logging around encoder output callbacks. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/12768abca8ac
Properly handle reconfiguration and configuration changes. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/7cbc5704562d
Don't leak AVPacket. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/c781f30e5605
Allow some decoder to take their color info from the codec string. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/d43f0c82f214
Guess more colorspace information when missing, based on the information present. r=pehrsons
https://hg.mozilla.org/integration/autoland/rev/279a5522919a
Add the ability to print a VideoInfo and a VideoDecoderConfigInternal. r=chunmin,media-playback-reviewers
https://hg.mozilla.org/integration/autoland/rev/ce3fa8ca07ec
Attach color space information to decoded video frame going over IPC. r=pehrsons
https://hg.mozilla.org/integration/autoland/rev/b7d997c42167
Set colorspace information on VideoFrame constructed from a GPUImage. r=pehrsons
https://hg.mozilla.org/integration/autoland/rev/26c23ac47959
Add a logging statement when outputing video frames, with the timestamp. r=pehrsons
https://hg.mozilla.org/integration/autoland/rev/f0ac27321cf8
Tell FFmpegVideoEncoder that it can encode AV1. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/ea4d0db67f89
Implement plumbing for SVC. r=chunmin,webidl,smaug
https://hg.mozilla.org/integration/autoland/rev/e99c66838796
Handle encoders having been reset while outputing encoded data. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/4d35297ae4bb
Consolidate some codec-specific checks from PEMs and add more checks for VP8 and VP9.r r=media-playback-reviewers,alwu
https://hg.mozilla.org/integration/autoland/rev/0e68a1cc5f26
Ask ffmpeg to pass duration through. r=media-playback-reviewers,alwu
https://hg.mozilla.org/integration/autoland/rev/7a47354c58c4
Resize at the input of the decoder if the source video frame is different from the resolution given to the encoder config. r=chunmin,media-playback-reviewers
https://hg.mozilla.org/integration/autoland/rev/cc1b04d5f70e
Clear out the colorspace member of a VideoFrame when closing it. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/05a066dcfb52
Adjust WPT expectations. r=chunmin,media-playback-reviewers
https://hg.mozilla.org/integration/autoland/rev/44fac5680126
Remove unused function to avoid breaking the build. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/d3c7d90b2150
Use a typical time base in FFmpegVideoEncoder if the framerate isn't known. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/f55b735f283a
Clearly mark the Reconfigure method for FFmpegVideoEncoder as not implemented. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/53c348265bb8
Validate H264 profile right after parsing. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/51dc14d395c6
Use FFMPEG_PIX_FMT_YUV420P for D196109 pixel format changes r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/6ecbca91effe
Use FFMPEG_PIX_FMT_YUV420P for D196477 pixel-format changes r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/8436bfea59b9
Include ImageContainer.h in VideoEncoder.cpp r=padenot
https://hg.mozilla.org/integration/autoland/rev/db7a0f8d337d
Take EncoderConfig in EncoderAgent::Configure r=padenot
https://hg.mozilla.org/integration/autoland/rev/306ad5c722c8
Remove extra empty line in EncoderTraits.cpp r=padenot
https://hg.mozilla.org/integration/autoland/rev/98d2ecb3f476
Bring GetExtraData back to ToMediaRawData r=media-playback-reviewers,padenot

Comment 69

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/43779 for changes under testing/web-platform/tests

Comment 70

[C.M.Chang[:chunmin] | PTO]

Attachment: Bug 1749047 - harzar fix

This patch uses RootedDictionary around EncodedVideoChunkMetadata,
VideoDecoderConfig and VideoColorSpaceInit in OutputEncodedData to
work around the JS hazard (unrooted) issues.

Comment 71

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b1577155be1c
harzar fix. CLOSED TREE

Comment 72

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/3695862104e5
https://hg.mozilla.org/mozilla-central/rev/9430aed8659c
https://hg.mozilla.org/mozilla-central/rev/029f5470314e
https://hg.mozilla.org/mozilla-central/rev/e903ac081227
https://hg.mozilla.org/mozilla-central/rev/2d5bc871fb9e
https://hg.mozilla.org/mozilla-central/rev/700cb49e7c8a
https://hg.mozilla.org/mozilla-central/rev/b6d97ebd3dda
https://hg.mozilla.org/mozilla-central/rev/ee31e639b046
https://hg.mozilla.org/mozilla-central/rev/bc4f18f14bd3
https://hg.mozilla.org/mozilla-central/rev/e3395cf67e5c
https://hg.mozilla.org/mozilla-central/rev/00a94805df47
https://hg.mozilla.org/mozilla-central/rev/5a2cef0729cc
https://hg.mozilla.org/mozilla-central/rev/3ce5091256fd
https://hg.mozilla.org/mozilla-central/rev/a65dfea63fc7
https://hg.mozilla.org/mozilla-central/rev/e59a8dc5add5
https://hg.mozilla.org/mozilla-central/rev/da42eb3f9664
https://hg.mozilla.org/mozilla-central/rev/23a3b0995f32
https://hg.mozilla.org/mozilla-central/rev/d8501e23246c
https://hg.mozilla.org/mozilla-central/rev/272ecdefa3db
https://hg.mozilla.org/mozilla-central/rev/b32edc2659a6
https://hg.mozilla.org/mozilla-central/rev/88d92a0509cd
https://hg.mozilla.org/mozilla-central/rev/0595d51073c6
https://hg.mozilla.org/mozilla-central/rev/a78bb8799ef4
https://hg.mozilla.org/mozilla-central/rev/bcb45500ba22
https://hg.mozilla.org/mozilla-central/rev/b9234a7fb938
https://hg.mozilla.org/mozilla-central/rev/be191f9ac293
https://hg.mozilla.org/mozilla-central/rev/2e334c8a1160
https://hg.mozilla.org/mozilla-central/rev/2214654fd856
https://hg.mozilla.org/mozilla-central/rev/12768abca8ac
https://hg.mozilla.org/mozilla-central/rev/7cbc5704562d
https://hg.mozilla.org/mozilla-central/rev/c781f30e5605
https://hg.mozilla.org/mozilla-central/rev/d43f0c82f214
https://hg.mozilla.org/mozilla-central/rev/279a5522919a
https://hg.mozilla.org/mozilla-central/rev/ce3fa8ca07ec
https://hg.mozilla.org/mozilla-central/rev/b7d997c42167
https://hg.mozilla.org/mozilla-central/rev/26c23ac47959
https://hg.mozilla.org/mozilla-central/rev/f0ac27321cf8
https://hg.mozilla.org/mozilla-central/rev/ea4d0db67f89
https://hg.mozilla.org/mozilla-central/rev/e99c66838796
https://hg.mozilla.org/mozilla-central/rev/4d35297ae4bb
https://hg.mozilla.org/mozilla-central/rev/0e68a1cc5f26
https://hg.mozilla.org/mozilla-central/rev/7a47354c58c4
https://hg.mozilla.org/mozilla-central/rev/cc1b04d5f70e
https://hg.mozilla.org/mozilla-central/rev/05a066dcfb52
https://hg.mozilla.org/mozilla-central/rev/44fac5680126
https://hg.mozilla.org/mozilla-central/rev/d3c7d90b2150
https://hg.mozilla.org/mozilla-central/rev/f55b735f283a
https://hg.mozilla.org/mozilla-central/rev/53c348265bb8
https://hg.mozilla.org/mozilla-central/rev/51dc14d395c6
https://hg.mozilla.org/mozilla-central/rev/6ecbca91effe
https://hg.mozilla.org/mozilla-central/rev/8436bfea59b9
https://hg.mozilla.org/mozilla-central/rev/db7a0f8d337d
https://hg.mozilla.org/mozilla-central/rev/306ad5c722c8
https://hg.mozilla.org/mozilla-central/rev/98d2ecb3f476
https://hg.mozilla.org/mozilla-central/rev/b1577155be1c

Comment 73

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged by moz-wptsync-bot

Comment 74

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged by moz-wptsync-bot

Comment 75

[C.M.Chang[:chunmin] | PTO]

This bug implemented WebCodecs' video encoder but mainly focused on Linux for interop 2023. Although most of the code would be the same for other platforms, the implementations on other platforms should be tracked. I've opened a meta bug tracking this in bug 1872733, and I am going to move some dependencies there.

Comment 77

[Hamish Willee]

FF123 docs for this can be tracked in https://github.com/mdn/content/issues/31913

Can you please clarify what this actually delivers?

1. What desktop/mobile platforms?
   • The title says "Implement VideoEncoder on Linux" so from that I would assume that this fully supports whatever the spec says for VideoEncoder on FF123, but only on Linux desktop - not Windows, macOS or Firefox for android. However when I look at the gating preference dom.media.webcodecs.enabled I see that this is enabled on nightly for all versions, not restricted at all.
2. Does this match the specification for VideoEncoder (and EncodedVideoChunk, VideoFrame, VideoColorSpace) or are there some known/obvious missing parts?
   • On quick scan it looks like MDN IDL matches the spec, so I guess I'm asking confirmation/things I might have missed.
   • VideoDecoder appears to be gated by same preference, but I assume does not yet need docs.
3. Are there any special things that need to be said about this in terms of compatibility/what is delivered?
   • What I mean is that this is that the docs indicate you can use this to encode specific frames to formats supported by browser encoders, but presumably FF only supports some formats. Do we need to say what they are, or is this something people should query when needed?

Comment 78

[Paul Adenot (:padenot)]

This delivers nothing in release, we'll handle the release notes, there seem to have been something going on.

Comment 79

[Hamish Willee]

Thanks Paul, it's not just release notes. Even though this is not released we would usually record support for the feature behind pref dom.media.webcodecs.enabled in the compatibility data too. Unless there is a reason not to. Hence the question about what platforms and anything that we don't expect to be supported by this. If you "actively don't want" any support in the compatibility data or experimental features page at this point then we'd replace dev-doc-needed flag with dev-doc-complete and move on.

Comment 80

[Paul Adenot (:padenot)]

It's a bit bumpy at the minute, it only really works on Linux desktop, but support for all platforms should come in the next few weeks.

Answering your questions now that I understand what this is about:

1. Linux desktop only right now. I can take care of updating the browser compat thing MDN uses, I often do it myself when I implement / ship new Web API. This is Nightly-only, and is slated to ride the train and reach release in a few cycles.
2. This matches the spec, and the decoder side also implements the spec
3. The only thing I can think of is that encoding video frames the h264 codec isn't going to work on 5-6 years old linux, such as ubuntu 18.04 LTS, but this will fixed before release

Thanks a lot for handling this, lmk here on in chat if you need more info!

Comment 81

[Hamish Willee]

Hi Paul,

Yes, please do update the compatibility data! When you do that, can you please cross link from your PR to https://github.com/mdn/content/issues/31913 for my tracking. I assume you are going to take care of that in the near future (i.e. while it is still behind the pref)?

I'll add entry to the MDN Experimental features page for this - from what you've said I don't think there is much else to do now.

Thanks a lot for handling this, lmk here on in chat if you need more info!

I certainly will! From what I can tell the docs look pretty complete so not much else for me to ask at this point.
But if you think there is something I should doing for this you can tag me here or on the MDN github topic with @hamishwillee.

Comment 82

[Hamish Willee]

OK I've added a note about this in MDN experimental features/release notes. The docs look broadly good and I don't propose to touch them.
This is waiting on compatibility to be provided by PaulA.
Work for me complete for now, so have marked this as "dev-doc-complete".

Comment 83

[Vadim Makeev]

Hey! Let us know if you need any help with the compat data in BCD for this feature. It’s less urgent since it’s an initial release on Linux only and the wider support is coming soon but we always try to keep the compact data up to date with the Firefox releases.

Comment 84

[Hamish Willee]

FYI I've added a BCD showing current progress https://github.com/mdn/browser-compat-data/pull/22289