1978232 - Implement EncodeBatch

Status: RESOLVED FIXED

(Core :: Audio/Video, task, P5)

Description

[C.M.Chang[:chunmin]]

MediaDataEncoder should provide an EncodeBatch(nsTArray<RefPtr<MediaData>>&& aSamples) interface, mirroring the one in MediaDataDecoder, to allow users to encode all collected pending input samples in a single call.

This bug will implement this for WMFMediaDataEncoder first

Comment 1

[C.M.Chang[:chunmin]]

Unlike MediaDataDecoder, I propose signaling whether EncodeBatch is supported through the result of MediaDataEncoder's InitPromise, rather than adding a separate CanEncodeBatch() method.

For implementing CanEncodeBatch in RemoteMediaDataEncoderChild, there are two options:

1. Always return true, and internally manage pending inputs within RemoteMediaDataEncoderChild if the underlying encoder does not support EncodeBatch.
2. Query the CanEncodeBatch capability from the underlying encoder via RemoteMediaDataEncoderParent to reflect the literal capability. However, this would require an asynchronous IPC round-trip, adding unnecessary complexity.

Since Encode() must be called only after Init(), and Init() is already asynchronous and responsible for checking the encoder’s capabilities, it is more straightforward to include a flag in the InitPromise result indicating whether EncodeBatch is supported. Besides, RemoteMediaDataEncoderChild is still able to support EncodeBatch this way, by managing pending inputs internally if needed.

Comment 2

[C.M.Chang[:chunmin]]

Attachment: Bug 1978232 - Enable batch encoding in MFTEncoder

Comment 3

[C.M.Chang[:chunmin]]

Attachment: Bug 1978232 - Correct InitPromise resolved values for several encoders

Comment 4

[C.M.Chang[:chunmin]]

Attachment: Bug 1978232 - Add batch endoing API to WMFMediaDataEncoder

Comment 5

[C.M.Chang[:chunmin]]

Attachment: Bug 1978232 - Enable batch encoding for RemoteMediaDataEncoder

Comment 6

[C.M.Chang[:chunmin]]

Attachment: Bug 1978232 - Minor cleanup in EncoderAgent

1. Remove unused EncoderAgent::Dry
2. Fix typo in EncoderAgent::Encode
3. Rename EncoderAgent::State::Flushing to Draining
   (No Flush API in MediaDataEncoder)
4. Shorten helper's name from DryUntilDrain to Dry

Comment 7

[C.M.Chang[:chunmin]]

Attachment: Bug 1978232 - Enable batch encoding in EncoderAgent and EncoderTemplate

Comment 8

[C.M.Chang[:chunmin]]

Attachment: Bug 1978232 - Simplify MediaDataEncoder gTests and improve error handling

1. Simplify utilities for Encode() and Drain()
2. Fail the tests with the error received instead of silently fall on length comparison failure

Comment 9

[C.M.Chang[:chunmin]]

Attachment: Bug 1978232 - Add batch encoding tests for H264 encoders

Comment 10

[C.M.Chang[:chunmin]]

Attachment: Bug 1978232 - Add batch encoding to various encoders

This patch extends batch encoding support to additional encoders, inclduing
FFmpeg encoders, GMP encoder, Android encoder, and macOS encoders. The
implementations for GMP and macOS are not optimized for best real-time
performance, but their usage will be limited via a preference. A more complete
implementation will be addressed in Bug 1984936.

Comment 11

[C.M.Chang[:chunmin]]

https://treeherder.mozilla.org/jobs?repo=try&author=cchang%40mozilla.com&selectedTaskRun=d8GA2OKvQ-uZhDkRg_oQjA.0

The try build shows that it encounters a build issue in Bug 1985779

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/840f5c23bbb5
https://hg.mozilla.org/integration/autoland/rev/b69b5e109e20
Correct InitPromise resolved values for several encoders r=media-playback-reviewers,padenot
https://github.com/mozilla-firefox/firefox/commit/ed66df45ac5c
https://hg.mozilla.org/integration/autoland/rev/162a9c7f2d8c
Enable batch encoding in MFTEncoder r=media-playback-reviewers,jolin
https://github.com/mozilla-firefox/firefox/commit/6d9fcd3e848b
https://hg.mozilla.org/integration/autoland/rev/ed01114ba3f0
Add batch endoing API to WMFMediaDataEncoder r=media-playback-reviewers,jolin
https://github.com/mozilla-firefox/firefox/commit/a459a99b77da
https://hg.mozilla.org/integration/autoland/rev/da26055f237b
Add batch encoding to various encoders r=media-playback-reviewers,jolin
https://github.com/mozilla-firefox/firefox/commit/eb6516a8687a
https://hg.mozilla.org/integration/autoland/rev/2f03e71f9c38
Enable batch encoding for RemoteMediaDataEncoder r=media-playback-reviewers,padenot
https://github.com/mozilla-firefox/firefox/commit/d35966d887b5
https://hg.mozilla.org/integration/autoland/rev/91d92a991a85
Simplify MediaDataEncoder gTests and improve error handling r=jolin
https://github.com/mozilla-firefox/firefox/commit/ae28af6a66e6
https://hg.mozilla.org/integration/autoland/rev/8797167998a3
Add batch encoding tests for H264 encoders r=jolin
https://github.com/mozilla-firefox/firefox/commit/7b82005781e6
https://hg.mozilla.org/integration/autoland/rev/11fcf466d916
Minor cleanup in EncoderAgent r=media-playback-reviewers,padenot
https://github.com/mozilla-firefox/firefox/commit/fa73101fbf4f
https://hg.mozilla.org/integration/autoland/rev/8c12efc0cf12
Enable batch encoding in EncoderAgent and EncoderTemplate r=media-playback-reviewers,padenot,jolin
https://github.com/mozilla-firefox/firefox/commit/3fd99c1145ff
https://hg.mozilla.org/integration/autoland/rev/a625072aae8c
apply code formatting via Lando

Comment 13

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/b69b5e109e20
https://hg.mozilla.org/mozilla-central/rev/162a9c7f2d8c
https://hg.mozilla.org/mozilla-central/rev/ed01114ba3f0
https://hg.mozilla.org/mozilla-central/rev/da26055f237b
https://hg.mozilla.org/mozilla-central/rev/2f03e71f9c38
https://hg.mozilla.org/mozilla-central/rev/91d92a991a85
https://hg.mozilla.org/mozilla-central/rev/8797167998a3
https://hg.mozilla.org/mozilla-central/rev/11fcf466d916
https://hg.mozilla.org/mozilla-central/rev/8c12efc0cf12
https://hg.mozilla.org/mozilla-central/rev/a625072aae8c

Comment 14

[Donal Meehan [:dmeehan]]

:chunmin, could you consider nominating this for a release note? (Process info)

Comment 15

[C.M.Chang[:chunmin]]

:chunmin, could you consider nominating this for a release note? (Process info)

Not sure if this is important to regular users since it's more for website/front-end engineers, but I can add some notes describing what this bug was doing.

Release Note Request (optional, but appreciated)
[Why is this notable]:
This work enables batch submission for WebCodecs' encode requests, improving throughput and reducing encode submission latency (especially right after encoder initialization). It's pref-gated so testers can experiment and compare performance.
[Affects Firefox for Android]: No practical impact. On non-Windows platforms the batch size defaults to 1, so behavior is unchanged unless explicitly overridden.
[Suggested wording]:
WebCodecs now has a pref-gated batch-encoding path for VideoEncoder (Bug 1978232). On Windows, a large default batch size (controlled by dom.media.webcodecs.batch-encoding-size) improves encoding throughput and reduces encodes submission latency. Other platforms default to a batch size of 1, preserving current behavior. When media.use-remote-encoder.video is enabled, batching further reduces cross-process overhead for better performance.
[Links (documentation, blog post, etc)]: This page (bug 1978232)?

Comment 16

[Donal Meehan [:dmeehan]]

Not sure if this is important to regular users since it's more for website/front-end engineers

Thanks, we have a section in the release notes for website/front-end engineers. I'll will add it to that section, but I will slightly reword it that it's only on by default on Windows.
Will update here once that's done.

Comment 17

[Donal Meehan [:dmeehan]]

Added to the Fx144 nightly release notes, please allow 30 minutes for the site to update.
Keeping the relnote-firefox flag as ? to keep it on the radar for inclusion in the final Fx144 release notes.

Comment 18

[Hamish Willee]

FF144 MDN work for this can be tracked in https://github.com/mdn/content/issues/41139

MDN mostly documents things that impact what front-end developers have to do to use an API.
To me this change looks to be "internal" - i.e. it makes WebCodecs more efficient, but it doesn't require any extra awareness from front end developers in order for them to benefit from it.
So I don't think there is any compatibility change or API interface change that needs to be documented.
Can you confirm?

If there is a compatibility or interface change for JavaScript developers, please outline what changes you would like to see in the documentation.

Comment 19

[C.M.Chang[:chunmin]]

You're right. This is more like an internal change. No compatibility or API interface change.

On Windows, this feature will be turned on automatically for H264 encoders when certain conditions are met. Users on other platforms can turn it for on for H264 encoders by setting the preference value (currently, non-H264 encoders are not allowed to use this feature).

Comment 20

[Donal Meehan [:dmeehan]]

Added to the Fx144 release notes