1749046 - (AudioEncoder) [WebCodecs] Implement AudioEncoder

Status: RESOLVED FIXED

(Core :: Audio/Video: Web Codecs, task, P1)

Description

[C.M.Chang[:chunmin]]

Implement https://w3c.github.io/webcodecs/#audioencoder-interface

Comment 1

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Add IsClosed to AudioData. r?chunmin

Depends on D192685

Comment 2

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Allow comparing audio config objects. r?chunmin

Depends on D197915

Comment 3

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Make AudioData::ToString public. r?chunmin

Depends on D197916

Comment 4

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Move some audio utilities to WebCodecUtils. r?chunmin

Depends on D197917

Comment 5

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Allow printing an AudioData for debugging purposes. r?chunmin

Depends on D197918

Comment 6

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Remove stray semicolon in AudioConverter.h. r?chunmin

Comment 7

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Extend the PEM interface to handle audio r?chunmin

Comment 8

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Allow converting all channels of an AudioData to f32 interleaved, but not from script. r?chunmin

Comment 9

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Allow printing an AudioDecoderConfigInternal, for logging. r?chunmin

Sorry for the clang-format changes folded in.

Comment 10

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Interface and implementation for AudioEncoder. r?chunmin,#webidl

This look like it's missing quite a few methods, but this is because the code is
shared and mostly identical in behaviour to the VideoEncoder class -- the
AudioEncoder class only implements the bits of behaviour that are
audio-specific. The audio-specific adjustments to the base class (which are not
numerous) are in the next patch of this series.

Comment 11

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Adjust EncoderTemplate to handle audio-specific things. r?chunmin

Comment 12

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Count the number of encoded packets in logs. r?chunmin

I found that useful when debugging, let's land it.

Comment 13

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Factor out the code to set the description to use it on the audio side, in EncoderTemplate. r?chunmin

Comment 14

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Generalize EncoderTraits so that it works for audio. r?chunmin

Comment 15

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Vendor libopusenc.c and libvorbisenc.c and support files in ffvpx. r?chunmin

Comment 16

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Clarify that AudioData instances hold interleaved audio data. r?chunmin

Comment 17

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Add Vorbis to the codec list in PEM. r?chunmin

Comment 18

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Allow draining an AudioPacketizer. r?chunmin

Comment 19

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Implement TimePacketizer. r?chunmin

Comment 20

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Interface changes to FFmpegDataEncoder to prepare for FFmpegAudioEncoder. r?chunmin

InitPromise now doesn't return the track type (since we're not using anything
related to tracks on the encode side).

Methods now either return a result or a nsresult.

Time base is handled in the child class (since it's going to be different from
audio and video).

Comment 21

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Implement FFmpegAudioEncoder. r?chunmin

It's very similar to the video counterpart, except:

• Conversion to planar is done as needed, f32 is always used.
• Most codec specific options are implemented later in the series.
• The time base is always the sameple-rate, as this is what ffmpeg does
  internally, this limits conversions.
• Codecs only accept a specific number of sample-frame per packet, this class
  repacketizes the input audio in the correct packet size (that can depend on the
  same rate, the codec, some other options).
• The encoder delay isn't really handled, except all packets are shifted, and
  mOriginalPresentationWindows is set as it should so that no timing information
  is lost. This will be changed later.

Comment 22

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Set the channel layout in FFmpegAudioEncoder. r?chunmin

Comment 23

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Add a way to iterate ffmpeg's list that end in 0 to FFmepgUtils.h. r?chunmin

Comment 24

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Resampling input audio to the encoder sample-rate. r?chunmin

Comment 25

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Add more codecs to the PEM interface, even though they have no implementation yet. r?chunmin

Comment 26

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Adjust expectation files for AudioEncoder WPTs. r?chunmin

Comment 27

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Put H264 enums in the correct header. r?chunmin

Comment 28

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Split EncoderConfig.h out of PlatformEncoderModule.h. r?chunmin

Comment 29

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Expose more functions to set ffmpeg options. r?chunmin

Comment 30

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Add Opus-specific configuration options. r?chunmin,#webidl

Comment 31

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Reflect Opus-specific configuration options to FFmpegAudioEncoder configuratin. r?chunmin

Comment 32

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Even when the configuration of an encoder wouldn't change because the config is identical, the decoder config must be emitted. r?chunmin

This fixes a test failure in audio encoder WPTs.

Comment 33

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Add a DTX flag to libopusenc.c. r?chunmin

Comment 34

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Enable DTX and handle DTX packets in FFmpegAudioEncoder. r?chunmin

Comment 35

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Move all ToString() methods on various objects to be nsCString containing utf-8. r?chunmin

Comment 36

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - The Opus code supports bitrate in the [6,510]k range. r?chunmin

Comment 37

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2610b665abfe
Add IsClosed to AudioData. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/60f8d9644c5d
Allow comparing audio config objects. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/98f8a6b55cb0
Make AudioData::ToString public. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/f31d976a1a57
Move some audio utilities to WebCodecUtils. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/00f57d5f27f6
Allow printing an AudioData for debugging purposes. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/8ef0b45aa258
Remove stray semicolon in AudioConverter.h. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/b12d9c0f52f1
Extend the PEM interface to handle audio r=chunmin
https://hg.mozilla.org/integration/autoland/rev/22e7f89634b7
Allow converting all channels of an AudioData to f32 interleaved, but not from script. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/88b7fc2cbc1f
Allow printing an AudioDecoderConfigInternal, for logging. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/3de7e6e305e1
Interface and implementation for AudioEncoder. r=chunmin,webidl,smaug
https://hg.mozilla.org/integration/autoland/rev/b116ff85089d
Adjust EncoderTemplate to handle audio-specific things. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/50d78605b761
Count the number of encoded packets in logs. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/eabb710ecc1c
Factor out the code to set the description to use it on the audio side, in EncoderTemplate. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/ca880ba5c83d
Generalize EncoderTraits so that it works for audio. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/7770bc65067f
Add Opus-specific configuration options. r=chunmin,webidl,smaug
https://hg.mozilla.org/integration/autoland/rev/e1f13ef3badd
Vendor libopusenc.c and libvorbisenc.c and support files in ffvpx. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/dced90edf02c
Clarify that AudioData instances hold interleaved audio data. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/dce9cbf1e6f4
Add Vorbis to the codec list in PEM. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/47240f85df7c
Add more codecs to the PEM interface, even though they have no implementation yet. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/31ec05902d0a
Allow draining an AudioPacketizer. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/b05081eca4fe
Implement TimePacketizer. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/7936d27e4a10
Interface changes to FFmpegDataEncoder to prepare for FFmpegAudioEncoder. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/221953f41e1c
Implement FFmpegAudioEncoder. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/fe2e5e695048
Set the channel layout in FFmpegAudioEncoder. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/9de63c54266d
Add a way to iterate ffmpeg's list that end in 0 to FFmepgUtils.h. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/42d651b12660
Resampling input audio to the encoder sample-rate. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/c36ec1343c7d
Adjust expectation files for AudioEncoder WPTs. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/10a16e8a79ad
Put H264 enums in the correct header. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/e96c1bc9e530
Split EncoderConfig.h out of PlatformEncoderModule.h. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/92a04a3d9162
Expose more functions to set ffmpeg options. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/96806c2bdb29
Reflect Opus-specific configuration options to FFmpegAudioEncoder configuratin. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/022e47d17cca
Even when the configuration of an encoder wouldn't change because the config is identical, the decoder config must be emitted. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/a1c56f710998
Add a DTX flag to libopusenc.c. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/88078e99231d
Enable DTX and handle DTX packets in FFmpegAudioEncoder. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/2a446c2233a2
Move all ToString() methods on various objects to be nsCString containing utf-8. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/29994d907858
The Opus code supports bitrate in the [6,510]k range. r=chunmin
https://hg.mozilla.org/integration/autoland/rev/8b7f3601508a
1860925: apply code formatting via Lando

Comment 38

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/45377 for changes under testing/web-platform/tests

Comment 39

[Paul Adenot (:padenot)]

Attachment: Bug 1749046 - Don't assert when checking audio encoding support on Windows.

Comment 40

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a56c3a8cfb52
Don't assert when checking audio encoding support on Windows.

Comment 41

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/2610b665abfe
https://hg.mozilla.org/mozilla-central/rev/60f8d9644c5d
https://hg.mozilla.org/mozilla-central/rev/98f8a6b55cb0
https://hg.mozilla.org/mozilla-central/rev/f31d976a1a57
https://hg.mozilla.org/mozilla-central/rev/00f57d5f27f6
https://hg.mozilla.org/mozilla-central/rev/8ef0b45aa258
https://hg.mozilla.org/mozilla-central/rev/b12d9c0f52f1
https://hg.mozilla.org/mozilla-central/rev/22e7f89634b7
https://hg.mozilla.org/mozilla-central/rev/88b7fc2cbc1f
https://hg.mozilla.org/mozilla-central/rev/3de7e6e305e1
https://hg.mozilla.org/mozilla-central/rev/b116ff85089d
https://hg.mozilla.org/mozilla-central/rev/50d78605b761
https://hg.mozilla.org/mozilla-central/rev/eabb710ecc1c
https://hg.mozilla.org/mozilla-central/rev/ca880ba5c83d
https://hg.mozilla.org/mozilla-central/rev/7770bc65067f
https://hg.mozilla.org/mozilla-central/rev/e1f13ef3badd
https://hg.mozilla.org/mozilla-central/rev/dced90edf02c
https://hg.mozilla.org/mozilla-central/rev/dce9cbf1e6f4
https://hg.mozilla.org/mozilla-central/rev/47240f85df7c
https://hg.mozilla.org/mozilla-central/rev/31ec05902d0a
https://hg.mozilla.org/mozilla-central/rev/b05081eca4fe
https://hg.mozilla.org/mozilla-central/rev/7936d27e4a10
https://hg.mozilla.org/mozilla-central/rev/221953f41e1c
https://hg.mozilla.org/mozilla-central/rev/fe2e5e695048
https://hg.mozilla.org/mozilla-central/rev/9de63c54266d
https://hg.mozilla.org/mozilla-central/rev/42d651b12660
https://hg.mozilla.org/mozilla-central/rev/c36ec1343c7d
https://hg.mozilla.org/mozilla-central/rev/10a16e8a79ad
https://hg.mozilla.org/mozilla-central/rev/e96c1bc9e530
https://hg.mozilla.org/mozilla-central/rev/92a04a3d9162
https://hg.mozilla.org/mozilla-central/rev/96806c2bdb29
https://hg.mozilla.org/mozilla-central/rev/022e47d17cca
https://hg.mozilla.org/mozilla-central/rev/a1c56f710998
https://hg.mozilla.org/mozilla-central/rev/88078e99231d
https://hg.mozilla.org/mozilla-central/rev/2a446c2233a2
https://hg.mozilla.org/mozilla-central/rev/29994d907858
https://hg.mozilla.org/mozilla-central/rev/8b7f3601508a
https://hg.mozilla.org/mozilla-central/rev/a56c3a8cfb52

Comment 42

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged by moz-wptsync-bot