2081651 - WebCodecs H.264 encoder ignores bitrate with system FFmpeg 4.x: stale ffmpeg58 version.h shifts AVCodecContext, x264 opens with qmin=35

Status: ASSIGNED

(Core :: Audio/Video: Web Codecs, defect)

Description

[Adriaan]

Attachment: Simple reproduction webpage. Only reproduces on affected configurations.

Steps to reproduce

1. On Linux with a system FFmpeg 4.x (here Ubuntu 22.04.5: libavcodec 58.134.100, libavutil 56.70.100, libx264 163), using the Mozilla .deb build of Firefox 157.0.1 (not the snap).
2. Open the attached repro.html over http://localhost (WebCodecs needs a secure context).
3. It encodes 60 frames of 1080p canvas content with VideoEncoder (avc1.640029) three times: CBR 31 Mbps, VBR 8 Mbps and VBR 2 Mbps, and prints the bitrate actually produced.

Actual results

All three come out the same: about 0.3-0.4 Mbps, with visibly blocky output (38.5 dB luma PSNR). The requested bitrate has no effect; asking for 100 Mbps, a quantizer, realtime latency mode or another H.264 profile does not change it either.

Cause: the bundled headers in dom/media/platforms/ffmpeg/ffmpeg58/include are a mix. libavcodec/avcodec.h was updated to FFmpeg 4.4.1 in bug 1745284 (2021-12-24), but libavcodec/version.h and libavutil/version.h are still FFmpeg 4.0's (58.18.100 and 56.14.100, last changed in 2018). That version.h does not define FF_API_OLD_ENCDEC, so three AVCodecContext fields the 4.4 library has (refcounted_frames, thread_safe_callbacks, debug_mv) are missing from Firefox's build. refcounted_frames sits right before qcompress, so every field from qcompress to rc_override_count is 4 bytes early in Firefox's view:

field	Firefox ffmpeg58 headers	FFmpeg 4.4.2
refcounted_frames	(absent)	464
qcompress	464	468
qmin	472	476
qmax	476	480
rc_buffer_size	484	488
rc_override	496	496 (realigned)

FFmpegVideoEncoder writes qmin = 10 and qmax = 35 (the media.ffmpeg.encoder.quantizer_* defaults), so libavcodec 4.4 reads qmin = 35 and qmax = -1, and libx264 opens with a minimum QP of 35.

Evidence:

• An LD_PRELOAD probe on avcodec_open2 shows what libavcodec receives from Firefox: bit_rate=31000000 rc_max=31000000 qmin=35 qmax=-1, while Firefox's own MOZ_LOG says quantizer: [10, 35].
• The same probe on x264_encoder_open_163 shows qp_min=35. The stream's x264 SEI reads qpmin=35 qpmax=69.
• Control: the system ffmpeg 4.4.2 CLI on the same libraries with -qmin 10 -qmax 35 gives qp_min=10 qp_max=35.
• The same Firefox with an FFmpeg 8.1 build on LD_LIBRARY_PATH (libavcodec62) gets qpmin=10 qpmax=35, and the output follows the bitrate (0.96 / 1.63 / 1.75 Mbps, 60.7 dB).

Decoding does not read the shifted fields, which is likely why the mismatch went unnoticed until the FFmpeg-based encoder started setting qmin/qmax. This is not bug 1903069 (that added bitrate changes on a running encoder; this is the initial configuration, and it reproduces on 157).

Expected results

The produced bitrate should follow the requested one, as it does with a newer system FFmpeg. libx264 should open with the quantizer range Firefox sets (10 to 35).

Suggested fix: update ffmpeg58/include/libavcodec/version.h and libavutil/version.h to the 4.4.1 versions the other headers came from. A static_assert on offsetof(AVCodecContext, qmin) per header set would catch this kind of drift.

Comment 1

[BMO Github Automation]

Attachment: [mozilla-firefox/firefox] Bug 2081651 - Update the ffmpeg58 version.h headers to FFmpeg 4.4.1. (#390)

(via GitHub webhook, authenticated as shared-secret)

Comment 2

[Adriaan]

Attachment: Bug 2081651 - Update the ffmpeg58 version.h headers to FFmpeg 4.4.1

Bug 1745284 updated the bundled ffmpeg58 libavcodec headers to 4.4.1 but
left libavcodec/version.h and libavutil/version.h at 4.0. Those do not
define FF_API_OLD_ENCDEC, FF_API_THREAD_SAFE_CALLBACKS, FF_API_DEBUG_MV or
FF_API_BUFFER_SIZE_T, so Firefox's build of AVCodecContext lacks fields a
system libavcodec 58 has, and qcompress through rc_override_count sit 4
bytes early (qmin at 472 instead of 476). The encoder's qmin=10/qmax=35
reach libx264 as qmin=35 and no qmax, so WebCodecs H.264 ignores the
requested bitrate. AVPacketSideData.size is also size_t instead of int.

With the 4.4.1 version.h files, AVCodecContext, AVCodec, AVPacket,
AVPacketSideData, AVCodecParameters, AVFrame, AVFrameSideData,
AVCodecHWConfig and AVHWFramesContext match FFmpeg 4.4's layouts exactly.

This also defines FF_API_THREAD_SAFE_CALLBACKS, so FFmpegVideoDecoder now
sets thread_safe_callbacks = 1 with libavcodec 58, as it already does
with libavcodec 59.

Comment 3

[Adriaan]

Patch is up for review on Phabricator.

A correction to the description above: the missing AVCodecContext fields come from four macros the stale version.h files leave undefined, not just FF_API_OLD_ENCDEC. refcounted_frames is behind FF_API_OLD_ENCDEC, thread_safe_callbacks behind FF_API_THREAD_SAFE_CALLBACKS and debug_mv behind FF_API_DEBUG_MV (all libavcodec), and FF_API_BUFFER_SIZE_T (libavutil) is a second mismatch: AVPacketSideData.size is size_t in Firefox's build and int in FFmpeg 4.4, so that struct is 24 bytes instead of 16. Firefox does not read packet side data directly today, so that one is latent.

With FFmpeg 4.4.1's libavcodec/version.h and libavutil/version.h, which is what the patch does, the layouts of AVCodecContext, AVCodec, AVPacket, AVPacketSideData, AVCodecParameters, AVFrame, AVFrameSideData, AVCodecHWConfig and AVHWFramesContext all match FFmpeg 4.4.2's own headers exactly. The patch has not been built or run end to end.

One behaviour change to be aware of: FF_API_THREAD_SAFE_CALLBACKS becomes defined for libavcodec 58, so the existing thread_safe_callbacks = 1 in FFmpegVideoDecoder now also applies on FFmpeg 4.x systems, matching what Firefox already does with libavcodec 59.

Comment 4

[BugBot [:suhaib / :marco]]

The Bugbug bot thinks this bug should belong to the 'Core::Audio/Video: Web Codecs' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 5

[BugBot [:suhaib / :marco]]

Set release status flags based on info from the regressing bug 1745284