1909369 - Improve the quility of encoded result of libx264 through FFmpegVideoEncoder

Status: RESOLVED FIXED

(Core :: Audio/Video: Web Codecs, defect, P1)

Description

[C.M.Chang[:chunmin]]

Attachment: mp4box.mp4

The re-encoded video file of https://vjeux.github.io/mp4-h264-re-encode/mp4box.html on my Ubuntu is terrible. On Windows and MacOS, the output video is clear. However, the output video's resolution on my Ubuntu is really poor.

1. Download video from https://bug1887554.bmoattachments.org/attachment.cgi?id=9392951
2. Go to https://vjeux.github.io/mp4-h264-re-encode/mp4box.html and upload the video downloaded above

Expected:
The quality of re-encoded video is at least acceptable.

Actual:
The quality of re-encoded video (see attachment) is terrible.

Comment 1

[C.M.Chang[:chunmin]]

The output generated from https://vjeux.github.io/mp4-h264-re-encode/mp4box.html is produced by libx264 through FFmpegVideoEncoder on my Ubuntu machine.

Note that the libvpx result is ok, at least in https://capture-and-encode.vercel.app/, which encoded the animation in vp8 and put it to WebM. The output video looks fine.

Comment 2

[C.M.Chang[:chunmin]]

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/home/cm/Downloads/bandicam-2024-03-15-15-21-23-445.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: mp41
    creation_time   : 2024-03-15T09:51:23.000000Z
    encoder         : Bandicam 5.4.2.1921 / GDI
    encoder-eng     : Bandicam 5.4.2.1921 / GDI
  Duration: 00:00:16.35, start: 0.000000, bitrate: 4734 kb/s
  Stream #0:0(eng): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p, 2240x1400, 4533 kb/s, 36.27 fps, 60 tbr, 60k tbn, 120k tbc (default)

The above is the result of running ffprobe -i for for the source file, while the below is the result of the output file:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/home/cm/Downloads/mp4box.mp4':
  Metadata:
    major_brand     : iso4
    minor_version   : 0
    compatible_brands: iso4
  Duration: 00:00:16.35, start: 0.000000, bitrate: 196 kb/s
  Stream #0:0(eng): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 2240x1400, 170 kb/s, 36.27 fps, 60 tbr, 90k tbn, 2000k tbc (default)
    Metadata:
      handler_name    : Track created with MP4Box.js
      vendor_id       : [0][0][0][0]

The bit rate goes down from 4533 kb/s, 170 kb/s, roughly 3.75%

Comment 3

[C.M.Chang[:chunmin]]

If mCodecContext->qmax = 56 is removed, the result of running ffprobe -i for for the output file will be

Stream #0:0(eng): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 2240x1400, 6567 kb/s, 36.27 fps, 60 tbr, 90k tbn, 2000k tbc (default)

The bitrate goes up from 4734 kb/s to 6567 kb/s (~139%), and the video looks normal.

Comment 4

[C.M.Chang[:chunmin]]

Attachment: Bug 1909369 - Sort include headers in FFmpegVideoEncoder.cpp by coding style guide

Comment 5

[C.M.Chang[:chunmin]]

Attachment: Bug 1909369 - Allow setting max and min quantizer values in pref

Depends on D217651

Comment 6

[C.M.Chang[:chunmin]]

Attachment: Bug 1909369 - Adjust default max quantizer values for FFmpegVideoEncoder

Depends on D217652

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/eecb3328ab63
Sort include headers in FFmpegVideoEncoder.cpp by coding style guide r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/85c5a7da7e19
Allow setting max and min quantizer values in pref r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/168e7b762692
Adjust default min and max quantizer values for FFmpegVideoEncoder r=media-playback-reviewers,padenot

Comment 8

[Iulian Moraru]

Backed out for causing crashtest failures on 1891117.html.
LATER EDIT: this also caused some wpt webcodecs failures, so far, on Linux 18.04 x64 WebRender opt and debug: https://treeherder.mozilla.org/logviewer?job_id=468173299&repo=autoland&lineNumber=1848

• Push with failures

• Failure log

• Backout link

[task 2024-07-26T22:36:01.458Z] 22:36:01     INFO - REFTEST TEST-START | dom/media/webcodecs/crashtests/1891117.html
[task 2024-07-26T22:36:01.459Z] 22:36:01     INFO - REFTEST TEST-LOAD | file:///builds/worker/workspace/build/tests/reftest/tests/dom/media/webcodecs/crashtests/1891117.html | 818 / 4053 (20%)
[task 2024-07-26T22:41:01.465Z] 22:41:01     INFO - REFTEST TEST-UNEXPECTED-FAIL | dom/media/webcodecs/crashtests/1891117.html | load failed: timed out waiting for reftest-wait to be removed
[task 2024-07-26T22:41:01.469Z] 22:41:01     INFO - REFTEST INFO | Saved log: START file:///builds/worker/workspace/build/tests/reftest/tests/dom/media/webcodecs/crashtests/1891117.html
[task 2024-07-26T22:41:01.470Z] 22:41:01     INFO - REFTEST INFO | Saved log: [CONTENT] OnDocumentLoad triggering WaitForTestEnd
[task 2024-07-26T22:41:01.471Z] 22:41:01     INFO - REFTEST INFO | Saved log: [CONTENT] WaitForTestEnd: Adding listeners
[task 2024-07-26T22:41:01.472Z] 22:41:01     INFO - REFTEST INFO | Saved log: Initializing canvas snapshot
[task 2024-07-26T22:41:01.473Z] 22:41:01     INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress
[task 2024-07-26T22:41:01.475Z] 22:41:01     INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: STATE_WAITING_TO_FIRE_INVALIDATE_EVENT
[task 2024-07-26T22:41:01.476Z] 22:41:01     INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: dispatching MozReftestInvalidate
[task 2024-07-26T22:41:01.476Z] 22:41:01     INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress
[task 2024-07-26T22:41:01.476Z] 22:41:01     INFO - REFTEST INFO | Saved log: [CONTENT] AfterPaintListener in file:///builds/worker/workspace/build/tests/reftest/tests/dom/media/webcodecs/crashtests/1891117.html
[task 2024-07-26T22:41:01.477Z] 22:41:01     INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: STATE_WAITING_FOR_REFTEST_WAIT_REMOVAL
[task 2024-07-26T22:41:01.478Z] 22:41:01     INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: waiting for reftest-wait to be removed
[task 2024-07-26T22:41:01.478Z] 22:41:01     INFO - REFTEST INFO | Saved log: [CONTENT] HandlePendingTasksAfterMakeProgress updating canvas
[task 2024-07-26T22:41:01.479Z] 22:41:01     INFO - REFTEST INFO | Saved log: [CONTENT] Sending update whole canvas for invalidation
[task 2024-07-26T22:41:01.479Z] 22:41:01     INFO - REFTEST INFO | Saved log: Updating entire canvas for invalidation
[task 2024-07-26T22:41:01.480Z] 22:41:01     INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress
[task 2024-07-26T22:41:01.480Z] 22:41:01     INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: STATE_WAITING_FOR_REFTEST_WAIT_REMOVAL
[task 2024-07-26T22:41:01.481Z] 22:41:01     INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: waiting for reftest-wait to be removed
[task 2024-07-26T22:41:01.481Z] 22:41:01     INFO - REFTEST TEST-END | dom/media/webcodecs/crashtests/1891117.html
[task 2024-07-26T22:41:01.691Z] 22:41:01     INFO - REFTEST TEST-START | dom/media/webcodecs/crashtests/1897631.html

Comment 9

[C.M.Chang[:chunmin]]

(In reply to Iulian Moraru from comment #8)

[task 2024-07-26T22:36:01.458Z] 22:36:01 INFO - REFTEST TEST-START | dom/media/webcodecs/crashtests/1891117.html
[task 2024-07-26T22:36:01.459Z] 22:36:01 INFO - REFTEST TEST-LOAD | file:///builds/worker/workspace/build/tests/reftest/tests/dom/media/webcodecs/crashtests/1891117.html | 818 / 4053 (20%)
[task 2024-07-26T22:41:01.465Z] 22:41:01 INFO - REFTEST TEST-UNEXPECTED-FAIL | dom/media/webcodecs/crashtests/1891117.html | load failed: timed out waiting for reftest-wait to be removed

This is a timeout issue. It looks like the adjustment for qmin and qmax were too aggressive. It sacrificed the performance for better video quality, so the time for compressing video is unacceptable in the tests. I've reverted qmin and adjusted a larger value for qmax and it seems to work: https://treeherder.mozilla.org/jobs?repo=try&revision=f48eac65c8b5885a45f136553322d628ce9fd7de&selectedTaskRun=FNmefGgoS5KNV2QSZPWbkA.0

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/07ceb2aa53d9
Sort include headers in FFmpegVideoEncoder.cpp by coding style guide r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/00d643b71ecf
Allow setting max and min quantizer values in pref r=media-playback-reviewers,padenot
https://hg.mozilla.org/integration/autoland/rev/df9bc2b61f72
Adjust default max quantizer values for FFmpegVideoEncoder r=media-playback-reviewers,padenot

Comment 11

[Mayank Bansal]

How long do other browsers take to run this demo?

From an end-users perspective, the new behaviour may seem that Firefox takes a much (superlinear) larger time to encode the file without user discernible improvement in quality.

Comment 12

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/07ceb2aa53d9
https://hg.mozilla.org/mozilla-central/rev/00d643b71ecf
https://hg.mozilla.org/mozilla-central/rev/df9bc2b61f72

Comment 13

[C.M.Chang[:chunmin]]

(In reply to Mayank Bansal from comment #11)

How long do other browsers take to run this demo?
From an end-users perspective, the new behaviour may seem that Firefox takes a much (superlinear) larger time to encode the file without user discernible improvement in quality.

The processing time for running this demo can vary significantly based on the specific machine and platform/OS being used, as it largely depends on the hardware capabilities available. The perceived quality improvement might differ from user to user. On my end, the quality enhancement is noticeable (I am not sure how you test it before it's merged into tree, on the nightly you built with the patches? or the executable on try? with what H264 codec on your Linux machine?).

Currently, on Linux, the FFmpegVideoEncoder utilizes libx264 for encoding if FFmpeg is installed, which is a software-based solution. This can result in longer processing times compared to hardware-accelerated encoding options available on Windows or macOS systems equipped with the necessary hardware. If you're looking for higher quality, you can adjust the media.ffmpeg.encoder.quantizer-min and media.ffmpeg.encoder.quantizer-max settings to lower values via the about:config page. For example, setting quantizer-min to 3 and quantizer-max to 30 or smaller typically provides a great quality, at least on my end.

Alternatively, you can experiment with enabling the GMP video encoder (introduced in bug 1896758), which uses openh264. This can be done by setting bothmedia.gmp.encoder.enabled and media.gmp.encoder.preferred to true in about:config page. However, please note that this alternative is still under evaluation. Feel free to open bugs if needed.

Comment 14

[C.M.Chang[:chunmin]]

Attachment: mp4box_smaller_qmax.mp4

Here is the new output video generated by the demo page on my Ubuntu on today's Nightly. The quality is much better than that of the old one (see attachment 9414242 [details]).