1887554 - Video re-encode demo has rapid unbound system-level memory growth. (Chrome doesnt do anything there)

Status: NEW

(Core :: Audio/Video: Web Codecs, defect)

Description

[Mayank Bansal]

Go to https://vjeux.github.io/mp4-h264-re-encode/mp4box.html OR https://vjeux.github.io/mp4-h264-re-encode/mp4wasm.html
Select teh attached sample video file

AR: Rapid, continuous and potentially unbound system-level memory growth. By "system-level" I mean that In Windows Task Manager, the memory is not attributed to any process (i.e. all Firefox processes use the ususla 200-300MB RAM). But when you open the system-level view, the memory keep on growing.

Proifle with all threads, IO and allocations: https://share.firefox.dev/3x595Rn
Profile with Media logging preset: https://share.firefox.dev/3VzeB90

---

Chrome throws some errors in the console and doesnt progress after a few second. Errors:

/favicon.ico:1

   Failed to load resource: the server responded with a status of 404 ()

mp4box.html:98 DOMException: Encoder creation error.
error @ mp4box.html:98
395mp4box.html:37 Uncaught (in promise) DOMException: Failed to execute 'encode' on 'VideoEncoder': Cannot call 'encode' on a closed codec.
at output (https://vjeux.github.io/mp4-h264-re-encode/mp4box.html:37:17)
122mp4box.html:37 Uncaught (in promise) DOMException: Failed to execute 'encode' on 'VideoEncoder': Cannot call 'encode' on a closed codec.
at output (https://vjeux.github.io/mp4-h264-re-encode/mp4box.html:37:17)
output @ mp4box.html:37
mp4box.html:131 Uncaught (in promise) DOMException: Failed to execute 'flush' on 'VideoEncoder': Cannot call 'flush' on a closed codec.
at mp4boxInputFile.onSamples (https://vjeux.github.io/mp4-h264-re-encode/mp4box.html:131:19)
mp4boxInputFile.onSamples @ mp4box.html:131
mp4box.html:1 A VideoFrame was garbage collected without being closed. Applications should call close() on frames when done with them to prevent stalls.
mp4box.html:1 A VideoFrame was garbage collected without being closed. Applications should call close() on frames when done with them to prevent stalls.

Given this, I dont know what is the expected behaviour. But from an end-user perspective, a rapid memory increase that you cant trace back to Firefox is not great.

Comment 1

[Mayank Bansal]

Attachment: Sample video file

Comment 2

[Mayank Bansal]

Attachment: about:support

Comment 3

[Mayank Bansal]

Bisection:
Bug 1749047 - Bring GetExtraData back to ToMediaRawData r=media-playback-reviewers,padenot

GetExtraData, which is removed in D196209, must be added back to
generate the extra data for H264 AVCC format, so the VideoEncoder can
forward those info with the encoded data.

Differential Revision: https://phabricator.services.mozilla.com/D197138

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1749047

Comment 5

[Paul Adenot (:padenot)]

We might be failing to free something when the WMF decoder decodes the frame or something like that.

Comment 6

[Karl Tomlinson (:karlt)]

Here there is only AMD Radeon(TM) Graphics, but bug 1888354 has Intel(R) UHD Graphics.

Comment 7

[stefan.dessens]

I have difficulty reproducing this on intel (12th gen). I get "DOMException: The given encoding is not supported." at mp4box.html:98. Then it spins the CPU for 2 seconds and stops, barely using any memory.

If I use

hardwareAcceleration: 'prefer-software',

On the decoder (using hardware encoding), then it still throws this error, but allocates ~6GB of GPU memory and an additional ~15GB that cannot be traced to any process (allocated, but never used driver memory?). Memory seems to be freed when the tab is closed. By default, task manager only shows CPU memory.

Don't think it's related to bug 1888354.

Comment 8

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:padenot, could you have a look please?

For more information, please visit BugBot documentation.

Comment 9

[Mayank Bansal]

Attachment: Video that doesnt explode memory.MP4

If you use this video as the input, the memory does not increase much. Make what you will of this.

Comment 10

[C.M.Chang[:chunmin]]

(In reply to Mayank Bansal from comment #3)

Bisection:
Bug 1749047 - Bring GetExtraData back to ToMediaRawData r=media-playback-reviewers,padenot

GetExtraData, which is removed in D196209, must be added back to
generate the extra data for H264 AVCC format, so the VideoEncoder can
forward those info with the encoded data.

Differential Revision: https://phabricator.services.mozilla.com/D197138

On Windows, this code should only be used when H264 is encoded by libx264 through FFmpeg. Did you test it with libx264? From the uploaded about:support, H264 hardware decoding is supported, I wonder if the hardware encoding is also supported. If it is, then H264 data will be encoded by WMFMediaDataEncoder rather then FFmpegVideoEncoder.

Comment 11

[C.M.Chang[:chunmin]]

Hi Mayank,

https://vjeux.github.io/mp4-h264-re-encode/mp4box.html can successfully re-encode the Sample video file on my PC (CPU: AMD Ryzen Threadripper PRO 3975WX 32-Cores 3.50 GHz, GPU: AMD Radeon pro wx 3200). I am not sure how the "system-level" memory was observed previously, but I do see the "Memory" under "In use (Compressed)" section in "Task Manager" is going up from 15 GB to 22 GB during the re-encoding. However, once the re-encoding is finished, the memory goes back to 15 GB. Did you see some memory leaks on you end?

Comment 12

[Mayank Bansal]

(In reply to C.M.Chang[:chunmin] from comment #11)

Hi Mayank,

https://vjeux.github.io/mp4-h264-re-encode/mp4box.html can successfully re-encode the Sample video file on my PC (CPU: AMD Ryzen Threadripper PRO 3975WX 32-Cores 3.50 GHz, GPU: AMD Radeon pro wx 3200). I am not sure how the "system-level" memory was observed previously, but I do see the "Memory" under "In use (Compressed)" section in "Task Manager" is going up from 15 GB to 22 GB during the re-encoding. However, once the re-encoding is finished, the memory goes back to 15 GB. Did you see some memory leaks on you end?

The "system-level" memory that I see is the same as the "In use (compressed)" that you see. It just means that this increase memory is not attributed by Task Manager to any single process.
I have a 16GB machine, so when the "In use (compressed)" reached from 5GB to 12GB, I called it as "unbound".

(In reply to C.M.Chang[:chunmin] from comment #10)

(In reply to Mayank Bansal from comment #3)

Bisection:
Bug 1749047 - Bring GetExtraData back to ToMediaRawData r=media-playback-reviewers,padenot

GetExtraData, which is removed in D196209, must be added back to
generate the extra data for H264 AVCC format, so the VideoEncoder can
forward those info with the encoded data.

Differential Revision: https://phabricator.services.mozilla.com/D197138

On Windows, this code should only be used when H264 is encoded by libx264 through FFmpeg. Do you test it with libx264? From the uploaded about:support, H264 hardware decoding is supported, I wonder if the hardware encoding is also supported. If it is, then H264 data will be encoded by WMFMediaDataEncoder rather then FFmpegVideoEncoder.

I just tested the demo with whatever are the default settings. I dont understand enough to "test it with libx264" :)
Happy to run any debug build or provide any logs from any application.

Comment 13

[C.M.Chang[:chunmin]]

Attachment: Screen Shot 2024-07-22 at 3.11.46 PM.png

On MacOS, while the demote is running, I can see that the RDD process (Remote Data Decoder) and another unnamed content process are using up memory. Once the re-encoding finishes, those memories are freed. I am not sure why this doesn't show on Windows.

Comment 14

[Mayank Bansal]

Behaviour if I use the other video file as input (https://bugzilla.mozilla.org/attachment.cgi?id=9402645) :

Nightly:

1. Takes 90s to run the demo
2. Output file is 4.5mb in size
3. "Total memory used" reaches to 8.5GB, but none of that is attributed to any process

Chrome:

1. Takes 30s to run the demo
2. Output file is 18.5MB in size
3. "Total memory used" reaches 8.5GB AND that is attributed to a Chrome process.

Let me know if i should open a new bug for this.

Comment 15

[Mayank Bansal]

Attachment: bandicam 2024-07-28 12-58-11-171.mp4

Tiny video file that leads to 600MB of temporary memory increase.
And here is a profile with everything enabled : https://share.firefox.dev/3LMobiA

Comment 16

[Mayank Bansal]

Attachment: Sample file that leads to 4GB increase.mp4

File that leads to ~4GB increase in system memory (5.6GB ->9.5GB).
Profile with all options enabled from the profiler: https://share.firefox.dev/3SqvhNQ

Comment 17

[Mayank Bansal]

Samply profile: https://share.firefox.dev/4iD7hC9
chunmin, is this profile more useful?

Comment 18

[C.M.Chang[:chunmin]]

Hey Mayank, thanks for the profile!. Is this recorded with the Firefox release? We recently added some WebCodecs markers in bug 1854596, which will be in FF138. I don't really see the WebCodecs' markers in the profiler, so it's hard to tell what's going on. I suspect this might be related to bug 1932526, but I cannot confirm it for now.

Comment 19

[Mayank Bansal]

This profile was from the latest Nightly. The profile was captured using samply with the thought that it may give more information on memory and windows internal stuff.