1376134 - Mediarecorder used on canvas mediastream with audio track results in increasing RAM usage/memory leak

Status: RESOLVED FIXED

(Core :: Audio/Video: Recording, defect, P3)

Description

[duncan83]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0
Build ID: 20170418123315

Steps to reproduce:

1. get video and audio via gUM/peerconnection (webcam/desktop + mic, remote peer)
2. draw video to canvas, and get canvas mediaStream via canvas.captureStream(). call this result canvasStream.
3. add audio track to canvas mediastream via canvasStream.addTrack(audioStream.getAudioTracks()[0]);
4. record result with Media Recorder API

fiddle: https://jsfiddle.net/nh9mhyv0/


Actual results:

Firefox constantly increases in RAM usage. Depending on the dimensions of the canvas, you could pretty much use up all system RAM in under a minute.

I could reproduce this in FF 53+, on both Linux and Windows 10.


Expected results:

canvasStream should have an audio track, and calling media recorder records the media stream as one expects, i.e. video + audio in one container without crashing firefox/system.

Comment 1

[duncan83]

In addition to the above, the resultant recording of a canvas stream + audio track via media recorder api is an empty blob

Comment 2

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Hi Duncan, thanks for filing an issue! Could you elaborate on "Expected results"? Doing a screen capture is typically going to produce a very high resolution recording. Where would you expect Firefox to store the recording if not in RAM?

Also, were you able to make Firefox crash?

Comment 3

[duncan83]

Attachment: ff-webcam.png

MediaRecorder used on webcam + microphone media stream. This use case is my "expectation", in that there is a gradual increase in RAM usage. Additionally, MediaRecorder API produces a video file.

Comment 4

[duncan83]

Attachment: ff-desktop.png

MediaRecorder on media stream derived from canvas.captureStream(). No audio track added to canvas stream via addTrack().

MediaRecorder begins recording at about 280 seconds, and produces a video file at about 590 seconds after the onstop event fires.

A gradual and steady increase in RAM usage over the time frame 280s - 590s, which is expected.

Comment 5

[duncan83]

Attachment: ff-desktop-canvas-audio.png

MediaRecorder on canvas stream derived from canvas.captureStream(). Additionally, the microphone track is added to the canvas media stream via .addTrack() method.

Timeframe:

6 seconds: begin drawing desktop capture stream to canvas

12 seconds: share microphone via gUM, and add audio input track to canvas stream via addTrack() method.

40 seconds: MediaRecorder begins recording the canvas media stream (with attached audio track). Results in a marked increase in RAM usage rate, compared to that of simply recording the webcam, or only desktop stream without audiotrack.

120 seconds: OS memory management kicks in and throttles Firefox's memory usage, bringing it to a crawl.

130 seconds: UI button to trigger MediaRecorder stop event is triggered.

140 seconds: No response from Firefox, browser refresh/F5 attempted.

220 seconds: Firefox produces a video file, but is simply an empty blob.

270 seconds: browser window is refreshed, leading to decline in memory usage.

Comment 6

[duncan83]

Hi Jan-Ivar,

Thank you for your response.

That the recording uses system RAM is fine. But the memory usage rates involved for the different streams are markedly different.

I have attached graphs of a few (unscientific) timed tests for your perusal; the axes are firefox RAM % usage vs time in seconds. Hopefully they can shed light on the issue I am facing.

More often than not, the test crashes firefox if run until OS memory management kicks in. Otherwise, it can take more than a minute for firefox to become responsive again.

P.S. both desktop capture recordings had minimal frame difference, with the only real variable being the existence of an attached audio track or not.

Comment 7

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

This bug got stuck in our triage process. Apologies.

Bryce, since you're looking at MediaRecorder these days, would you mind seeing if you can reproduce this? The last graph for memory usage looks alarming when compared to the others.

Comment 8

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

Will do.

Comment 9

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

Confirm that I'm seeing the same in central. The addition of the audio stream results in memory usage growing, as well as an empty blob from the recorder. I'm going to take a look at if the proposed changes in Bug 1296531 have an interaction with this.

Comment 10

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

1296531's changes didn't seem to remedy the issue. I'd anticipated the changes may have had some impact, but the issue seemed to still take place.

For testing I'm using a modified fiddle which stops the recorder as well as changing some of the handling[1]. I think the issue of getting an empty blob is that we weren't getting the final piece of data being emitted by the recorder. This would also lead to issues if one attempted to re-record, as you'd end up with a mangled recData array. So I think the 0 sized recording is a red herring at this stage.

The ballooning memory usage seems to be caused by buffering raw data. If we look at the logging from the fiddles when we record a video + audio stream we don't get the recorder start event fired until we stop the recording, and at this we process the buffered data. In normal usage we would expect this to take place throughout the recording (and the start event to happen when we begin recording).

I'm currently looking into why this is happening. Investigation at the moment indicates that the opus encoder is not becoming initialized. This gates encoding of video frames (code where we stall is here[2]), so we end up accumulating raw video frames while we wait for the audio encoder to init. Raw video frames hanging out in memory result in our size blowout.

My suspicion is that no audio data is being delivered, and we only end up performing an init when the audio track ends[3]. Hence the above behaviour seen when stopping the recorder.

[1]: https://jsfiddle.net/SingingTree/pxth16v2/
[2]: https://searchfox.org/mozilla-central/source/dom/media/encoder/MediaEncoder.cpp#251
[3]: https://searchfox.org/mozilla-central/source/dom/media/encoder/TrackEncoder.cpp#105

Comment 11

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

I believe my previous comment is correct. The root cause of the issue appears to be our handling of direct and normal/indirect listeners.

I'm using the following fiddle for testing: https://jsfiddle.net/SingingTree/u6822hmy/ For others wishing to test, the buttons aren't nicely disabled, so be sure to start the audio capture before the recording.

When testing a recording with the audio added via the canvas.captureStream I believe we run into the same issue as the initial bug (the canvas helps provide a more minimal example). In this case I observe the following:

1. A DOMMediaStream[1] is created for the canvas capture video stream. The mInputStream is initialized to the canvas video stream.
2. DOMMediaStream::AddTrack[2] is called on the above stream when adding the audio track. My understanding is that DOMMediaStream will now contain have this track as part of its playback stream, but the input steam will still be the canvas stream.
3. The MediaRecorder is created, and inits its encoders. During init the recorder will decide it's directly connected and listen on the DOMMediaStream's input stream[3].
4. MediaEncoder::NotifiedQueuedAudioData[4] ignores audio data arriving because it believes it will be sent via the direct connection. However, since we're connected to the input steam, not the playback stream, the audio data is not sent directly.
5. The audio encoder is not initialized until we shut down our recorder. Lack of init blocks the pipeline, resulting in the issue described in the above comment.

I'm a little bogged down in other tasks ATM, but hope to have some more time soon to look into a fix for this ^

[1]: https://searchfox.org/mozilla-central/source/dom/media/DOMMediaStream.h
[2]: https://searchfox.org/mozilla-central/source/dom/media/DOMMediaStream.cpp#620
[3]: https://searchfox.org/mozilla-central/source/dom/media/MediaRecorder.cpp#783
[4]: https://searchfox.org/mozilla-central/source/dom/media/encoder/MediaEncoder.cpp#126

Comment 12

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

I'm going to wait on the landing of Bug 1296531 to continue working on this. The reworks there will impact the fixes I've been looking at.

Comment 13

[Andreas Pehrson [:pehrsons]]

FWIW a workaround for this issue for now is to record `new MediaStream([audioTrack, ...canvas.captureStream()])`.

Bug 1296531 is targeting Firefox 58.

Comment 14

[Bulk Bug Changes for mreavy's org]

Mass change P2->P3 to align with new Mozilla triage process.

Comment 15

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

With Bug 1296531 rebased and landed I'm seeing this work correctly: the audio encoder is fed data and initializes. We have a fail over where if the audio encoder isn't initialized after a certain amount of time it will try best effort init (see bug 1336367). I had thought we may hit that still, but we do not appear to.

Tested with `media.navigator.audio.full_duplex` true and false, as this changes if we have a direct listener used or not. No issue with either.

I'm going to resolve this. Please reopen if required.