1577198 - MediaRecorder generates too many keyframes for a low timeSlice

Status: RESOLVED FIXED

(Core :: Audio/Video: Recording, defect, P2)

Description

[Justin D'Arcangelo [:justindarc]]

When starting a MediaRecorder with a time slice that is less than or equal to the frame rate (e.g. a 100ms timeSlice for a video that is 3 FPS), a keyframe is generated for every single frame.

navigator.mediaDevices.getDisplayMedia({
  audio: false,
  video: { frameRate: 3 } // Capture screen recording at 3 FPS
}).then(mediaStream => {
  const mediaRecorder = new MediaRecorder(mediaStream, {
    mimeType: 'video/webm; codecs=vp8'
  });
  mediaRecorder.ondataavailable = (event) => {
    // Get recorded video data here from `event.data`...
  };
  mediaRecorder.start(100); // Use 100ms `timeSlice`
});

In the above example, the recorded video data extracted in mediaRecorder.ondataavailable contains a keyframe for every single frame resulting in a very large video.

Using the same exact code in Chrome, the frame data of the recorded video is sparsely keyframed appropriately (roughly one keyframe for every 10-20 frames). The recorded video data in Chrome is much, much smaller as a result.

This results in poor performance when using MediaRecorder for certain use cases. For example, to transmit low-FPS video data (like a live screen recording) over a WebSocket. In Firefox, every frame will be hundreds of KB in size since they are all keyframes. However, Chrome's implementation properly encodes a "normal" amount of keyframes where only a single keyframe is hundreds of KB in size followed by many inter-frames that are <20KB in size.

Comment 1

[Andreas Pehrson [:pehrsons]]

This is sadly expected due to bug 1411857. Bug 1464268 might be able to fix this better, long-term.

Comment 2

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1577198 - Catch potential bugs where we try to start a webm with a P-frame. r?bryce!

Comment 3

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1577198 - Don't write cluster sizes, and remove the timeslice-based keyframe interval from the vp8 encoder. r?bryce!

This patch stops serializing the cluster size in the cluster header, meaning
that we can push single frames to a blob, even when they are in an open cluster.
This will decrease latency when the MediaRecorder is recording a video track.

This also lets us remove the previous hack we had in place for decreasing the
latency - namely to set a short keyframe interval when the timeslice was short.
With this removed, video quality will go up for short timeslices.

This could have an impact on which players support our encoded videos. Manual
tests show ourselves, Chrome and ffplay handling this change without issues.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e7bdc0a78e33
Catch potential bugs where we try to start a webm with a P-frame. r=bryce
https://hg.mozilla.org/integration/autoland/rev/7dfc286be921
Don't write cluster sizes, and remove the timeslice-based keyframe interval from the vp8 encoder. r=bryce

Comment 5

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/e7bdc0a78e33
https://hg.mozilla.org/mozilla-central/rev/7dfc286be921