1581203 - MediaRecorder::ondataavailable sometimes has empty data on Windows

Status: RESOLVED DUPLICATE of bug 1464268

(Core :: Audio/Video: Recording, defect)

Description

[Justin D'Arcangelo [:justindarc]]

I am seeing an issue that appears to be Windows-only with the following code:

navigator.mediaDevices.getDisplayMedia({
  audio: false,
  video: { frameRate: 3 }
}).then(mediaStream => {
  const mediaRecorder = new MediaRecorder(mediaStream, {
    mimeType: 'video/webm; codecs=vp8'
  });
  mediaRecorder.ondataavailable = (event) => {
    // `event.data` sometimes contains an empty (`size` 0) `Blob` on Windows-only
  };
  mediaRecorder.start(100); // Use 100ms `timeSlice`
});

In the above example, ondataavailable always contains a non-empty Blob on macOS, but on Windows, event.data.size is sometimes 0.

This inconsistent behavior was seen on Windows 10 x64 with Firefox 69.0.

Comment 1

[C.M.Chang[:chunmin]]

(In reply to Justin D'Arcangelo [:justindarc] from comment #0)

In the above example, ondataavailable always contains a non-empty Blob on macOS, but on Windows, event.data.size is sometimes 0.

This inconsistent behavior was seen on Windows 10 x64 with Firefox 69.0.

I indeed got evet with size 0 sometimes with the test in Firefox 69, 70, 71, but it's really rare. It happens when I select another window/screen rather than the current window of the test page. It always comes as the second event. But it only happens around 10% when I test with the above script. I am not sure if it's a bug.

Andreas, would you mind taking a look?

Comment 2

[Justin D'Arcangelo [:justindarc]]

(In reply to C.M.Chang[:chunmin] from comment #1)

I indeed got evet with size 0 sometimes with the test in Firefox 69, 70, 71, but it's really rare.

It is rare, but it does happen. On macOS, I have not seen it happen at all, so something seems to be inconsistent between the two OSes. This caused a bug for me because I was not expecting to ever receive an empty Blob in the event handler, but occasionally we were seeing it. I was able to work around the issue in my code by checking for event.data.size === 0, but I still thought it was worth reporting due to the difference between macOS and Windows. I did not test on Linux.

Comment 3

[Justin D'Arcangelo [:justindarc]]

I should also note that this seems to reproduce more frequently when the machine is constrained on resources.

Comment 4

[Andreas Pehrson [:pehrsons]]

We give no guarantees that the blob contains data, on any platform. The spec does not guarantee this either.

What I can understand is that it's annoying and unintuitive.

What happens internally is that we only send data to the blob once a keyframe has been been encoded, because that let's us finish writing the previous webm cluster since its length just got known -- a keyframe starts a new cluster.

If the timeslice is short we have a mitigation in place to reduce the keyframe interval. But the code to issue the blobs is separate from the code handling the keyframes, so there's no guarantee that a cluster gets written to every blob before being issued.

The mitigation has a downside of creating unusually large blobs, since the keyframe interval is very short.

It's unfortunate that so many examples online are riddled with timeslices. Use it only if you really must. If a timeslice is omitted, or if it's at least large (seconds), we can optimize better. First the keyframe interval is longer so there's better compression, second we're able to store the blobs on disk instead of in memory (the limit is at 1MB per blob by default IIRC).

For use cases where you need low latency to ship the recording off to a server you're generally better off with RTCPeerConnection than with MediaRecorder.

I'm duping this to bug 1464268 as fixing that bug should set up our internals to also prevent getting zero-sized blobs from the timeslice mechanism.

Comment 5

[agowatch]

It might be beneficial to include this in the documentation. Timeslice is useful for realtime applications like screen sharing. Is requestData better suited to such applications?

FWIW, Chrome handles these short timeslices without producing a keyframe at every timeslice interval.

Comment 6

[Andreas Pehrson [:pehrsons]]

(In reply to agowatch from comment #5)

It might be beneficial to include this in the documentation. Timeslice is useful for realtime applications like screen sharing.

What's the use case? What's done with the recorded data that requires low latency?

Is requestData better suited to such applications?

The spec for requestData mentions this explicitly: Note that blob will be empty if no data has been gathered yet.

So it shouldn't come as a surprise if one has read the spec.

Not setting a timeslice would make our keyframe interval longer (1s IIRC), and that would be observable with requestData. So the latency would in practice increase compared to a short timeslice because of our mitigation from bug 1411857.

FWIW, Chrome handles these short timeslices without producing a keyframe at every timeslice interval.

Comment 7

[Justin D'Arcangelo [:justindarc]]

(In reply to Andreas Pehrson [:pehrsons] from comment #6)

(In reply to agowatch from comment #5)

It might be beneficial to include this in the documentation. Timeslice is useful for realtime applications like screen sharing.

What's the use case? What's done with the recorded data that requires low latency?

The use case is mentioned here: "realtime applications like screen sharing"

While I agree that RTCPeerConnection would be better suited for such a thing, MediaRecorder should be able to be used for sending encoded video data over a WebSocket. As mentioned in Comment 5, Chrome is able to handle short timeslices without producing a keyframe for every interval.

My understanding of ondataavailable was that if/when that event fires, the Blob will actually have data. I thought that the only way to run into an empty Blob would be if I called requestData() on my own.

Comment 8

[Andreas Pehrson [:pehrsons]]

(In reply to Justin D'Arcangelo [:justindarc] from comment #7)

(In reply to Andreas Pehrson [:pehrsons] from comment #6)

(In reply to agowatch from comment #5)

It might be beneficial to include this in the documentation. Timeslice is useful for realtime applications like screen sharing.

What's the use case? What's done with the recorded data that requires low latency?

The use case is mentioned here: "realtime applications like screen sharing"

That leaves a lot to be imagined.

While I agree that RTCPeerConnection would be better suited for such a thing, MediaRecorder should be able to be used for sending encoded video data over a WebSocket. As mentioned in Comment 5, Chrome is able to handle short timeslices without producing a keyframe for every interval.

I don't think MediaRecorder was originally designed for realtime output. It was designed for creating media files that can be stored and later played back -- like a camera app on a phone.

RTCPeerConnection was designed for realtime output. Sending realtime video over a TCP socket (you mentioned WebSocket) tends to be not so great. And that's not only because it's TCP, but because MediaRecorder produces a lot more bits than a peer connection since it contains plenty of key frames to make it seekable. An RTCPeerConnection does what it can to avoid keyframes altogether. This becomes especially important for large video frames like you get from capturing the entire screen.

My understanding of ondataavailable was that if/when that event fires, the Blob will actually have data. I thought that the only way to run into an empty Blob would be if I called requestData() on my own.

I agree that seems intuitive, but did you actually read that in the spec?

Comment 9

[Justin D'Arcangelo [:justindarc]]

My understanding of ondataavailable was that if/when that event fires, the Blob will actually have data. I thought that the only way to run into an empty Blob would be if I called requestData() on my own.

I agree that seems intuitive, but did you actually read that in the spec?

Yes. The spec mentions that calling requestData() may yield an empty Blob (https://www.w3.org/TR/mediastream-recording/#dom-mediarecorder-requestdata), but it does not say anything about the possibility of an empty Blob in the ondataavailable event handler (https://www.w3.org/TR/mediastream-recording/#dataavailable).