1340302 - Large lag when playing video from MediaRecorder via MediaSource

Status: NEW

(Core :: Audio/Video: Recording, defect, P3)

Description

[David Jameson]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36

Steps to reproduce:

1. Go to https://www.groupboard.com/gum_record/
2. Click record
3. Look at the video in the second window (which is played back using MediaSource).


Actual results:

Firefox 54.0a1 has a 4 second lag.
Firefox 51.0.1 has a 2 second lag.
Chrome canary has < 1 second lag.


Expected results:

Ideally lag should be minimal, similar to Chrome.

Comment 1

[Byron Campen [:bwc]]

I can't get https://www.groupboard.com/gum_record/ to work at all on 51, 53, or 54. Lots of JS errors.

Comment 2

[David Jameson]

Sorry, forgot to mention you'll need to set media.mediasource.webm.enabled to true in about:config.

Comment 3

[Byron Campen [:bwc]]

Lag is pretty bad for me too, yeah.

Comment 4

[Randell Jesup [:jesup] (needinfo me)]

I will note that there's no requirement that MediaRecorder streams when immediately played back be low-latency, though certainly there's advantage to it and not having it be *too* high). Don't assume it's realtime like webrtc; if you're just playing it back somewhere and want realtime, use webrtc.

Also, is the latency as bad when doing audio-only recording?

Comment 5

[David Jameson]

Any idea what parameters to use for audio only? It says "video format or mime type is not supported" when I tell webrtc to use audio only and set the mime types to 'video/webm; codecs="opus"':

https://www.groupboard.com/gum_record/index2.html

The problem seems to be in MediaRecorder: you can see from the console.logs that there is a delay of a few seconds before ondataavailable is called, whereas on Chrome it's being called about 10 times per second (as we request) right from the beginning. I guess it is just the vp8 encoding parameters, which we don't have access to via MediaRecorder. It would be nice if there was a "realtime" flag we could pass into the options to the MediaRecorder constructor.

Comment 6

[Andreas Pehrson [:pehrsons]]

On my machine we're getting 2 second blobs.

We are not violating the spec which says this:
> If timeslice is not undefined, then once a minimum of timeslice milliseconds of data have been collected, or some minimum time slice imposed by the UA, whichever is greater, start gathering data into a new Blob blob, and queue a task, using the DOM manipulation task source, that fires a blob event named dataavailable at target with blob.


(In reply to David Jameson from comment #5)
> Any idea what parameters to use for audio only? It says "video format or
> mime type is not supported" when I tell webrtc to use audio only and set the
> mime types to 'video/webm; codecs="opus"':

Skip the mime type altogether and just pass a MediaStream with one audio track. The blob will tell you the mime type. I think there's a way to figure it out on the MediaRecorder object too. If you really want to pass a mime type, try 'audio/ogg; codecs="opus"'.

---

If you want to go deeper; start firefox with the following env var set: `MOZ_LOG=timestamp,sync,MediaRecorder:4`
Our code is not gonna output blobs faster than you see logs on stderr containing "Session.Extract ...".
This is run on a dedicated thread as often as possible (bound by the encoders).
If you see "Session.Extract" more often than you get blobs, we can look into it.

Future refactors may improve things.

Comment 7

[David Jameson]

Agreed, Firefox isn't violating the spec. However it would certainly be more useful to call ondataavailable every 0.1s if that is what we request, rather than a minimum of every 2 seconds. Chrome gives much smaller Blobs (presumably closer to the 0.1s that we request).

I didn't have much luck specifying either "audio/ogg" or 'audio/ogg; codecs="opus"', as both gave errors saying it was an invalid format for addSourceBuffer. I'm setting the Blob mime type to "audio/ogg", so I don't think reading it back will show anything different.

I'll check out the stderr logs.

Comment 8

[David Jameson]

When I set that env var, I see Session.Extract much more often than I get blobs (> 10 times per second, compared to one blob every 2 seconds).

Comment 9

[Andreas Pehrson [:pehrsons]]

(In reply to David Jameson from comment #7)
> I didn't have much luck specifying either "audio/ogg" or 'audio/ogg;
> codecs="opus"', as both gave errors saying it was an invalid format for
> addSourceBuffer. I'm setting the Blob mime type to "audio/ogg", so I don't
> think reading it back will show anything different.

I don't know about addSourceBuffer, but MediaRecorder should accept it. Also, if you don't give MediaRecorder a mime type, it should still spit out blobs with their actual mime type.


(In reply to David Jameson from comment #8)
> When I set that env var, I see Session.Extract much more often than I get
> blobs (> 10 times per second, compared to one blob every 2 seconds).

Thanks, that's something we can look closer at.

Comment 10

[Bulk Bug Changes for mreavy's org]

Mass change P2->P3 to align with new Mozilla triage process.