Large lag when playing video from MediaRecorder via MediaSource

NEW
Unassigned

Status

()

Core
Audio/Video: Recording
P3
normal
Rank:
25
a year ago
5 months ago

People

(Reporter: David Jameson, Unassigned)

Tracking

54 Branch
x86_64
Mac OS X
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

a year ago
User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36

Steps to reproduce:

1. Go to https://www.groupboard.com/gum_record/
2. Click record
3. Look at the video in the second window (which is played back using MediaSource).


Actual results:

Firefox 54.0a1 has a 4 second lag.
Firefox 51.0.1 has a 2 second lag.
Chrome canary has < 1 second lag.


Expected results:

Ideally lag should be minimal, similar to Chrome.
(Reporter)

Updated

a year ago
OS: Unspecified → Mac OS X
Hardware: Unspecified → x86_64

Updated

a year ago
Component: Untriaged → WebRTC: Audio/Video
Product: Firefox → Core
I can't get https://www.groupboard.com/gum_record/ to work at all on 51, 53, or 54. Lots of JS errors.
Flags: needinfo?(davidj)
(Reporter)

Comment 2

a year ago
Sorry, forgot to mention you'll need to set media.mediasource.webm.enabled to true in about:config.
Lag is pretty bad for me too, yeah.
Status: UNCONFIRMED → NEW
Ever confirmed: true
I will note that there's no requirement that MediaRecorder streams when immediately played back be low-latency, though certainly there's advantage to it and not having it be *too* high). Don't assume it's realtime like webrtc; if you're just playing it back somewhere and want realtime, use webrtc.

Also, is the latency as bad when doing audio-only recording?
Rank: 25
Component: WebRTC: Audio/Video → Audio/Video: Recording
Priority: -- → P2
(Reporter)

Comment 5

a year ago
Any idea what parameters to use for audio only? It says "video format or mime type is not supported" when I tell webrtc to use audio only and set the mime types to 'video/webm; codecs="opus"':

https://www.groupboard.com/gum_record/index2.html

The problem seems to be in MediaRecorder: you can see from the console.logs that there is a delay of a few seconds before ondataavailable is called, whereas on Chrome it's being called about 10 times per second (as we request) right from the beginning. I guess it is just the vp8 encoding parameters, which we don't have access to via MediaRecorder. It would be nice if there was a "realtime" flag we could pass into the options to the MediaRecorder constructor.
On my machine we're getting 2 second blobs.

We are not violating the spec which says this:
> If timeslice is not undefined, then once a minimum of timeslice milliseconds of data have been collected, or some minimum time slice imposed by the UA, whichever is greater, start gathering data into a new Blob blob, and queue a task, using the DOM manipulation task source, that fires a blob event named dataavailable at target with blob.


(In reply to David Jameson from comment #5)
> Any idea what parameters to use for audio only? It says "video format or
> mime type is not supported" when I tell webrtc to use audio only and set the
> mime types to 'video/webm; codecs="opus"':

Skip the mime type altogether and just pass a MediaStream with one audio track. The blob will tell you the mime type. I think there's a way to figure it out on the MediaRecorder object too. If you really want to pass a mime type, try 'audio/ogg; codecs="opus"'.

---

If you want to go deeper; start firefox with the following env var set: `MOZ_LOG=timestamp,sync,MediaRecorder:4`
Our code is not gonna output blobs faster than you see logs on stderr containing "Session.Extract ...".
This is run on a dedicated thread as often as possible (bound by the encoders).
If you see "Session.Extract" more often than you get blobs, we can look into it.

Future refactors may improve things.
(Reporter)

Comment 7

a year ago
Agreed, Firefox isn't violating the spec. However it would certainly be more useful to call ondataavailable every 0.1s if that is what we request, rather than a minimum of every 2 seconds. Chrome gives much smaller Blobs (presumably closer to the 0.1s that we request).

I didn't have much luck specifying either "audio/ogg" or 'audio/ogg; codecs="opus"', as both gave errors saying it was an invalid format for addSourceBuffer. I'm setting the Blob mime type to "audio/ogg", so I don't think reading it back will show anything different.

I'll check out the stderr logs.
(Reporter)

Comment 8

a year ago
When I set that env var, I see Session.Extract much more often than I get blobs (> 10 times per second, compared to one blob every 2 seconds).
(In reply to David Jameson from comment #7)
> I didn't have much luck specifying either "audio/ogg" or 'audio/ogg;
> codecs="opus"', as both gave errors saying it was an invalid format for
> addSourceBuffer. I'm setting the Blob mime type to "audio/ogg", so I don't
> think reading it back will show anything different.

I don't know about addSourceBuffer, but MediaRecorder should accept it. Also, if you don't give MediaRecorder a mime type, it should still spit out blobs with their actual mime type.


(In reply to David Jameson from comment #8)
> When I set that env var, I see Session.Extract much more often than I get
> blobs (> 10 times per second, compared to one blob every 2 seconds).

Thanks, that's something we can look closer at.
Flags: needinfo?(davidj)
Mass change P2->P3 to align with new Mozilla triage process.
Priority: P2 → P3
You need to log in before you can comment on or make changes to this bug.