806426 - getUserMedia stream frozen if played before being inserted into the DOM

Status: RESOLVED WORKSFORME

(Core :: Audio/Video, defect, P2)

Description

[Guyllaume Doyer]

Attachment: After DOM Insertion

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.40 Safari/537.11

Steps to reproduce:

This may not be a bug however Chrome and Opera works as expected

1) Access the cam with getUsedMedia, attach it to a new video tag (not already in the DOM)
2) call the play() method BEFORE inserting it the element into the DOM
3) Insert the element into the DOM


Actual results:

The video is frozen (maybe on the 1st frame captured).


Expected results:

The video should be playing whitout being frozen. This works correctly in Chrome and Opera.

The proble does not occur when the play() is performed AFTER having inserted the element into the DOM.

Comment 1

[Jason Smith [:jsmith]]

Attachment: Before DOM Insertion

Comment 2

[Jason Smith [:jsmith]]

Confirmed on 10/31 build.

Comment 3

[Randell Jesup [:jesup] (needinfo me)]

roc: the testcase (the second attachment) causes MediaStreamGraph to block after < ~ 0.010s, and it remains blocked.  My guess is that when the video element is added to the DOM it doesn't unblock.

The other issue is that it continues to pull frames and buffer them.  If it does unblock and start playing, it will be N seconds behind and remain so, AND more critically it ramps up memory use at a frightening rate (buffering unlimited media in memory).

To reprise other conversations, realtime sources should toss frames if they're not wanted *now*, and buffered-playback streams should block the source decode after some small amount of buffering of media.  GetUserMedia already handles part of this: as new frames come in, it throws away the previous frame, and only sends a copy to the stream on NotifyPull.  However, this depends on NotifyPull stopping when we're blocked.  Or perhaps we can be notified of Blocking changes and stop updating the "current" picture while blocked, and keep sending the same image to NotifyPull (which hopefully would not make a new copy).  Downside here is that you may have 1 or more frames of "old" video at the start of playback, which will be annoying.

Comment 4

[Randell Jesup [:jesup] (needinfo me)]

Critical due to the memory consumption issue

Comment 5

[Randell Jesup [:jesup] (needinfo me)]

Attachment: Throw away audio/video when MediaStream is blocked

close-to-complete patch to mitigate the memory spike when an sink element is blocked by throwing away input data without feeding it to MediaStreamGraph.  Does not solve the 'video element is still blocked after adding to the DOM', and doesn't address any buffering issues, but may work as a stopgap to avoid OOM (and delay when a stream does start).

Comment 6

[Randell Jesup [:jesup] (needinfo me)]

Comment on attachment 677302 [details] [diff] [review]
Throw away audio/video when MediaStream is blocked

roc: there are some bugs in this patch especially around first-time use;  Are we going to resolve the larger issue, or video-element-is-blocked issue?  For either resolving neither, or if we only resolve the blocked video element, we'll need to use this stopgap to avoid OOM.  If we solve the more general problem within MediaStreamGraph itself, then this patch is moot.

Comment 7

[Robert O'Callahan (:roc) (email my personal email if necessary)]

A relatively easy way to solve this is to never allow the SourceMediaStream to block. Wrap it in a TrackUnionStream with FLAG_BLOCK_OUTPUT but not FLAG_BLOCK_INPUT, and give that stream out instead. (Sorry the documentation for these flags in MediaStreamGraph.h is wrong!)

The downside of that approach is that it'll make it harder to improve the latency for WebRTC (though it won't make latency worse than it is now). But I still think it's the right thing to do.

Comment 8

[Randell Jesup [:jesup] (needinfo me)]

The memory issue appears to be resolved.

I assume therefore that when we unblock we would not be "behind", though we need to make sure when we do so we flush any "old" frames in the graph, if there are any.  (May be ok.)

We still have a problem with the video element not unblocking when it's added to the DOM.  I haven't read the spec, but certainly it's a parity issue even if it's a gray area of the spec.

To really see the problem, run the first test ("After DOM Insertion"), click the button to turn on the camera, go Back, then try the second test.

No longer a GetUserMedia issue per-se I think; moving to Audio/Video

Comment 9

[Randell Jesup [:jesup] (needinfo me)]

Spoke too soon - the video changes (NotifyPull-based with grabbing the "latest" frame) solved it there, but it still exists for audio (which is push-based off the audio input callbacks - unlike video skipping frames is horrible in audio, while merely annoying/bad in video).

So, this means a paused output will buffer "forever" chewing ram, and unpausing will lead to an N second-delayed playback.

I can wrap it in a TrackUnion stream for now, but we cannot live with an extra 30ms delay forever on inputs; we'll need to either resolve the issue in MediaStream with TrackUnion latency, or back out any TrackUnion change and resolve the issue more directly in MediaStream.

(I suspect solving it without TrackUnion is preferable, but as we need to control latency in other cases where we *will* need to combine streams (such as audio and video), it may still be valid.)

Comment 11

[Randell Jesup [:jesup] (needinfo me)]

Attachment: convert to TrackUnionStreams for getUserMedia (and add hint support)

WIP patch, still buffers

Comment 12

[Randell Jesup [:jesup] (needinfo me)]

Spun off bug 822956 for the buffering issue; downgrading remaining issue to blocking- for getUserMedia