805717 - Remove MediaStream hint mechanism

Status: RESOLVED DUPLICATE of bug 1129263

(Core :: WebRTC, defect, P2)

Description

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Currently we have nsDOMMediaStream::Set/GetHintContents to signal whether a stream will contain audio, video or both. This is a bit ugly and fragile. Really, this information can only be obtained accurately asynchronously, by observing the tracks that appear in a stream.

Consumers of a stream can listen for NotifyQueuedTrackChanges to discover which tracks are available. AFAIK the only thing stopping us from dropping HintContents in favour of NotifyQueuedTrackChanges is that we need to know when multiple tracks have been added as a group.

I considered modifying NotifyQueuedTrackChanges to notify for all changed tracks at once, but that's a bit more code. Instead I suggest waiting for the NotifyHasCurrentData notification to fire with aHasCurrentData true, while remembering which tracks have been seen by NotifyQueuedTrackChanges (counting the number of video and audio tracks created and not destroyed should suffice).

Comment 1

[Randell Jesup [:jesup] (needinfo me)]

Adding ekr and ehugg; this would affect MediaPipeline I believe.

Comment 2

[Eric Rescorla (:ekr)]

I don't really understand how this is supposed to work.

The way that the PeerConnection interface works is that you add some arbitrary number of mediastreams and then you ask for a createoffer. At the time createoffer is called, we need to know exactly what tracks are available. Are you suggesting we have some interlock system that blocks create offer/createanswer until aHasCurrentData is true? What performance guarantees do we have about when that will happen?

Comment 3

[Randell Jesup [:jesup] (needinfo me)]

So one option on the hints thing is to try to make sure all the tracks exist and can be type-interrogated (by initial segment, etc) before returning success from getUserMedia.  This means splitting ::Start() in the sources into a ::AddTracks() and a ::StartCapture().  This would let us ::AddTracks before calling OnSuccess (and ::AddTracks wouldn't block), then do the ::StartCaptures (which can block quite a while).

We can discuss what our options are to "pre-roll" a captured stream from a media element to get the tracks added, and if such a pre-roll needs to be in the JS, in the PeerConnection input, in the captureStreamUntilEnded/MediaStreamGraph code, etc

Comment 4

[Robert O'Callahan (:roc) (email my personal email if necessary)]

(In reply to Eric Rescorla from comment #2)
> The way that the PeerConnection interface works is that you add some
> arbitrary number of mediastreams and then you ask for a createoffer. At the
> time createoffer is called, we need to . Are you suggesting we have some interlock system that blocks
> create offer/createanswer until aHasCurrentData is true? What performance
> guarantees do we have about when that will happen?

I thought you'd told me that the offer was (or could be) created asynchronously after createOffer was called. I guess I misunderstood, sorry.

(In reply to Randell Jesup [:jesup] from comment #3)
> So one option on the hints thing is to try to make sure all the tracks exist
> and can be type-interrogated (by initial segment, etc) before returning
> success from getUserMedia.  This means splitting ::Start() in the sources
> into a ::AddTracks() and a ::StartCapture().  This would let us ::AddTracks
> before calling OnSuccess (and ::AddTracks wouldn't block), then do the
> ::StartCaptures (which can block quite a while).

That would work for some cases, but really it would be better if createOffer could be asynchronous. I don't want to have to make every way you create a MediaStream asynchronous so we can delay returning the MediaStream until it has some tracks in it. We need to be able to handle dynamic track add/remove anyway, so "knowing exactly what tracks are available" isn't really possible (as we discussed before).

Comment 5

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Eric told me on IRC that I had understood him correctly the first time; createOffer is asynchronous. The important thing is to minimize the time required to obtain the track types for the getUserMedia case, since this is on the critical path for "Post-Answer Delay". I think Randell's idea in comment #3 would do that, so I think we're all set.