Open Bug 1171907 Opened 5 years ago Updated 4 years ago

[meta] Multiple issues with HTMLMediaElement::CaptureStream streams over WebRTC

Categories

(Core :: WebRTC: Audio/Video, defect)

41 Branch
defect
Not set

Tracking

()

REOPENED
Tracking Status
firefox41 --- affected

People

(Reporter: pehrsons, Unassigned)

References

Details

(Keywords: meta)

1. Streaming a big buck bunny mp4 on Mac doesn't work because we get MAC_IO_SURFACE frames but MediaPipelineTransmit only approves of PLANAR_YCBCR and CAIRO_SURFACE.
- IMO we should treat all Images except PLANAR_YCBCR equal and try to read out a DataSourceSurface from them. If that works out, we'll only bail if we can't deal with the SurfaceFormat.

2. Seeking in the original HTMLMediaElement ONCE makes the remote video pause. Seeking again makes it play again.
- This is because on seek, MediaDecoder replaces the SourceMediaStream piped into the TrackUnionStream currently backed by the DOMMediaStream we are accessing in JS code. When that happens, the tracks in the underlying StreamBuffer of said TrackUnionStream live on a little longer so the tracks coming from the new SourceMediaStream get assigned different values.
- The optimal fix would be to have the MediaDecoder feed us data without recreating the source stream.

3. When doing captureStream() on a preloaded ("loadedmetadata" has already occurred) media element (I was doing it in the "playing" handler), playback is halted altogether on said preloaded element.
- Not sure why. Probably something about clocks. Will try to recreate in a mochitest.

4. When I try to play() a captured media element streaming over a PeerConnection that has ended (stream has not ended), we crash here:
>   * frame #0: 0x0000000101f58a1d XUL`mozilla::WebrtcAudioConduit::ValidateCodecConfig(mozilla::AudioCodecConfig const*, bool) const [inlined] std::string::size() const + 4 at basic_string.h:606
>     frame #1: 0x0000000101f58a19 XUL`mozilla::WebrtcAudioConduit::ValidateCodecConfig(mozilla::AudioCodecConfig const*, bool) const [inlined] std::string::empty() const at basic_string.h:686
>     frame #2: 0x0000000101f58a19 XUL`mozilla::WebrtcAudioConduit::ValidateCodecConfig(this=0x000000012b9fa330, codecInfo=0x000000012c684000, send=true) const + 25 at AudioConduit.cpp:1031
>     frame #3: 0x0000000101f58595 XUL`mozilla::WebrtcAudioConduit::ConfigureSendMediaCodec(this=0x000000012b9fa330, codecConfig=0x000000012c684000) + 101 at AudioConduit.cpp:364
>     frame #4: 0x0000000101f6d6f3 XUL`mozilla::MediaPipelineTransmit::PipelineListener::ProcessVideoChunk(this=<unavailable>, conduit=<unavailable>, chunk=<unavailable>) + 3107 at MediaPipeline.cpp:1229
>     frame #5: 0x0000000101f6bc4d XUL`mozilla::MediaPipelineTransmit::PipelineListener::NewData(this=0x0000000126733f20, graph=<unavailable>, tid=<unavailable>, offset=<unavailable>, events=<unavailable>, media=0x000000011eecb460) + 989 at MediaPipeline.cpp:958
>     frame #6: 0x0000000101f6c028 XUL`mozilla::MediaPipelineTransmit::PipelineListener::NotifyQueuedTrackChanges(this=<unavailable>, graph=<unavailable>, tid=<unavailable>, offset=<unavailable>, events=<unavailable>, queued_media=<unavailable>) + 808 at MediaPipeline.cpp:894
>     frame #7: 0x000000010353004a XUL`mozilla::TrackUnionStream::CopyTrackData(this=0x000000011f6f6300, aInputTrack=0x00000001152f3ec0, aMapIndex=<unavailable>, aFrom=<unavailable>, aTo=649728, aOutputTrackFinished=0x0000000137b6e87f) + 714 at TrackUnionStream.cpp:286


I'll try to get some manual test pages or mochitests demonstrating this behavior up soon.
(In reply to Andreas Pehrson [:pehrsons] (Telenor) from comment #0)
> 2. Seeking in the original HTMLMediaElement ONCE makes the remote video
> pause. Seeking again makes it play again.
> - This is because on seek, MediaDecoder replaces the SourceMediaStream piped
> into the TrackUnionStream currently backed by the DOMMediaStream we are
> accessing in JS code. When that happens, the tracks in the underlying
> StreamBuffer of said TrackUnionStream live on a little longer so the tracks
> coming from the new SourceMediaStream get assigned different values.

They get assigned different TrackIDs, I mean. You typically get 1 and 2 after play(), then 3 and 4 after first seek, 1 and 2 after seconds seek, and so on. If you're a really fast seeker you can get 5 and 6! ;-)
Depends on: 1172394
Depends on: 1172395
Depends on: CVE-2015-2732
(In reply to Andreas Pehrson [:pehrsons] (Telenor) from comment #0)
> 3. When doing captureStream() on a preloaded ("loadedmetadata" has already
> occurred) media element (I was doing it in the "playing" handler), playback
> is halted altogether on said preloaded element.
> - Not sure why. Probably something about clocks. Will try to recreate in a
> mochitest.

This seems to have been me confusing test cases. The real cause seems to be media from another origin. See bug 1172395.
Keywords: meta
Summary: Multiple issues with HTMLMediaElement::CaptureStream streams over WebRTC → [meta] Multiple issues with HTMLMediaElement::CaptureStream streams over WebRTC
Issue 1 in comment 0 is being fixed by bug 1169125.
Depends on: 1169125
These are now all fixed. Issue 2's seeking will keep creating new tracks, and this will have to be handled in js with sender.replaceTrack or something similar.
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Bug 1172394 was reopened so I'm reopening this to keep track of media element capture.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
You need to log in before you can comment on or make changes to this bug.