1181981 - mozGetMediaSource doesn't work on youtube videos

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, defect, P2)

Description

[Paul Rouget [:paul]]

+++ This bug was initially created as a clone of Bug #1177793 +++

Using mozCaptureStream() on any youtube video will freeze/pause the video.

It can be due to the fact videos on youtube use a mediasource instead of a file.

Some more context can be found in bug 1177793.

Comment 1

[Andreas Pehrson [:pehrsons]]

Attachment: mozCaptureStream() on Youtube with MSE and hardcoded track IDs

I did some testing of this with my current WIP patch in bug 1172394 applied (uploading it soon).

One obvious problem was that the audio track and the video track were added by the MediaDecoderStateMachine with the same ID.

With my patch (pull based feeding of the stream) it caused some assertions.

Without my patch (push based feeding of the stream) it's probably causing a freeze because not all added tracks have data. They can't all ever get data either, since you add data per ID. I'm guessing you're on a prod build because there should really be assertions going off if we for instance try to push video data to an audio track.


I hardcoded some track IDs with my patch applied and it just worked! See the screenshot.

Comment 2

[Paul Rouget [:paul]]

(In reply to Andreas Pehrson [:pehrsons] (Telenor) from comment #1)
> Without my patch (push based feeding of the stream) it's probably causing a
> freeze because not all added tracks have data. They can't all ever get data
> either, since you add data per ID. I'm guessing you're on a prod build
> because there should really be assertions going off if we for instance try
> to push video data to an audio track.

See bug 1177793 comment 2.

> I hardcoded some track IDs with my patch applied and it just worked! See the
> screenshot.

Great! I'm surprised you don't have the black screen issue.

Should this bug be a dupe of bug 1172394?

Any ETA for a fix? (July?)

Thanks for looking at this.

Comment 3

[Andreas Pehrson [:pehrsons]]

(In reply to Paul Rouget [:paul] from comment #2)
> Should this bug be a dupe of bug 1172394?
No, I expect this to work once we fix the duplicate track ids.

> Any ETA for a fix? (July?)
If the source of the duped track ids is simple to fix it should be in sometime next week. We did some debugging and it was indeed coming from MediaSourceReader (=MSE), not sure where the exact origin is yet though.

Comment 4

[Andreas Pehrson [:pehrsons]]

It's coming from here: https://hg.mozilla.org/mozilla-central/file/e7e69cc8c07b/media/libstagefright/binding/DecoderData.cpp#l115

My understanding of this issue is that audio and video come as separate chunks with MSE and so they can have the same track IDs. However, both HTMLMediaElement and DecodedStream rely on the track IDs to be unique for mozCaptureStream() to work.

I guess we can have MediaDecoderStateMachine assign unique output trackIDs when Metadata gets read or perhaps even rewrite to make them unique. Unless, of course, we can fix it at the source.

jya, can you take care of this? Or tell me your preferred solution and I can implement it. Some MSE-person should write a test for this as well IMHO.

Comment 5

[Jean-Yves Avenard [:jya]]

This should be fixed with the new MSE btw, as the track ID are properly set.

Could you try setting media.mediasource.format-reader to true and see how that goes for you?

Comment 6

[Andreas Pehrson [:pehrsons]]

Attachment: Debug patch

(In reply to Jean-Yves Avenard [:jya] from comment #5)
> This should be fixed with the new MSE btw, as the track ID are properly set.
> 
> Could you try setting media.mediasource.format-reader to true and see how
> that goes for you?

Didn't do much. See the attached patch to better understand what is failing. This is the output with that patch trying to play (no need to capture to a stream) a video on youtube:
> MP4VideoInfo track ID 2 => 1
> MP4VideoInfo track ID 1 => 1
> Assertion failure: !(HasAudio() && HasVideo()) || mInfo.mAudio.mTrackId != mInfo.mVideo.mTrackId (Duplicate track ids in MediaInfo), at /Users/pehrsons/Comoyo/gecko-dev/dom/media/MediaDecoderStateMachine.cpp:1893


I'm on self-built m-c from July 8.

Comment 7

[Paul Rouget [:paul]]

Any update on this? Anything I can do to help?

Comment 8

[Ralph Giles (:rillian)]

Maybe this doesn't work with MSE video?

Comment 9

[Ralph Giles (:rillian)]

This is supposed to be fixed, apparently. Can you confirm please?

Comment 10

[Paul Rouget [:paul]]

(In reply to Ralph Giles (:rillian) from comment #9)
> This is supposed to be fixed, apparently. Can you confirm please?

It's not.

Open any youtube video, open the js console, and type:

```
let v = document.querySelector('video');
let stream = v.mozCaptureStream();
```

Video freezes.

Comment 11

[Jean-Yves Avenard [:jya]]

Attachment: [MSE[ Force TrackId for MSE tracks.

YouTube sets the track ID at 1 for both audio and video tracks. Attempting to use mozCaptureStream would cause an assert as it always expect tracks to have a different track Id.

Comment 12

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/be37af7fca76

Comment 13

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/be37af7fca76