viblast.com/demo-hls HLS test using MSE does not play video

RESOLVED FIXED in Firefox 42

Status

()

P1
normal
RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: cpeterson, Assigned: jya)

Tracking

(Blocks: 1 bug)

unspecified
mozilla42
x86
macOS
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox39 affected, firefox40 affected, firefox41 affected, firefox42 fixed)

Details

Attachments

(3 attachments)

(Reporter)

Description

4 years ago
+++ This bug was initially created as a clone of Bug #1118533 +++

In bug 1118533, nikolay.dimitrov@viblast.com reports that Viblast's HLS tests using MSE doesn't work in Firefox. He has tried with the current Firefox Nightly 42, latest stable 39 and latest beta 40 and media.mediasource.format-reader=true.

http://viblast.com/demo-hls/

Here is a different stream that still doesn't work. You have to click somewhere in the page to start the video.

http://viblast.com/embed.html?cdn-stream=http://cdn2.viblast.com/streams/hls_vod/playlist.m3u8

Comment 1

4 years ago
Thank you for filling this bug, Chris.

I want to provide more information:
Viblast uses WebRTC to deliver P2P video. It allows playback of HLS and DASH streams using MSE. In order to do so we transmux the video from MPEG-TS to fMP4 in JavaScript (for HLS streams).

We want to make our solution compatible with Firefox, but sadly we have to use Flash fallback currently. 

I've tested with the following streams:
1) http://viblast.com/embed.html?cdn-stream=http://cdn2.viblast.com/streams/hls_vod/playlist.m3u8
2) http://viblast.com/embed.html?cdn-stream=http://cdn2.viblast.com/streams/hls_vod_adaptive/playlist.m3u8
3) http://viblast.com/embed.html?cdn-stream=http://cdn2.viblast.com/streams/hls_vod_planes/playlist.m3u8
4) http://viblast.com/embed.html?cdn-stream=http://cdn2.viblast.com/streams/big_buck_bunny/adaptive/playlist.m3u8
5) http://viblast.com/embed.html?cdn-stream=http://dash.edgesuite.net/dash264/TestCases/3b/sony/SNE_DASH_CASE3B_SD_REVISED.mpd
6) http://viblast.com/embed.html?cdn-stream=http://live.unified-streaming.com/loop/loop.isml/loop.mpd?format=mp4


1) is HLS Video-on-Demand playlist. 
2) is the same as 1, but has different qualities (for adaptive playback)
3) is a different HLS Video-on-Demand playlist. It doesn't have audio.
4) is the same as 3, but it is looped by our servers to look like a live streaming playlist. Viblast will auto correct the timestamps before feeding them to MSE, so the timestamps of the first chunk will start from 0.
5) This is a DASH playlist, VoD. It has separate video and audio and it is the only one that works with Firefox.
6) This is a DASH Live playlist. Doesn't work in Firefox, most probably because the timestamps are not corrected and they do not start from 0. Chrome handles this by using timestampOffset. Note: This stream sometimes only plays for 10-20 seconds (not a Firefox problem, the server is not working correctly all the time).

You can try with different HLS/DASH playlists by changing the cdn-stream variable in this url:
http://viblast.com/embed.html?cdn-stream=http://cdn2.viblast.com/streams/hls_vod/playlist.m3u8
(Reporter)

Updated

4 years ago
status-firefox39: --- → affected
status-firefox40: --- → affected
status-firefox41: --- → affected
status-firefox42: --- → affected
Priority: P5 → P1
(Reporter)

Comment 2

4 years ago
Jean-Yves, can you please take a look at this MSE problem? I see the Viblast player download TS data, but the player timeline doesn't advance and no audio plays.

http://cdn2.viblast.com/streams/hls_vod/playlist1.ts
http://cdn2.viblast.com/streams/hls_vod/playlist2.ts
http://cdn2.viblast.com/streams/hls_vod/playlist3.ts
...
Flags: needinfo?(jyavenard)
(Assignee)

Comment 3

4 years ago
In example 1):
The esds atom in the init segment is invalid.

The ESDescriptor is marked to have a size of 27 bytes, but the DecoderConfig it contains is also marked as being 27 bytes. That can't be, it must be at least 4 bytes shorter (as the first 2 bytes is the es_id and then stream_priority (also 2 bytes).

We strictly enforce size restriction, there's been a lot of security sensitive flaw in stagefright. and now anything with invalid atom size is rejected.

Now, of course we could just totally ignore the ESDescriptor field, like what ffmpeg down and as such chrome (which using ffmpeg).
Flags: needinfo?(jyavenard)
(Assignee)

Comment 4

4 years ago
In the init segment, set the size from 25 to 17 (so it doesn't include the Descriptor, and it will all play.

Additional note. You should check the events generated by the sourcebuffer ; per spec, the "error" event is fired on the sourcebuffer upon completion of appendBuffer as the input data is invalid. Yet, your player continues to append data as if nothing was wrong.

Stream 2, causes an assert with the new MSE, as the init segment found has an invalid size, need to look into handling this case (though it has the same issue and have an invalid init segment)

Stream 3: same invalid init segment.

(In reply to nikolay.dimitrov from comment #1)
> 6) This is a DASH Live playlist. Doesn't work in Firefox, most probably
> because the timestamps are not corrected and they do not start from 0.
> Chrome handles this by using timestampOffset. Note: This stream sometimes
> only plays for 10-20 seconds (not a Firefox problem, the server is not
> working correctly all the time).

We do not correct timestamp, however this has nothing to do with the use of timestampOffset which can only be handled by the JS.
Chrome has an incorrect behaviour is that it always assumes the first segment timestamp will be 0, and this can lead to invalid A/V sync as video is then shifted incorrectly.

But having said that, this isn't the problem here.
Having a mediasegment that doesn't start at 0 is fine. It's only gaps in the media that can be problematic. We skip over gaps up to 125ms in duration. Anything more than that and you'll get a stall (with the related "waiting" event fired).
(Assignee)

Updated

4 years ago
Assignee: nobody → jyavenard
(Assignee)

Updated

4 years ago
See Also: → bug 1185179
(Reporter)

Updated

4 years ago
Depends on: 1185179

Comment 5

4 years ago
Hi Jean-Yves,

Svetlin from Viblast here. Thank you so much for your reply it really helped us a lot. We fixed our converter and now the ESDS boxes are correct and FF plays the streams.

However when we did more testing we encountered other problems and inconsistent behavior:
 * The audio for stream 1) works just fine on the 64-bit version of FF 42 for windows. But It doesn't work on the 32-bit version or on the version for OS X.
 * Changing from one quality to another works for some streams (http://viblast.com/demo-hls/) smoothly. But for others (http://viblast.com/demo-dash/) either doesn't work at all or the video tag flickers and the loading spinner is displayed for a short while.
 * Example 6) still doesn't work and there is nothing on the firefox console. This is actually one of the dash-if (http://dashif.org/) test vectors so it would be nice to know if the problem is in the decoder or the encoder.

HTML5 playback is very important to us and we would be happy to change our tools to produce streams that work with FF.
(Reporter)

Comment 6

4 years ago
(In reply to svetlin.mladenov from comment #5)
> Svetlin from Viblast here. Thank you so much for your reply it really helped
> us a lot. We fixed our converter and now the ESDS boxes are correct and FF
> plays the streams.
> 
> However when we did more testing we encountered other problems and
> inconsistent behavior:
>  * The audio for stream 1) works just fine on the 64-bit version of FF 42
> for windows. But It doesn't work on the 32-bit version or on the version for
> OS X.

Is stream 1) the link from comment 1 above? The audio plays correctly for me with Firefox 42 on OS X 10.10 and 32-bit Firefox on Windows 8.1 (in a VM).

1) http://viblast.com/embed.html?cdn-stream=http://cdn2.viblast.com/streams/hls_vod/playlist.m3u8


>  * Changing from one quality to another works for some streams
> (http://viblast.com/demo-hls/) smoothly. But for others
> (http://viblast.com/demo-dash/) either doesn't work at all or the video tag
> flickers and the loading spinner is displayed for a short while.

The demo-dash player doesn't appear to switch quality in Firefox or Chrome. Do I need to do anything after changing the Quality drop menu from "Auto" to another value, such as "1920x1080 (HD)"?


>  * Example 6) still doesn't work and there is nothing on the firefox
> console. This is actually one of the dash-if (http://dashif.org/) test
> vectors so it would be nice to know if the problem is in the decoder or the
> encoder.

I can reproduce this problem, but it is not limited to Firefox. Example 6 never starts playing in Firefox 42 or IE 11. The video starts playing in Chrome and Safari, but stalls after about 30 seconds.

Comment 7

4 years ago
Yes, that's the correct stream. We've tested again and there is still no sound on Firefox Nightly on OS X.

About the quality change - The quality switch works with a slight delay because of the buffer we have. To be able to see it, please switch to 320x240, wait about 30-40 seconds until the quality is reduced and then switch to Full HD.


About 6) This is a third-party server and it is not working correctly - the media segments are not being updated. However it should play for 20-30 seconds using the existing segments. This is the case in Chrome and Safari, but doesn't even start in Firefox.
You can see the same stream here:
http://dashif.org/reference/players/javascript/1.4.0/samples/dash-if-reference-player/index.html
(Select "Unified Streaming Live")
(Reporter)

Updated

4 years ago
Blocks: 1187136
(Reporter)

Updated

4 years ago
Blocks: 1187143
(Reporter)

Comment 8

4 years ago
(In reply to nikolay.dimitrov from comment #7)
> Yes, that's the correct stream. We've tested again and there is still no
> sound on Firefox Nightly on OS X.

I filed new bug 1187143 to track this audio issue. However, I am not able to reproduce this bug.


> About the quality change - The quality switch works with a slight delay
> because of the buffer we have. To be able to see it, please switch to
> 320x240, wait about 30-40 seconds until the quality is reduced and then
> switch to Full HD.

I am still unable to reproduce this stream quality problem. In both Chrome and Firefox (on OS X), the quality successfully drops to 320x180 and then jumps to 1600x900. I had to expand the tiny embedded video to full-screen to see the difference in quality. <:)

http://viblast.com/demo-dash/


> About 6) This is a third-party server and it is not working correctly - the
> media segments are not being updated. However it should play for 20-30
> seconds using the existing segments. This is the case in Chrome and Safari,
> but doesn't even start in Firefox.
> You can see the same stream here:
> http://dashif.org/reference/players/javascript/1.4.0/samples/dash-if-
> reference-player/index.html
> (Select "Unified Streaming Live")

I filed new bug 1187136 to track this stream issue.
Blocks: 1185611

Comment 9

4 years ago
>  * Changing from one quality to another works for some streams (http://viblast.com/demo-hls/) smoothly.
> But for others (http://viblast.com/demo-dash/) either doesn't work at all or the video tag flickers
> and the loading spinner is displayed for a short while.

Sorry about that. I made a mistake. http://viblast.com/demo-dash/ is the stream that switches qualities smoothly. http://viblast.com/demo-hls/ is the one that cannot switch qualities and the video just freezes. To reproduce it just go to http://viblast.com/demo-hls/ and wait for 4-5 seconds. Then Viblast will switch qualities and the video will freeze.

Another stream we have problems with is http://viblast.com/embed.html?cdn-stream=http://cdn2.viblast.com/streams/hls_vod_adaptive/playlist.m3u8. Open the page and wait for about 10-20 seconds. Then Viblast will switch qualities. Under 64-bit Firefox Window 7 the quality change is successful but there is flickering. Under 32-bit Firefox Window 7 the video just freezes.

I made some testing and maybe the problems are related to the fact that hls streams use a single buffer (codecs="avc1.42e01e, mp4a.40.2") and Viblast appends the video and audio streams to it.

Comment 10

4 years ago
I just did some more testing and when audio is disabled http://viblast.com/embed.html?cdn-stream=http://cdn2.viblast.com/streams/hls_vod_adaptive/playlist.m3u8 switches between qualities smoothly and correctly. Unfortunately I can't provide you with a public demo.

It indeed seems to be a problem with using just one buffer and multiplexing audio and video into it.
(Reporter)

Comment 11

4 years ago
(In reply to svetlin.mladenov from comment #9)
> Sorry about that. I made a mistake. http://viblast.com/demo-dash/ is the
> stream that switches qualities smoothly. http://viblast.com/demo-hls/ is the
> one that cannot switch qualities and the video just freezes. To reproduce it
> just go to http://viblast.com/demo-hls/ and wait for 4-5 seconds. Then
> Viblast will switch qualities and the video will freeze.

I can reproduce this on both 32-bit and 64-bit Firefox (tested on Windows 8.1 and 10).


> Another stream we have problems with is
> http://viblast.com/embed.html?cdn-stream=http://cdn2.viblast.com/streams/
> hls_vod_adaptive/playlist.m3u8. Open the page and wait for about 10-20
> seconds. Then Viblast will switch qualities. Under 64-bit Firefox Window 7
> the quality change is successful but there is flickering. Under 32-bit
> Firefox Window 7 the video just freezes.

For me, the video stalls on both 32-bit and 64-bit Firefox (tested on Windows 8.1).


> I made some testing and maybe the problems are related to the fact that hls
> streams use a single buffer (codecs="avc1.42e01e, mp4a.40.2") and Viblast
> appends the video and audio streams to it.

Appending audio and video to the same source buffer was a known problem, but I thought it had been fixed..
(Assignee)

Comment 12

4 years ago
(In reply to Chris Peterson [:cpeterson] from comment #11)
> (In reply to svetlin.mladenov from comment #9)
> > Sorry about that. I made a mistake. http://viblast.com/demo-dash/ is the
> > stream that switches qualities smoothly. http://viblast.com/demo-hls/ is the
> > one that cannot switch qualities and the video just freezes. To reproduce it
> > just go to http://viblast.com/demo-hls/ and wait for 4-5 seconds. Then
> > Viblast will switch qualities and the video will freeze.
> 
> I can reproduce this on both 32-bit and 64-bit Firefox (tested on Windows
> 8.1 and 10).
> 
> 
> > Another stream we have problems with is
> > http://viblast.com/embed.html?cdn-stream=http://cdn2.viblast.com/streams/
> > hls_vod_adaptive/playlist.m3u8. Open the page and wait for about 10-20
> > seconds. Then Viblast will switch qualities. Under 64-bit Firefox Window 7
> > the quality change is successful but there is flickering. Under 32-bit
> > Firefox Window 7 the video just freezes.
> 
> For me, the video stalls on both 32-bit and 64-bit Firefox (tested on
> Windows 8.1).
> 
> 
> > I made some testing and maybe the problems are related to the fact that hls
> > streams use a single buffer (codecs="avc1.42e01e, mp4a.40.2") and Viblast
> > appends the video and audio streams to it.
> 
> Appending audio and video to the same source buffer was a known problem, but
> I thought it had been fixed..

It has..

It works.

However, there are issues inherent to how MSE works when using a single sourcebuffer for both audio and video

Depending on how the MP4 was muxed, if you use partial append: it will likely break. And there's nothing we can do about that.

Comment 13

4 years ago
What is "partial append"?

Partial boxes are never appended to the sourcebuffer. The append pattern is as follows:
appendBuffer(ftyp+moov) // the moov box contains 2 tracks - one for audio and one for video
appendBuffer(moof+mdat) // this moof-mdat pair contains video frames
appendBuffer(moof+mdat) // this one contains audio frames
... and so on until a quality switch. At that point:
appendBuffer(ftyp+moov) // the new moov box contains updated info about the news audio/video streams
appendBuffer(moof+mdat) // video
appendBuffer(moof+mdat) // audio
(Assignee)

Comment 14

4 years ago
In this stream I see a demuxing error when attempting to retrieve audio

In the first append buffer, I see 18s worth of video, but we have an error extracting audio after 9.941333s.

So we have a stall after playing 9s of video, as there's no matching audio then.

Will investigate the demuxing error to see what's wrong in the video (or our demuxer).
(Assignee)

Comment 15

4 years ago
Sorry , I misread my logs.

The first append segment is :
moof|mdat|moof|mdat

the first moof|mdat is video only and has [0.083333, 10.083333)
the second moof|mdat is audio only and has [0.000000, 9.941333)

The gap at the end is rather big 142ms ; though we should in theory skip over it as we tolerate up to 150ms.
however our code doesn't consider this last moof+mdat to be complete, this cause havoc to the next append buffer 

Interestingly, that stream make Chrome crashes :) So we're not the only one choking on it.

I'll have a closer look on Monday

Comment 16

4 years ago
Just to make sure that we are on the same page: we're talking about this stream - http://viblast.com/embed.html?cdn-stream=http://cdn2.viblast.com/streams/hls_vod_adaptive/playlist.m3u8, right?

The timestamps as specified in the fragmented-mp4 file are continues. I.e. if the audio ends at 9.941333 then the next moof will have a baseMediaDecodeTime=9.941333 (or the equivalent in the particular timescale).

I can send you the content of the array buffers as they are appended into the source buffer, if that's going to help.
You may find it helpful to install https://github.com/doublec/aboutmedia/blob/master/aboutmedia.xpi and navigating to about:media to see the buffered ranges.
(Assignee)

Updated

4 years ago
Component: Audio/Video → Audio/Video: Playback
(Assignee)

Updated

4 years ago
Depends on: 1188758
(Assignee)

Comment 19

4 years ago
When used in the MP4ContainerParser, the MoofParser set the trackID as 0 ; indicating that all tracks are to be parsed. However it set later the trackID to the first one found, causing to ignore all following tracks.
Attachment #8640477 - Flags: review?(ajones)
(Assignee)

Comment 20

4 years ago
Print the atom's offset, makes it easier to verify that byte ranges are properly calculated.
Attachment #8640478 - Flags: review?(ajones)
Comment on attachment 8640476 [details] [diff] [review]
[MSE] P1. Do not overwise stored init data until known as valid.

Review of attachment 8640476 [details] [diff] [review]:
-----------------------------------------------------------------

Typo in the patch comment: 'overwise' should probably be 'overwrite'?

::: dom/media/mediasource/TrackBuffersManager.h
@@ +157,5 @@
>    // Those are used to parse the incoming input buffer.
>  
>    // Recreate the ContainerParser and only feed it with the previous init
>    // segment found.
> +  void RecreateParser(bool aReuseInitData);

Is the documentation still true, given the new argument?
Attachment #8640476 - Flags: review?(gsquelart) → review+
(Assignee)

Updated

4 years ago
Depends on: 1188887
(Assignee)

Comment 22

4 years ago
The MP4Demuxer was ignoring all moof not containing the first track ID found.
As we typically only dealt with one sourcebuffer per content type (audio/video) it was never a problem. If the fragmented mp4 stream had been muxed so each moof contained both trackID it would have worked fine.

But that particular usage was no good.

All streams play nicely now except the last one:
http://viblast.com/embed.html?cdn-stream=http://live.unified-streaming.com/loop/loop.isml/loop.mpd?format=mp4

What is this ?format=mp4 argument?
This one shows:
JavaScript error: http://portal.viblast.com/static/resources/d59e8b5e9752499f93a8fc12ad44c2cd/viblast.js line 104 > eval, line 1: InvalidStateError: An attempt was made to use an object that is not, or is no longer, usable

The reason for this is that the player attempts to seek when readyState == HAVE_NOTHING.

This appears to be a bug in our HTMLVideoElement implementation. I've opened bug 1188887 about it.


Stream http://viblast.com/embed.html?cdn-stream=http://cdn2.viblast.com/streams/big_buck_bunny/adaptive/playlist.m3u8 is a bit weird. I got some fairly long pause when seeking, and chrome doesn't let me seek at all.
(Assignee)

Comment 23

4 years ago
(In reply to Gerald Squelart [:gerald] from comment #21)
> Comment on attachment 8640476 [details] [diff] [review]
> [MSE] P1. Do not overwise stored init data until known as valid.
> 
> Review of attachment 8640476 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Typo in the patch comment: 'overwise' should probably be 'overwrite'?
> 
> ::: dom/media/mediasource/TrackBuffersManager.h
> @@ +157,5 @@
> >    // Those are used to parse the incoming input buffer.
> >  
> >    // Recreate the ContainerParser and only feed it with the previous init
> >    // segment found.
> > +  void RecreateParser(bool aReuseInitData);
> 
> Is the documentation still true, given the new argument?

thanks for the review.

good points !

Comment 24

4 years ago
When will the new fixes be available in nightly? Presumably tomorrow or the day after tomorrow, right?

The http://viblast.com/embed.html?cdn-stream=http://live.unified-streaming.com/loop/loop.isml/loop.mpd?format=mp4 stream appears to be broken even under Chrome. It appears that the seek to position 0 before the stream starts somehow fixes it for chrome. If this seek is removed then the stream doesn't play. It looks like a really twisted case. Maybe we should notify unified streaming for this issue.

> What is this ?format=mp4 argument?

format=mp4 makes the server use fragmented mp4 as the media container. Video and audio streams are separate.

format=ts makes the server use mpeg-ts as the media container. The video and audio streams are multiplexed into the same stream.
(Assignee)

Comment 25

4 years ago
all changes must be peer reviewed first, and then submitted to inbound repository. Only once they have passed all regression tests will they make it to mozilla-central from which Nightly is build from.
So, I'm hoping that yes by tomorrow night it should be available.

For the http://viblast.com/embed.html?cdn-stream=http://live.unified-streaming.com/loop/loop.isml/loop.mpd?format=mp4, Firefox doesn't support seeking before readyState is >= HAVE_METADATA (at least until bug 1188887 is fixed). Fixing bugs on HTML elements always takes time ; so if you could find a work around in your player it would probably work better.

Like wait on the loadedmetadata event and then seek.
Attachment #8640477 - Flags: review?(ajones) → review+
Attachment #8640478 - Flags: review?(ajones) → review+
https://hg.mozilla.org/mozilla-central/rev/759a1be250b9
https://hg.mozilla.org/mozilla-central/rev/e22318ab0a26
https://hg.mozilla.org/mozilla-central/rev/5e130ad70aa7
Status: NEW → RESOLVED
Last Resolved: 4 years ago
status-firefox42: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla42
Depends on: 1190877
No longer depends on: 1190877
You need to log in before you can comment on or make changes to this bug.