Incorrect buffered range with site

RESOLVED FIXED in Firefox 47

Status

()

defect
P2
normal
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: jya, Assigned: jya)

Tracking

(Blocks 1 bug)

Trunk
mozilla49
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox46 wontfix, firefox47 fixed, firefox48 fixed, firefox49 fixed)

Details

()

Attachments

(1 attachment)

Assignee

Description

3 years ago
https://people.mozilla.org/~jyavenard/tests/mse_mp4/wowza-stall.html

Buffered range is returned as :

{{ 0,2.668977 }{ 2.999288,3.036 }}

it should be:
{{ 0,2.668977 }{ 2.999288,5.472045 }}

There's a problem with the dts of this file that cause the gap detection to be triggered, to be investigated.

I suspect something to do with the moof parser
Assignee

Comment 1

3 years ago
Actually, our implementation is perfectly up to spec, it's Safari / Chrome that aren't.

We determine the dts exactly like what ffmpeg calculates:

jyambp:wowza-stall jyavenard$ ffprobe -show_entries packet=dts_time,duration_time t.mp4 
ffprobe version 2.8 Copyright (c) 2007-2015 the FFmpeg developers
  built with Apple LLVM version 7.0.2 (clang-700.1.81)
  configuration: --prefix=/opt/local --enable-swscale --enable-avfilter --enable-avresample --enable-libmp3lame --enable-libvorbis --enable-libopus --enable-libtheora --enable-libschroedinger --enable-libopenjpeg --enable-libmodplug --enable-libvpx --enable-libspeex --enable-libass --enable-libbluray --enable-lzma --enable-gnutls --enable-fontconfig --enable-libfreetype --enable-libfribidi --disable-indev=jack --disable-outdev=xv --mandir=/opt/local/share/man --enable-shared --enable-pthreads --cc=/usr/bin/clang --enable-vda --arch=x86_64 --enable-yasm --enable-gpl --enable-postproc --enable-libx264 --enable-libxvid
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 't.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 1
    compatible_brands: isomavc1
    creation_time   : 1970-01-01 00:00:01
  Duration: 00:00:05.47, start: 0.000000, bitrate: 137 kb/s
    Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 135 kb/s, 5.48 fps, 29.97 tbr, 22500 tbn, 10 tbc (default)
    Metadata:
      creation_time   : 1970-01-01 00:00:02
      handler_name    : VideoHandler
      encoder         : dailymotion/hls.js
[PACKET]
dts_time=0.000000
duration_time=N/A
[/PACKET]
[PACKET]
dts_time=0.200000
duration_time=N/A
[/PACKET]
[PACKET]
dts_time=0.233022
duration_time=N/A
[/PACKET]
[PACKET]
dts_time=0.601022
duration_time=N/A
[/PACKET]
[PACKET]
dts_time=0.801022
duration_time=N/A
[/PACKET]
[PACKET]
dts_time=0.834000
duration_time=N/A
[/PACKET]
[PACKET]
dts_time=1.201022
duration_time=N/A
[/PACKET]
[PACKET]
dts_time=1.402000
duration_time=N/A
[/PACKET]
[PACKET]
dts_time=1.435022
duration_time=N/A
[/PACKET]
[PACKET]
dts_time=1.802000
duration_time=N/A
[/PACKET]
[PACKET]
dts_time=2.002000
duration_time=N/A
[/PACKET]
[PACKET]
dts_time=2.035022
duration_time=N/A
[/PACKET]
[PACKET]
dts_time=2.403022
duration_time=N/A
[/PACKET]
[PACKET]
dts_time=2.603022
duration_time=N/A
[/PACKET]
[PACKET]
dts_time=2.636000
duration_time=N/A
[/PACKET]
[PACKET]
dts_time=2.668978
duration_time=N/A
[/PACKET]
[PACKET]
dts_time=3.036000
duration_time=N/A
[/PACKET]
[PACKET]
dts_time=3.203022
duration_time=N/A
[/PACKET]
[PACKET]
dts_time=3.236000
duration_time=N/A
[/PACKET]
[PACKET]
dts_time=3.269022
duration_time=N/A
[/PACKET]
[PACKET]
dts_time=3.804000
duration_time=N/A
[/PACKET]
[PACKET]
dts_time=4.004000
duration_time=0.033333
[/PACKET]
[PACKET]
dts_time=4.037022
duration_time=0.033333
[/PACKET]
[PACKET]
dts_time=4.070000
duration_time=0.033333
[/PACKET]
[PACKET]
dts_time=4.605022
duration_time=0.033333
[/PACKET]
[PACKET]
dts_time=4.805022
duration_time=0.033333
[/PACKET]
[PACKET]
dts_time=4.838000
duration_time=0.033333
[/PACKET]
[PACKET]
dts_time=5.205022
duration_time=0.033333
[/PACKET]
[PACKET]
dts_time=5.406000
duration_time=0.033333
[/PACKET]
[PACKET]
dts_time=5.439022
duration_time=0.033333
[/PACKET]

At end time 3.036, we have a dts of 2.668977, the duration of that sample is 0.036712. The next sample has a dts of 3.036000

the difference between those two dts time is 3.036000-2.668977 = 0.367023s

Per spec:
https://w3c.github.io/media-source/#sourcebuffer-coded-frame-processing
step 6:
"If last decode timestamp for track buffer is set and the difference between decode timestamp and last decode timestamp is greater than 2 times last frame duration:
 [snip]
 5. Set the need random access point flag on all track buffers to true."

that the need random access point is set mean that all frames will be ignored until one is found that is a keyframe as per step 10:
"10. If the need random access point flag on track buffer equals true, then run the following steps:

    1. If the coded frame is not a random access point, then drop the coded frame and jump to the top of the loop to start processing the next coded frame.
    2. Set the need random access point flag on track buffer to false.
"

the difference of 0.367023s is almost 10 times more than the previous duration....

Fix your timestamps, they aren't valid. See in ISOBMFF 14496-12, chapter 8.15.1 for the definition of time to sample boxes:
"The value of DT for a sample is always the sum of the deltas of the preceding samples. Note that the total of the decoding deltas is the duration of the media in this track."

here the sum of the deltas (e.g. the duration) isn't equal to the duration of the media in this track.
duration = dts_next - dts_previous
or: dts_next = dts_previous + duration
whichever way you want to see it.

The dts are bad in the original ts file too, but broken ts in mpeg-ts are not unheard of.

hls.js need to ensure it doesn't produce file with invalid dts.

I don't think we should loosen our MSE implentation simply to get around badly muxed file
Status: NEW → RESOLVED
Closed: 3 years ago
Flags: needinfo?(g.du.pontavice)
Resolution: --- → INVALID
Assignee

Comment 2

3 years ago
the original TS files can be found there:
https://people.mozilla.org/~jyavenard/mediatest/hls/s2/

Two ways to fix this:

1- Fix the TS encoder used to produce the original files
2- Have hls.js rewrite the dts so they are valid.
Assignee

Comment 3

3 years ago
This bug is identical to bug 1268868 ; following bug 1268868, hls.js will be able to detect the gaps and will skip over them normally. But would be better to not have gaps in the first place
See Also: → 1268868
Assignee

Comment 4

3 years ago
Upon reflection, a quick fix would be for samples that do not have a duration, to always use the delta of the 2 dts.

example:
video/avc frame(pts:2999288 end:3036000, dts:2668977, duration:36712, kf:1)
video/avc frame(pts:3036000 end:3203022, dts:3036000, duration:167022, kf:0)
video/avc frame(pts:3203022 end:3236000, dts:3203022, duration:32978, kf:0)

The first frame should have a duration of 167022 (3.036000 - 2.668977 = 0.367023)

Duration aren't that important anyway as only the pts is used during composition.
Assignee

Comment 5

3 years ago
I'll reopen, because the issue is made worse with how we handle the cts/dts in the moof parser.

In order to ensure that all samples in a moof are contiguous, the cts are adjusted so that cts end is always cts / start

but the dts aren't adjusted and as such do not increment by the duration.
Status: RESOLVED → REOPENED
Resolution: INVALID → ---
Assignee

Comment 6

3 years ago
CTS are adjusted so that all frames within a moof are contiguous and gapless. This means that the duration of each sample are updated accordingly. In MP4, the definition of a sample's duration is the delta between two decoding timestamp. As such, when changing the duration, the decode timestamp should be updated accordingly.

Review commit: https://reviewboard.mozilla.org/r/50131/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/50131/
Attachment #8748060 - Flags: review?(ajones)
Assignee

Updated

3 years ago
Assignee: nobody → jyavenard
Assignee

Comment 7

3 years ago
this is the primary cause of several stalls reported by hls.js user and other MSE sites, including bug 1268868
Assignee

Updated

3 years ago
Flags: needinfo?(g.du.pontavice)
Comment on attachment 8748060 [details]
MozReview Request: Bug 1269325: [mp4] Recalculate dts after adjusting cts. r?kentuckyfriedtakahe

https://reviewboard.mozilla.org/r/50131/#review47079

I'm good with this fix but as much as I hate to say it, this change probably deserves a comment to explain why we're doing it.
Attachment #8748060 - Flags: review?(ajones) → review+
Assignee

Comment 9

3 years ago
Comment on attachment 8748060 [details]
MozReview Request: Bug 1269325: [mp4] Recalculate dts after adjusting cts. r?kentuckyfriedtakahe

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/50131/diff/1-2/
Assignee

Comment 10

3 years ago
Comment on attachment 8748060 [details]
MozReview Request: Bug 1269325: [mp4] Recalculate dts after adjusting cts. r?kentuckyfriedtakahe

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/50131/diff/2-3/

Comment 12

3 years ago
@jyavenard@mozilla.com hey I am really thankful for the support given by you,I tested on latest build given by you , Here I could reproduce the issue too :( . 

Here are my findings,this is the link that has ts chunks and chunklist.m3u8 https://drive.google.com/open?id=0BzmeWR8wPIT0VzhpaFVtTFRMZ1E where you can reproduce the issue.

I combined all ts chunks using this command "ffmpeg -i  'concat:media_b400000_318.ts|media_b400000_319.ts|media_b400000_320.ts' -vcodec copy output123.ts "

Then converted  output123.ts to .mp4 using "ffmpeg-3.0-64bit-static/ffmpeg -i output123.ts -vcodec copy output.mp4"

Ran this command to compare the dts_values ./ffprobe -show_entries packet=dts_time,duration_time ~/charles_ts/ff_mse_chunks/output.mp4

./ffprobe -show_entries packet=dts_time,duration_time ~/charles_ts/ff_mse_chunks/output.mp4
ffprobe version 3.0-static http://johnvansickle.com/ffmpeg/  Copyright (c) 2007-2016 the FFmpeg developers
  built with gcc 5.3.1 (Debian 5.3.1-8) 20160205
  configuration: --enable-gpl --enable-version3 --disable-shared --disable-debug --enable-runtime-cpudetect --enable-libmp3lame --enable-libx264 --enable-libx265 --enable-libwebp --enable-libspeex --enable-libvorbis --enable-libvpx --enable-libfreetype --enable-fontconfig --enable-libxvid --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libtheora --enable-libvo-amrwbenc --enable-gray --enable-libopenjpeg --enable-libopus --enable-libass --enable-gnutls --enable-libvidstab --enable-libsoxr --enable-frei0r --enable-libfribidi --disable-indev=sndio --disable-outdev=sndio --enable-librtmp --enable-libmfx --cc=gcc
  libavutil      55. 17.103 / 55. 17.103
  libavcodec     57. 24.102 / 57. 24.102
  libavformat    57. 25.100 / 57. 25.100
  libavdevice    57.  0.101 / 57.  0.101
  libavfilter     6. 31.100 /  6. 31.100
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.101 /  2.  0.101
  libpostproc    54.  0.100 / 54.  0.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/home/denimuser/charles_ts/ff_mse_chunks/output.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.25.100
  Duration: 00:00:09.00, start: 0.000000, bitrate: 437 kb/s
    Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv), 1280x720 [SAR 1:1 DAR 16:9], 436 kb/s, 5 fps, 5 tbr, 90k tbn, 10 tbc (default)
    Metadata:
      handler_name    : VideoHandler
[PACKET]
dts_time=0.000000
duration_time=0.200000
[/PACKET]
[PACKET]
dts_time=0.200000
duration_time=0.219000
[/PACKET]
[PACKET]
dts_time=0.419000
duration_time=0.178000
[/PACKET]
[PACKET]
dts_time=0.597000
duration_time=0.200000
[/PACKET]
[PACKET]
dts_time=0.797000
duration_time=0.200000
[/PACKET]
[PACKET]
dts_time=0.997000
duration_time=0.202000
[/PACKET]
[PACKET]
dts_time=1.199000
duration_time=0.193000
[/PACKET]
[PACKET]
dts_time=1.392000
duration_time=0.200000
[/PACKET]
[PACKET]
dts_time=1.592000
duration_time=0.210000
[/PACKET]
[PACKET]
dts_time=1.802000
duration_time=0.174000
[/PACKET]
[PACKET]
dts_time=1.976000
duration_time=0.200000
[/PACKET]
[PACKET]
dts_time=2.176000
duration_time=0.404000
[/PACKET]
[PACKET]
dts_time=2.580000
duration_time=0.200000
[/PACKET]
[PACKET]
dts_time=2.780000
duration_time=0.012000
[/PACKET]
[PACKET]
dts_time=2.792000
duration_time=0.200000
[/PACKET]
[PACKET]
dts_time=2.992000
duration_time=0.200000
[/PACKET]
[PACKET]
dts_time=3.192000
duration_time=0.216000
[/PACKET]
[PACKET]
dts_time=3.408000
duration_time=0.190000
[/PACKET]
[PACKET]
dts_time=3.598000
duration_time=0.200000
[/PACKET]
[PACKET]
dts_time=3.798000
duration_time=0.213000
[/PACKET]
[PACKET]
dts_time=4.011000
duration_time=0.180000
[/PACKET]
[PACKET]
dts_time=4.191000
duration_time=0.200000
[/PACKET]
[PACKET]
dts_time=4.391000
duration_time=0.243000
[/PACKET]
[PACKET]
dts_time=4.634000
duration_time=0.155000
[/PACKET]
[PACKET]
dts_time=4.789000
duration_time=0.200000
[/PACKET]
[PACKET]
dts_time=4.989000
duration_time=0.200000
[/PACKET]
[PACKET]
dts_time=5.189000
duration_time=0.202000
[/PACKET]
[PACKET]
dts_time=5.391000
duration_time=0.225000
[/PACKET]
[PACKET]
dts_time=5.616000
duration_time=0.200000
[/PACKET]
[PACKET]
dts_time=5.816000
duration_time=0.193000
[/PACKET]
[PACKET]
dts_time=6.009000
duration_time=0.222000
[/PACKET]
[PACKET]
dts_time=6.231000
duration_time=0.200000
[/PACKET]
[PACKET]
dts_time=6.431000
duration_time=0.153000
[/PACKET]
[PACKET]
dts_time=6.584000
duration_time=0.217000
[/PACKET]
[PACKET]
dts_time=6.801000
duration_time=0.200000
[/PACKET]
[PACKET]
dts_time=7.001000
duration_time=0.200000
[/PACKET]
[PACKET]
dts_time=7.201000
duration_time=0.178000
[/PACKET]
[PACKET]
dts_time=7.379000
duration_time=0.210000
[/PACKET]
[PACKET]
dts_time=7.589000
duration_time=0.200000
[/PACKET]
[PACKET]
dts_time=7.789000
duration_time=0.223000
[/PACKET]
[PACKET]
dts_time=8.012000
duration_time=0.231000
[/PACKET]
[PACKET]
dts_time=8.243000
duration_time=0.200000
[/PACKET]
[PACKET]
dts_time=8.443000
duration_time=0.200000
[/PACKET]
[PACKET]
dts_time=8.643000
duration_time=0.156000
[/PACKET]
[PACKET]
dts_time=8.799000
duration_time=0.200000

Comment 13

3 years ago
This is the link where I compared all the duration values https://docs.google.com/spreadsheets/d/1kzHXJRCMSMFHstzcJHi8DF_U7UkdPp6IS_aTTL-_noo/edit?usp=sharing did not notice the huge difference between DTS_values more than duration.

Please let me know if I am missing something here , I did not understand why i am not able to play the stream.

Thanks for help :)
Assignee

Comment 14

3 years ago
you have remuxed your files, it has nothing to do with what hls.js as muxed and is sending via MSE to firefox

Comment 15

3 years ago
With those ts chunks in this link https://drive.google.com/open?id=0BzmeWR8wPIT0VzhpaFVtTFRMZ1E I am able to repro the issue , I combined them into mp4 to see the dts_values variation, would remuxing change the timestamps ?

Comment 16

3 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/61091afacd23
Status: REOPENED → RESOLVED
Closed: 3 years ago3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla49
Assignee

Comment 17

3 years ago
(In reply to suhasreddy.1993 from comment #15)
> With those ts chunks in this link
> https://drive.google.com/open?id=0BzmeWR8wPIT0VzhpaFVtTFRMZ1E I am able to
> repro the issue , I combined them into mp4 to see the dts_values variation,
> would remuxing change the timestamps ?

it would.

But in any case, you're probably not using a version with the fix.
that stream above doesn't stall for me anymore.
Streams start showing 10:27:24, plays for 9s (which display 10:27:33 at the time).

FWIW, Safari will stall at 2s
And chrome will only play 8s of it (and the last frame displayed is 10:27:32), it also takes a long time to start (>15s) even with the files being local.

so it wasn't Firefox that has issue with the timestamps generated by hls.js

The ultimate fix would be for hls.js to change how the timestamps are generated.
Assignee

Comment 19

3 years ago
Comment on attachment 8748060 [details]
MozReview Request: Bug 1269325: [mp4] Recalculate dts after adjusting cts. r?kentuckyfriedtakahe

Approval Request Comment
[Feature/regressing bug #]: 1269325
[User impact if declined]: playback stall.
[Describe test coverage new/current, TreeHerder]: In central, lots of mochitest, manually tested
[Risks and why]: Low. They aren't very common mp4 to start with and the issue only affects MSE. The way to bug was crafted, it can't regress existing working file
[String/UUID change made/needed]: None
Attachment #8748060 - Flags: approval-mozilla-beta?
Attachment #8748060 - Flags: approval-mozilla-aurora?
Assignee

Comment 20

3 years ago
Verified as fixed by users in https://github.com/dailymotion/hls.js/issues/333
Comment on attachment 8748060 [details]
MozReview Request: Bug 1269325: [mp4] Recalculate dts after adjusting cts. r?kentuckyfriedtakahe

While this is a valid issue, if it isn't such a common scenario, let's uplift only to Aurora and let it ride the trains from there. Aurora48+, Beta47-
Attachment #8748060 - Flags: approval-mozilla-beta?
Attachment #8748060 - Flags: approval-mozilla-beta-
Attachment #8748060 - Flags: approval-mozilla-aurora?
Attachment #8748060 - Flags: approval-mozilla-aurora+
Depends on: 1271847
Assignee

Comment 23

3 years ago
Ritu, I've been made aware that this bug is what's causing TED videos to not switch to HTML5 with Firefox and instead to force them to use Flash.

Seeing that flash is a nasty beast we must get rid off, it may be worth reconsidering the issue.

Having said that we will want bug 1271847 too.
Flags: needinfo?(rkothari)
(In reply to Jean-Yves Avenard [:jya] from comment #23)
> Ritu, I've been made aware that this bug is what's causing TED videos to not
> switch to HTML5 with Firefox and instead to force them to use Flash.
> 
> Seeing that flash is a nasty beast we must get rid off, it may be worth
> reconsidering the issue.
> 
> Having said that we will want bug 1271847 too.

Ok. I'll take your word for it. :) I've approved bug 1271847 for uplift to Aurora48. Does it also impact Beta47? If yes, please nominate a patch for beta on that bug.
Flags: needinfo?(rkothari)
Comment on attachment 8748060 [details]
MozReview Request: Bug 1269325: [mp4] Recalculate dts after adjusting cts. r?kentuckyfriedtakahe

I've been told this impacts playback of some TED videos to fallback to Flash instead of using an HTML5 player. Let's uplift to Beta.
Attachment #8748060 - Flags: approval-mozilla-beta- → approval-mozilla-beta+
(Whoever backports this to beta: please backport this *together with* bug 1271847's patch, once that bug has beta uplift approval.  Otherwise, by backporting this bug on its own, we'll be (temporarily) introducing bug 1271847 on beta.)
Assignee

Updated

3 years ago
Depends on: 1232313
Assignee

Comment 27

3 years ago
bug 1232313 needs to be uplifted first before this one can go in.
Assignee

Updated

3 years ago
Blocks: 1272916
No longer blocks: 1278005
Depends on: 1278005
You need to log in before you can comment on or make changes to this bug.