MP4+H264 keyframe workarounds likely cause more seeking issues than they solve
Categories
(Core :: Audio/Video: Playback, defect, P3)
Tracking
()
| Tracking | Status | |
|---|---|---|
| firefox162 | --- | fixed |
People
(Reporter: tnibler, Assigned: tnibler)
References
Details
Attachments
(2 files, 3 external files, 2 obsolete files)
Hi! I would like to make the case for removing the special keyframe handling and seeking logic
from the MP4 demuxer.
Or at least revive the discussion, because I think it causes more problems than it solves.
Issues related to this come up quite a lot: bug 1481496, bug 1762450, bug 1709081, bug 1932283 (Comment 7).
I would guess that bug 1978244 and bug 1620469 refer to the same thing (video either freezes until next IDR or jumps to end of stream).
As I understand it, it started with a strange WPT sample MP4 where the container indicated
that some frames were keyframes when they actually weren't.
That made the WPT flaky on Apple (bug 1294552), everything else seems to play that file without issues.
The fix was to discard the keyframe flag from MP4 packets (bug 1300296) and parse NAL headers
to find IDR frames (and recovery point SEI units). And also that broken file was simply removed from WPT.
Then the IMDB seeking issues came up: bug 1481496, bug 1539182, bug 1359058.
Some IMDB videos are encoded with only one NAL unit type 5/IDR at the beginning,
with later recovery points only marked in the container, which is now ignored.
With effectively one keyframe, the videos became unseekable as you would expect.
The fix for that now is to be much more permissive in what counts as a keyframe,
as any I or SI slice is considered a valid recovery point,
even when the correct information was parsed from the MP4 but explicitly discarded.
So, the problem was a non-compliant MP4 that pretends frames are keyframes when they are not,
and now as a result Firefox lies about even more stuff being keyframes!
The difference being that this actually breaks seeking on perfectly compliant files (Vimeo for instance), on all platforms.
Strangely enough the code even announces that it's about to do the wrong thing:
Warning log: "Frame incorrectly marked as %skeyframe ", keyframe ? "" : "non-"
Then proceeds with the incorrect value: sample->mKeyframe = keyframe;
At the very least, if Apple decoders choked on something, then those should get the workaround treatment, not everything else.
Because after all this the Apple code path is actually compliant now, doesn't treat every I-frame as IDR
and retries seeking from an earlier point if no suitable keyframe was found.
For reference, here's chromium just using AV_PKT_FLAG_KEY straight from FFmpeg [1] [2].
I couldn't find any place where it would be overridden or modified by codec-specific logic.
I've also attached some test files to reproduce these problems.
All of them and the other examples from the linked threads start working for me
after removing the special fix logic and just doing it the "naive" way.
If fraudulent keyframe flags are really a common issue we can of course do additional checks,
but in my opinion they should not break the common case with valid files, and the current implementation
is quick to do so often times.
| Assignee | ||
Comment 1•2 months ago
|
||
The spec does not require that decoding can start at any I or SI slice,
and valid files will fail to play by attempting anyway.
Updated•2 months ago
|
| Assignee | ||
Comment 2•2 months ago
|
||
| Assignee | ||
Comment 3•2 months ago
|
||
There are real-world examples of files with very few actual IDR frames,
and instead other I frame types that are not always keyframes but in
this case marked as keyframes on the container level.
Discarding these keyframes can force seeking to start from much earlier
in the stream, which is slow.
| Assignee | ||
Comment 4•2 months ago
|
||
Patch 1 fixes corrupt.html and vimeo.html, and breaks imdb.mp4.
Patch 2 fixes imdb.mp4 by seeking from the IDR at the beginning every time.
Patch 3 makes it use keyframes as marked by the container and seeking is fast again.
Comment 5•1 month ago
|
||
The severity field is not set for this bug.
:jimm, could you have a look please?
For more information, please visit BugBot documentation.
Updated•1 month ago
|
Comment 6•1 month ago
|
||
(In reply to Thomas Nibler [:tnibler] from comment #0)
So, the problem was a non-compliant MP4 that pretends frames are keyframes when they are not,
and now as a result Firefox lies about even more stuff being keyframes!
The difference being that this actually breaks seeking on perfectly compliant files (Vimeo for instance), on all platforms.
Thanks for filing this. Would you mind providing some example link how Firefox breaks things on the actual streaming websites? or I should just check the files you provided?
I've tried your test cases (reproduction.html / vimeo.html and imdb_noaudio.mp4)
- Both videos on the
reproduction.htmlwill become stalled after seek, same on Chromium and Firefox. vimeo.htmlcan play a little, then stuck. On Chromium, it can't even be loaded.imdb_noaudio.mp4on Firefox is able to seek, and audio and video play successfully. On Chromium, the video will frozen after seek.
Patch 1 fixes
corrupt.htmlandvimeo.html, and breaksimdb.mp4.
Patch 2 fixesimdb.mp4by seeking from the IDR at the beginning every time.
Patch 3 makes it use keyframes as marked by the container and seeking is fast again.
I'd like to fully understand the pros/cons of existing behavior, versus the solution you proposed. In addition, given that we didn't have enough test coverage for those past issues, we would need to add them as well to ensure no breakage if we'd like to make a change. Thanks!
| Assignee | ||
Comment 7•1 month ago
|
||
The test cases are collected from the linked bugs and comments. Reproducing on the sites themselves is tricky, that's why I have these cherry picked segments and timestamps. Most of the time seeking anywhere will work of course. With "bad" timestamps usually decoding will start from a non IDR I-frame (and stall at first), and it takes a tiny bit until the next real keyframe for the image to move again, but mostly unnoticeable. A full freeze happens when seeking after a "fake" keyframe that's past the last real keyframe, and the DASH/HLS client has not loaded the next segment yet, where decoding could resume on a real IDR. That's the situation reproduced in the examples.
On a slow connection you can make it work on Vimeo for example, but with a lot of trying. I can anecdotally confirm it happens in real usage though and other users have reported it too.
vimeo.html can play a little, then stuck
Sorry I broke that somehow, it's missing the codec string video/mp4; codecs="avc1.64000d". It plays to the end (1:04-1:06, ~2 sec) on Chromium for me, not at all on latest FF main with SW/HW decoding. Does it really show an image for you? Interesting!
Both videos on the reproduction.html will become stalled after seek, same on Chromium and Firefox
Seeking to 5:12 does not work for me and the original reporter Bug 1709081, but to 5:10 where the previous keyframe is an actual IDR works. In Chromium both timestamps work and it plays to the end at 5:15.
imdb_noaudio.mp4 on Firefox is able to seek, and audio and video play successfully. On Chromium, the video will frozen after seek.
Hmm, it works in Chromium for me, there's funny things going on here. But yes in firefox it works, and it takes these steps:
- Parsing the container, where I-frames at regular intervals are marked as keyframes.
- Because of bug 1294552, these are ignored.
- That broke bug 1539182, bug 1359058, because of which all I-frames are now treated like IDR frames. That fixes imdb, but also causes the other mentioned issues, where seeking just past a non-IDR I-frame makes playback stall.
My reasoning now is that the bug 1294552 at the start of this chain of fixes is moot, the file was recognized as broken and the WPT removed. Real-world cases of users trying to play similarly broken files have not come up as far as I can tell. So the fix is no longer required.
Removing that solves the IMDB regression, because keyframes marked as such only in the container will now be treated as real keyframes, and seeking works again.
Removing the now obsolete fix for the IMDB bug solves the other listed bugs, because we will no longer try to decode from frames that are not and never claimed to be recovery points.
The patches undo those workarounds in matching steps, the intermediate states don't make much sense by themselves.
Pros: fix occasional seeking issues and remove special case code paths.
Possible cons: a weird broken MP4 with marked keyframes that are not actually keyframes might have seeking problems on Apple, at least with the H264 decoder in 2016 hardware. If that turns out to be real, we can always opt to ignore container info on that affected Apple platform.
All tests pass with the changes, there's not a ton of them exercising seeking edge cases though. So it's definitely a good idea to add more of those.
I can't think of any way that this would cause breakage, at the very most I guess: an MP4 with no IDR frames at all, no keyframes marked in the container, where it just so happens that starting decoding from I-frames works for some reason, would not be seekable anymore.
Even that could be remedied though, if we notice there's zero keyframes marked in the container and zero IDR frames to seek to you can always decide to try anyway with a regular I-frame, which is the default MO right now.
| Assignee | ||
Comment 8•1 month ago
|
||
Comment 9•1 month ago
|
||
(In reply to Thomas Nibler [:tnibler] from comment #4)
Patch 1 fixes
corrupt.htmlandvimeo.html, and breaksimdb.mp4.
Patch 2 fixesimdb.mp4by seeking from the IDR at the beginning every time.
Patch 3 makes it use keyframes as marked by the container and seeking is fast again.
Since I’m not sure under which licenses those videos are distributed, we can’t use them directly in our tests. Instead, we could use FFmpeg to create similar samples or write a gtest to cover these cases. Could you help with that?
Sorry I broke that somehow, it's missing the codec string video/mp4; codecs="avc1.64000d". It plays to the end (1:04-1:06, ~2 sec) on Chromium for me, not at all on latest FF main with SW/HW decoding. Does it really show an image for you? Interesting!
Sorry if I wasn’t clear. I tested your test cases on Nightly, not on the build containing your fixes. On Nightly, vimeo.html plays for about three seconds and then stalls.
Also, when you say “fixes,” do you mean that the stall will no longer occur? For the imdb case, could you elaborate on what the fix does? Does it mean seeking only to the starting position rather than to an arbitrary position?
Thanks!
| Assignee | ||
Comment 10•1 month ago
|
||
(In reply to Alastor Wu [:alwu] from comment #9)
On Nightly,
vimeo.htmlplays for about three seconds and then stalls.
Very interesting, just tried it on my Android and it plays indeed, but on two Arch Linux and NixOs machines it doesn't play until I seek before ~64.365s, starting anywhere after that I-frame there does not work. For the original reporter it seems to be even worse. I'll add a screen recording showing the picture freeze and eventual error on the actual site.
Also, when you say “fixes,” do you mean that the stall will no longer occur?
Yes, I mean you can start playback at any point without stalling, and seek anywhere without the image freezing for a moment.
For the
imdbcase, could you elaborate on what the fix does?
It doesn't really fix anything since it currently plays perfectly fine, sort of accidentally. It just serves as a good half-broken edge case.
You can start playing anywhere and seek to any time, even when the seek points aren't keyframes. There is only one IDR, but it so happens that all other I-frames also work as recovery points, even though neither the MP4 nor AVC information say that.
If you follow the spec and stop considering all I-frames as IDRs it will not be seekable anymore, as found in bug 1539182.
The more logical thing to do if there's no keyframe near the seek target is to jump backwards and try again from an earlier keyframe, but that path is currently only enabled on Apple. Enabling that makes it possible to seek anywhere in imdb again, while staying compliant.
Also actually I need to correct what I wrote above, because apparently I can't read very well: Chromium also doesn't trust MP4 container info about keyframes. By that logic you'd have to decode all the way from the first IDR frame for every seek in imdb, and that seems to be the case. Especially w/o hardware decoding, pause -> seek -> play is noticeably slow in chromium, in Firefox it's instant.
Patch 3 should probably be dropped then to keep things consistent and avoid breakage. That means seeking in videos like imdb, which have a single IDR but frequent non-IDRs that somehow still work as recovery points will become slower. IMO, encoding videos like that is user error and quite rare, so the tradeoff seems acceptable.
In exchange all the other cases, where the MP4 is compliant but Firefox is not, will work correctly.
we could use FFmpeg to create similar samples or write a gtest to cover these cases. Could you help with that?
Yes of course, I can do that :)
| Assignee | ||
Comment 11•1 month ago
|
||
Updated•1 month ago
|
| Assignee | ||
Comment 12•1 month ago
|
||
These tests replicate different instances of broken seeking, for example
bug 1709081, bug 1932283, bug 1481496, bug 1539182.
single-idr and nidr-after-idr are related to the fact that Firefox
currently
treats all I and SI slices as IDR frames i.e, valid points to start
decoding
with blank decoder state.
This often times doesn't work, manifesting in full stalls or a frozen
image
until the next real keyframe is reached, or jumping further forward than
the target seek time (including to the end of the stream).
In these cases the files are not at fault and seeking issues are a bug.
sync-lies and single-idr-no-stss are "broken" MP4 files, where the
container
designates samples as sync that are not actually keyframes at the codec
level. Seeking to these not-keyframes will not work and causes varying
issues
across user reports.
Firefox, Chromium and Safari currently handle this by ignoring keyframe
flags in the container and parsing NAL types themselves.
The tests ensure this workaround won't break.
Updated•1 month ago
|
Updated•1 month ago
|
Updated•13 days ago
|
Comment 13•10 days ago
|
||
Comment 14•10 days ago
|
||
| bugherder | ||
https://hg.mozilla.org/mozilla-central/rev/a58c158534a9
https://hg.mozilla.org/mozilla-central/rev/8ca5d465f499
https://hg.mozilla.org/mozilla-central/rev/500a0cf00cf8
Description
•