1692881 - Firefox drops huge amount of video frames on 4k+ videos with a longer keyframe interval

Status: VERIFIED FIXED

(Core :: Audio/Video: Playback, defect, P1)

Description

[Jeffrey G]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0

Steps to reproduce:

Go to YouTube.com
Play UHD 4K Video 1: https://www.youtube.com/watch?v=f0NdOE5GTgo

Play UHD 4K60FPS Video 2: https://www.youtube.com/watch?v=h3fUgOKFMNU

Play Newer UHD 4K60FPS Video 3: https://www.youtube.com/watch?v=ENnWAqufvCI

Actual results:

UHD 4K Video 1 plays fine.
UHD 4K60FPS Video 2 plays fine.
Newer UHD 4K60FPS Video 3 does not play correctly with over half of the frames dropping.

Expected results:

Video 3 should play correctly the same as Video 1 and 2.

Comment 1

[Jeffrey G]

Used YouTube stats for Nerd to record framerate.

Comment 2

[Jeffrey G]

Video 3 Freezes multiple times making the video unwatchable unless the video is downgraded to a lesser resolution.

Comment 3

[Jeffrey G]

Attachment: Screenshot from 2021-02-15 11-02-32.png

The screenshot shows very fast network speed but over half the frames are dropping.

Comment 4

[Jeffrey G]

I have noticed this with other videos as well. It is not just this video.

Comment 5

[Jeffrey G]

I have noticed that video 2 is actually newer than video 3 and video 2 plays fine. So the problem is not the newness of the video but something else. They both seem to use the same codecs which makes this seem strange.

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Audio/Video: Playback' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Comment 7

[Alastor Wu [:alwu]]

I can reproduce this.

Comment 8

[Alastor Wu [:alwu]]

Here is the profiled result where video sink kept dropping frames.

Comment 9

[Jeffrey G]

Thank God! I mean I am sorry for this problem, but I was worried it just might only be my system OS.

Comment 10

[Jeffrey G]

There are many other videos too not just this one.

I think most of the videos I saw like this seem to be recorded using the same type of device. I could be wrong, but most of them seem to be Kpop fancams which may be recorded using high-end 4K cell phones like the Samsung Galaxy S20 or late-model iPhones.

Comment 11

[Jeffrey G]

There is also the possibility that they could be using high-end cameras like RED.

Comment 12

[Jeffrey G]

The sideways full-screen recording may be a factor. I have noticed that this type of recording is increasingly becoming more and more popular.

Comment 13

[Jeffrey G]

Do you think this could be related to bug 1688947 ? - https://bugzilla.mozilla.org/show_bug.cgi?id=1688947

Comment 14

[Alastor Wu [:alwu]]

Attachment: Log

There are two problems,

(1) can we make decoding faster?
Bug 1688947 will probably help if we can utilize GPU as much as we can so that the speed of video decoding can catch up with the speed of playing audio. I noticed that watching same 4K video on Chrome would still cause a lot of dropped frames, but video can be played way more smoothly on Chrome.

Therefore, I think the most critical problem is next one.

(2) the mechanism of synchronizing audio and video
Those dropped video frames happened in VideoSink where we do the sychronization for audio and video. However, at the time we presented them, they were late than current audio clock so that we dropped them.

By checking the log, you can see those time differences were accumulated from 12812 us to 940583us, we finally start another mechanism to skip to the next key frame until the time difference was large enough. However, within those period, we had already made video frozen for several seconds. (I think the improvement here should be doing that earlier) Even if we got new frames, we would also not display them immediately, because those video frames were beyond the current audio clock, so the frozen period was actully really long.

Improving this mechanism should make the result way better, and I need to think about what the best way is to do so.

Comment 15

[Alastor Wu [:alwu]]

wrong comment.

Comment 16

[Jeffrey G]

I found another video with excessive frame drops:
https://www.youtube.com/watch?v=v-MW34KSOSM

Comment 17

[Jeffrey G]

Attachment: Screenshot from 2021-02-21 06-04-11.png

Excessive frame drops: https://www.youtube.com/watch?v=v-MW34KSOSM

Comment 18

[Jeffrey G]

The second video (see screenshot) is from an entirely different channel and most likely filmed with a different device.

Comment 19

[Jeffrey G]

Attachment: Screenshot from 2021-02-22 18-06-51.png

I am getting about 5 to 10% frame drops on Linus Tech Tips 4K videos now. I have the new experimental WebAPI: WebGPU enabled. https://www.youtube.com/watch?v=eS4bNKLxEL0

Comment 20

[Alastor Wu [:alwu]]

Per comment14, as this issue is related with the mechanism of synchronizing audio and video, which is not easy to be modified. We might need a full evaluation about that design before doing any modification, which won't happen soon. Sorry about that.

In addition, as this issue won't happen on all 4K videos and the workaround for users exists, which is to watch video with lower resolution. So I'm going to change the severity of this bug to S3.

(In reply to Jeffrey G from comment #19)

I am getting about 5 to 10% frame drops on Linus Tech Tips 4K videos now. I have the new experimental WebAPI: WebGPU enabled. https://www.youtube.com/watch?v=eS4bNKLxEL0

Does enaling that help with other videos which have huge dropped frames?
Thank you.

Comment 21

[Jeffrey G]

I can't answer the above question because I have reinstalled my system with the new Centos Stream 8 Linux. It could be that because of the good work being done here on this bug I can now play the problem videos stated earlier in this bug report however, I now have a new problem: I am getting a 5% frame drop on ANY YouTube video I watch - even low definition videos:

So, I submitted a new bug: 1695401
https://bugzilla.mozilla.org/show_bug.cgi?id=1695401

I don't know if the new bug I submitted is related to the work being done here.

Comment 22

[Jeffrey G]

50% frame drop to 5% frame drop is a huge improvement by the way. Thank you!

Comment 24

[Alastor Wu [:alwu]]

Update: I'm working on this again, now I already have some WIP.

Comment 25

[Alastor Wu [:alwu]]

Updated some detailed investigation on this document.

Comment 26

[Alastor Wu [:alwu]]

Attachment: Bug 1692881 - part1 : let MDSM detect whether the video decoding is too slow and force to request the next key frame.

If the decoded video MDSM received is already late, then it indicates that video decoding is too slow.

If we keep decoding but in the same speed, it's very likely all following decoded videos would still be behind the clock and need to be discarded.

The reader has its own mechanism to seek to the next keyframe, but that will only happen when the time of next keyframe is behind the given time threshold, which is the current time.

When the video has pretty long keyframe interval (eg. 5s), then that mechanism won't be triggered because the even if the decoded video frames are late, the next keyframe still far exceeds the current time, so the reader doesn't think it's necessary to skip.

Therefore, MDSM should force the reader to seek to the next keyframe directly, because it know both the decoded time of video frames and the current playback time.

Comment 27

[Alastor Wu [:alwu]]

Attachment: Bug 1692881 - part2 : keep decoded video has at least same amount of duration as decoded audio data.

MDSM will stop decoding video when it reaches the certain amount of frames, but that will not work well if the video decoding is way too slow than the audio decoding.

In that situation, video should keep decoding as much as possible, in order to catch up with the audio decoding.

But we also don't need to decode video without any constraint, set the duration of decoded video to at least as the same as the decoded audio seems fair and enough.

Depends on D117242

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/db7fd455cc6a
part1 : let MDSM detect whether the video decoding is too slow and force to request the next key frame. r=bryce
https://hg.mozilla.org/integration/autoland/rev/97fb4c742d68
part2 : keep decoded video has at least same amount of duration as decoded audio data. r=bryce

Comment 29

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/db7fd455cc6a
https://hg.mozilla.org/mozilla-central/rev/97fb4c742d68

Comment 30

[Catalin Sasca, Desktop QA [:csasca]]

Reproduced the issue on 87.0 under Ubuntu 20.04 by using the links provided in Comment 0. The third link would show the first frame then will remain frozen for the rest of the video (only audio works)

The issue is fixed on 92.0a1 (2021-08-02) and 91.0b9, the video is much more fluent (will still keep freezing every ~3 sec but it will render again the video - as the laptop I was testing on will run 4k 30fps videos fluently but 60fps is a bit too much).