1158236 - Transition to new-style keyframe skipping

Status: RESOLVED WORKSFORME

(Core :: Audio/Video: Playback, defect)

Description

[Bobby Holley (:bholley)]

Chris and I discussed this last night, and I want to get this all on file so that we don't forget everything.

There are basically two kinds of keyframe skipping, with mostly-orthogonal goals:

Old-style: For legacy decoders that decode audio and video on the same thread, we need to proactively sacrifice video decoding in order to maintain smooth audio. In this model, we discard future, potentially-playable frames, and instead opt to continue displaying some random frame for much longer than its intended duration (i.e. until the subsequent keyframe). This slack causes the MDSM to stop requesting new video frames for a while, which lets us buffer up audio and maintain smooth audio playback at the expense of jerky video playback [1].

Old-style skipping is unnecessary when we decode audio and video on separate threads, because the processing is decoupled. However, if audio is humming along glitch-free and video is struggling, we need a mechanism to prevent the video from lagging further and further behind. This is new-style skipping.

New-style: For modern decoders that decode audio and video on separate threads, we need a way to ensure that video decoding keeps up with the audio clock (the assumption is that audio decoding is cheap enough that we don't have to worry about it keeping up, as long as it runs on its own thread). In this model, we want to unconditionally discard any video frames earlier that K, where K is the closest keyframe to currentTime while maintaining k.timestamp <= currentTime. Decoders can do this unconditionally, and don't need the MDSM to decide about it (though they do need currentTime from the MDSM).

Old-style skipping is a nasty heuristic that we can get rid of with better architecture. New-style skipping is a necessary piece of proper real-time code, and we should fully support and test it.

[1] http://blog.pearce.org.nz/2014/12/firefox-video-playbacks-skip-to-next.html

Comment 1

[Bobby Holley (:bholley)]

We should wait for jya's new architecture (bug 1148292). Once we have that, we can keep our implementation of new-style keyframe skipping there, and port all the decoders to it.

Comment 2

[Bobby Holley (:bholley)]

Attachment: (Probably not for landing) - Hoist keyframe skipping into WebMReader and redo the algorithm. v1

I wrote this patch to convert WebMReader before I realized that new-style
keyframe skipping only works with separate audio and video decode threads,
which we don't have for WebM yet. So we should wait until jya refactors
things and then see what it looks like. Just getting this posted somewhere
in case we ever want to come back to it.

Comment 3

[Anthony Jones (:ajones, :kentuckyfriedtakahe, :k17e)]

Can we close this bug no that we have MediaFormatReader?

Comment 4

[Jean-Yves Avenard [:jya]]

This is now done for all readers as there can only be one.