There are two problems,
(1) can we make decoding faster?
Bug 1688947 will probably help if we can utilize GPU as much as we can so that the speed of video decoding can catch up with the speed of playing audio. I noticed that watching same 4K video on Chrome would still cause a lot of dropped frames, but video can be played way more smoothly on Chrome.
Therefore, I think the most critical problem is next one.
(2) the mechanism of synchronizing audio and video
Those dropped video frames happened in
VideoSink where we do the sychronization for audio and video. However, at the time we presented them, they were late than current audio clock so that we dropped them.
By checking the log, you can see those time differences were accumulated from
12812 us to
940583us, we finally start another mechanism to skip to the next key frame until the time difference was large enough. However, within those period, we had already made video frozen for several seconds. (I think the improvement here should be doing that earlier) Even if we got new frames, we would also not display them immediately, because those video frames were beyond the current audio clock, so the frozen period was actully really long.
Improving this mechanism should make the result way better, and I need to think about what the best way is to do so.