Update : currently my patches already got r+, but the reviewer want me to add more test to ensure the A/V won't be broken when we change the playback rate for the media stream track. When I was watching normal videos on Youtube and changed its playback rate during capturing the stream, I didn't notice that anything wrong on the A/V sync. But for a more precise check, I started testing on this [video](https://www.youtube.com/watch?v=A-LmPaRqGvk) where we can clearly notice the defect if the A/V isn't sync. Then, I found out most of time the A/V sync works well if you capture stream before video starts playing. But sometime it would be broken, one example here is to start video first without capturing the stream, then modify the playback rate to 0.5x, then we can see the A/V out of sink. Here is the [profiled result](https://share.firefox.dev/3bGcloT). When `DecodedStream`'s clocktime is `4040854`, `VideoSink` already outputed some frames that are later than the clocktime. At that time the frames in `VideoSink` are like this, it should only play the first frames, instead of those frames later than the clock time. ``` [Child 674857: MediaDecoderStateMachine #1]: V/MediaDecoder DecodedStream=7f94230e4800 time is now 4040854 [Child 674857: MediaDecoderStateMachine #1]: D/MediaDecoder DecodedStream=7f94230e4800 Dropping audio [4014500,4034500] sz=66 [Child 674857: MediaDecoderStateMachine #1]: V/MediaDecoder VideoSink=7f9410d857c0 playing video frame 4037000 (id=7b) (vq-queued=5) [Child 674857: MediaDecoderStateMachine #1]: V/MediaDecoder VideoSink=7f9410d857c0 playing video frame 4071000 (id=7c) (vq-queued=5) [Child 674857: MediaDecoderStateMachine #1]: V/MediaDecoder VideoSink=7f9410d857c0 playing video frame 4104000 (id=7d) (vq-queued=5) [Child 674857: MediaDecoderStateMachine #1]: V/MediaDecoder VideoSink=7f9410d857c0 playing video frame 4137000 (id=7e) (vq-queued=5) [Child 674857: MediaDecoderStateMachine #1]: V/MediaDecoder VideoSink=7f9410d857c0 playing video frame 4171000 (id=7f) (vq-queued=5) ``` But the weird thing is that, when you change the rate to other value than 0.5x, such like 1.5x, 2.0x, it would only have a short period of times where A/V sync is broken (eg. ~1s), then adjust back to normal soon. But for 0.5x, it would still stuck on the wrong A/V. However, if you pause the video, and resume it again, the A/V would back to normal immediately no matter what playback rate you set before. I need to check the logic in `VideoSink` to see what happened. In addition, I also noticed another issue that is sometime if we change the playback rate dramastically, eg. from 2.0x to 0.5x (only happening from very fast to very slow), then the audio clock would get stuck. The reason of clock getting stuck is because MDSM stopped decoding audio data as we have already had many audio data due to previous fast playback rate (`MDSM` would control how many audio data we [need to buffered](https://searchfox.org/mozilla-central/rev/98a9257ca2847fad9a19631ac76199474516b31e/dom/media/MediaDecoderStateMachine.cpp#2869-2873) based on the playback rate.) I haven't fully understood why that would happen comparing with simply playing video in a very slow playback rate. And wonder if those two issues have any similarity in the root cause.
Bug 1517199 Comment 54 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
Update : currently my patches already got r+, but the reviewer want me to add more test to ensure the A/V won't be broken when we change the playback rate for the media stream track. When I was watching normal videos on Youtube and changed its playback rate during capturing the stream, I didn't notice that anything wrong on the A/V sync. But for a more precise check, I started testing on this [video](https://www.youtube.com/watch?v=A-LmPaRqGvk) where we can clearly notice the defect if the A/V isn't sync. Then, I found out most of time the A/V sync works well if you capture stream before video starts playing. But sometime it would be broken, one example here is to start video first without capturing the stream, then modify the playback rate to 0.5x, then we can see the A/V out of sink. Here is the [profiled result](https://share.firefox.dev/3bGcloT). When `DecodedStream`'s clocktime was `4040854`, `VideoSink` already outputed some frames that were later than the clocktime. At that time the frames in `VideoSink` are like this, it should only play the first frames, instead of those frames later than the clock time. ``` [Child 674857: MediaDecoderStateMachine #1]: V/MediaDecoder DecodedStream=7f94230e4800 time is now 4040854 [Child 674857: MediaDecoderStateMachine #1]: D/MediaDecoder DecodedStream=7f94230e4800 Dropping audio [4014500,4034500] sz=66 [Child 674857: MediaDecoderStateMachine #1]: V/MediaDecoder VideoSink=7f9410d857c0 playing video frame 4037000 (id=7b) (vq-queued=5) [Child 674857: MediaDecoderStateMachine #1]: V/MediaDecoder VideoSink=7f9410d857c0 playing video frame 4071000 (id=7c) (vq-queued=5) [Child 674857: MediaDecoderStateMachine #1]: V/MediaDecoder VideoSink=7f9410d857c0 playing video frame 4104000 (id=7d) (vq-queued=5) [Child 674857: MediaDecoderStateMachine #1]: V/MediaDecoder VideoSink=7f9410d857c0 playing video frame 4137000 (id=7e) (vq-queued=5) [Child 674857: MediaDecoderStateMachine #1]: V/MediaDecoder VideoSink=7f9410d857c0 playing video frame 4171000 (id=7f) (vq-queued=5) ``` But the weird thing is that, when you change the rate to other value than 0.5x, such like 1.5x, 2.0x, it would only have a short period of times where A/V sync is broken (eg. ~1s), then adjust back to normal soon. But for 0.5x, it would still stuck on the wrong A/V. However, if you pause the video, and resume it again, the A/V would back to normal immediately no matter what playback rate you set before. I need to check the logic in `VideoSink` to see what happened. In addition, I also noticed another issue that is sometime if we change the playback rate dramastically, eg. from 2.0x to 0.5x (only happening from very fast to very slow), then the audio clock would get stuck. The reason of clock getting stuck is because MDSM stopped decoding audio data as we have already had many audio data due to previous fast playback rate (`MDSM` would control how many audio data we [need to buffered](https://searchfox.org/mozilla-central/rev/98a9257ca2847fad9a19631ac76199474516b31e/dom/media/MediaDecoderStateMachine.cpp#2869-2873) based on the playback rate.) I haven't fully understood why that would happen comparing with simply playing video in a very slow playback rate. And wonder if those two issues have any similarity in the root cause.