1879417 - [wmfme] sometime media engine would incorrectly drop huge amount of video frames

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, defect, P1)

Description

[Alastor Wu [:alwu]]

We've discussed this issue with Microsoft, and found out that this is our problem by incorrectly sending out-of-order frames. The problem is caused by unexpected drain triggered by MediaFormatReader (MFR). Let's break down more details below.

Why draining?

We have a mechanism to tell the content process stopping sending input data if the MFCDM process already has enough data.

However, this mechanism doesn't fit well with out MFR mechanism. As we reuse MFR for getting the demuxed samples, MFR has its own mechanism to check whether sending input data to decoder.

The media engine playback uses Direct composition, so it can't return any video frames before the Dcomp surface handle is ready, which results in returning empty decoded data in the decode promise. Therefore, MFR would keep sending more and more input in order to get an output data.

However, when the Dcomp surface is ready is not something we can control. If MFR has send all available demuxed samples to the MFCDM process, then MFR would start drainning the decoder.

How does draining cause dropped frames?

Draining might trigger the internal seek, which would seek back to the last video sample. In the video I was testing, the key frame interval is around 5s. If the last video sample is not a key frame, then we would get the data which is approximately 5 seconds before (in the worst situation) and have been sent to the media engine before. Those samples which have been sent twice would all be discarded by the media engine.

Comment 1

[Alastor Wu [:alwu]]

Attachment: Bug 1879417 - part1 : putting the video decode promise on hold if we already have enough data and Dcomp surface is not ready yet.

The detailed explanation of the issue is described in [1].

By postponing resolving the decode promise, MFR would not keep sending
unlimited input data to the MFCDM process.

By avoiding that behavior, now we won't experience the unexpected
draining which is caused by sending all available video input data.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1879417#c0

Comment 2

[Alastor Wu [:alwu]]

Attachment: Bug 1879417 - part2 : resolve pending input promise when sample is popped.

The problem is discovered when playing the file red-46x48.mp4 in
test_playback_rate_playpause.html. That file only contains ONE frame
which is 1 second long.

There is an issue within the media engine, which The won't return an
output frame to us, unless we tell it the stream is ended.

Therefore, after sending the only frame to the media engine, we need to
(1) keep the media format reader to keep requesting data by resolving
the pending promise when the input has been popped.

(2) send the ended event to the media engine in order to get the first
frame.

Depends on D201173

Comment 3

[Alastor Wu [:alwu]]

Attachment: Bug 1879417 - part3 : remove unused method.

Depends on D202336

Comment 4

[Alastor Wu [:alwu]]

I'd like to add more context here. In the original design, the media engine would communicate with the external state machine directly. Media engine side would send different events to the state machine to control how many input it needs. The amount of data we store in advance is controlled by MFMediaEngineStream::HasEnoughRawData(), and we have different threshold for audio and video.

However, as we use MediaFormatReader as an intermediate bridge to access demuxed data, its internal mechanism sometime would interrupt our original design. In a special situation, even if the media engine sends Audio/VideoEnough to the state machine, the media format reader still keep sending input sample to the media engine, which causes the problem mentioned in the comment 0.

The reason is because that the format reader has its own mechanism to determine whether it needs to send more input. If the media engine hasn't created its dcomp handle, we won't return any output to the format reader, so it would keep sending more input to the media engine. When I said won't return any output, that means the decoded promise would be resolved with an empty array without a actual decoded data.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3d6e92f92e64
part1 : putting the video decode promise on hold if we already have enough data and Dcomp surface is not ready yet. r=jolin
https://hg.mozilla.org/integration/autoland/rev/f1772dfe55a9
part2 : resolve pending input promise when sample is popped. r=jolin
https://hg.mozilla.org/integration/autoland/rev/743f9b5bd8e0
part3 : remove unused method. r=media-playback-reviewers,jolin

Comment 6

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/3d6e92f92e64
https://hg.mozilla.org/mozilla-central/rev/f1772dfe55a9
https://hg.mozilla.org/mozilla-central/rev/743f9b5bd8e0