1795507 - Fix the green frame issue for the media engine playback

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, defect, P2)

Description

[Alastor Wu [:alwu]]

When using media engine playback, sometime it would display green frames for a short period of time, which usually happens on the start position, or changing the resolution.

Comment 1

[Alastor Wu [:alwu]]

Attachment: WIP: Bug 1795507 - part1 : only return video frame when the dcomp handle is ready.

Comment 2

[Alastor Wu [:alwu]]

Attachment: Bug 1795507 - part1 : do not use the fake data creator for the video stream, and only return decoded video data when the real data exists.

The goal of this patch aims to solve the green frame issue which usually
happens on the start position or when changing the resolution.

The reason of causing the green frame is because of the fake data we
created. Therefore, we should only return decoded video data when the
video stream is able to generate one.

We added a new media queue to store the raw data, which would be used to
generate the output decoded data. The previous one we have is used to
feed raw data into the media engine, which has a different purpose.

Comment 3

[Alastor Wu [:alwu]]

Attachment: Bug 1795507 - part2 : remove FakeDecodedDataCreator and let the engine audio stream to return its own fake data.

Depends on D159537

Comment 4

[Alastor Wu [:alwu]]

Attachment: Bug 1795507 - part3 : make output data return promise and simplify the threading for the video stream.

Depends on D159538

Comment 5

[Alastor Wu [:alwu]]

Attachment: Bug 1795507 - part4 : update the frame status after really receiving the first decoded frame.

Depends on D161550

Comment 6

[Alastor Wu [:alwu]]

Attachment: Bug 1795507 - part5 : consider having pending drain promise as end of track as well in order to solve the deadlock waiting between resolving draining promise and notify EOS from the format reader.

This issue is discovered when playing red-46x48.mp4 in test_playback_rate_playpause.html.

That video only contains one frame, and after reaching EOS, the format
reader will drain the decoder. However, the media engine won't return a
decoded frame before we tell it the track is already ended.

That causes a deadlock, notifying EOS (from reader) waiting for the
drain promise, and the drain promise waiting for first decoded frame, and the
first decoded frame waiting for EOS.

Therefore, we would consider having pending drain promise as a sign of EOS as well,
in order to get the decoded frame and revolve the drain promise.

Depends on D161551

Comment 7

[Alastor Wu [:alwu]]

Attachment: Bug 1795507 - part6 : ignore incorrect events during pausing.

This patch solves the intermittent failure on the try server for
test_playback_rate_playpause.html.

The media engine has an internal issue which would dispatch events in
incorrect status. Eg. ended event should only happen during playback.

The root cause is that the media engine dispatches two ended events,
and the first one is incorrectly dispatched even if the playback hasn't
started yet. That incorrect event is racing with the media decoder's
status change.

When ExternalEngineStateMachine notifies first incorrect ended,
the media decoder hasn't reached a correct state (still in loading), so
the decoder ignores the ended notification from the state machine [1].
However, when the external state machine receives the second correct
ended event, it won't notify media decoder again [2].

Above results along with another addition factor causes the issue, which
is that the media engine doesn't update its current time precisely,
so the test couldn't enter the next step because the current time is
still zero.

If the ended event gets dispatched correctly, current time would be
adjusted to 1 based on the duration [3].

[1] https://bit.ly/3WOR16n
[2] https://bit.ly/3ThmAms
[3] https://bit.ly/3WNvH1a

Depends on D161623

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d9c04494c01b
part1 : do not use the fake data creator for the video stream, and only return decoded video data when the real data exists. r=jolin
https://hg.mozilla.org/integration/autoland/rev/7bf4fc8cffea
part2 : remove FakeDecodedDataCreator and let the engine audio stream to return its own fake data. r=jolin
https://hg.mozilla.org/integration/autoland/rev/6b9f04afb3b5
part3 : make output data return promise and simplify the threading for the video stream. r=jolin
https://hg.mozilla.org/integration/autoland/rev/38abe9245e33
part4 : update the frame status after really receiving the first decoded frame. r=jolin
https://hg.mozilla.org/integration/autoland/rev/0a995fb26153
part5 : consider having pending drain promise as end of track as well in order to solve the deadlock waiting between resolving draining promise and notify EOS from the format reader. r=jolin
https://hg.mozilla.org/integration/autoland/rev/34bb5287b9df
part6 : ignore incorrect events during pausing. r=jolin

Comment 9

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/d9c04494c01b
https://hg.mozilla.org/mozilla-central/rev/7bf4fc8cffea
https://hg.mozilla.org/mozilla-central/rev/6b9f04afb3b5
https://hg.mozilla.org/mozilla-central/rev/38abe9245e33
https://hg.mozilla.org/mozilla-central/rev/0a995fb26153
https://hg.mozilla.org/mozilla-central/rev/34bb5287b9df