884678 - [A/V] Video frame skip fast while youtube streaming

Status: RESOLVED DUPLICATE of bug 883814

(Firefox OS Graveyard :: General, defect, P1)

Description

[leo.bugzilla.gecko]

Precondition: Adapt patch from Bug 882552

STR:
1. Start browser and visit youtube.
2. Play any video file.

When video frame is lagged, it skipped many frames.
So it seems like fast play for 2~4 seconds.

Comment 1

[leo.bugzilla.gecko]

I think the problem occur more frequently by a long video clip.

Please try this URL
https://www.youtube.com/watch?v=V1IBH1bz5-c

Comment 2

[Sotaro Ikeda [:sotaro]]

There is a path to show video playback more quickly than playback rate in nsBuiltinDecoderStateMachine::AdvanceFrame().

When all video frames in VideoQueue() have very late timestamps, AdvanceFrame() renders one video frame. If VideoQueue() size is always one, AdvanceFrame() renders all video frames regardless of their timestamps.

Comment 3

[Sotaro Ikeda [:sotaro]]

Attachment: patch - Do not render too late video frames

Comment 4

[Sotaro Ikeda [:sotaro]]

Leo, can you check if attachment 767064 [details] [diff] [review] works for the problem?

Comment 5

[leo.bugzilla.gecko]

(In reply to Sotaro Ikeda [:sotaro] from comment #4)
> Leo, can you check if attachment 767064 [details] [diff] [review] works for
> the problem?

Your patch affects youtube performance somehow.
I cannot see fast play symptom and symptoms in Bug 883814 is also removed.

However, one problem occur.
1. Start buffering with A/V stopping
2. After a while, Audio is resumed.
3. Video freezes even though audio is played well (maybe...freezing duration is affected by buffering duration) - I think this is because player is dropping old video frames.
4. After that, video frame suddenly jump to proper timestamp

And I got timeout error from video codec sometimes.

Comment 6

[leo.bugzilla.gecko]

I think network connection is not good, now.

I got time out error so many times while buffering...

Comment 7

[Sotaro Ikeda [:sotaro]]

The problem could happen when playback stopped by decoded audio queue or decoded video queue becomes empty. And it seems that the state transition to DECODER_STATE_BUFFERING did not happen.

The decoded video queue could be 3 at most. But decoded audio queue could store 1 sec audio data. Therefore when audio and video playback stopped because of this. The audio track is 1 sec ahead of the video track. When resuming, video frames are dropped until the video track catch up to the audio track.

Comment 8

[Sotaro Ikeda [:sotaro]]

(In reply to leo.bugzilla.gecko from comment #5)
> (In reply to Sotaro Ikeda [:sotaro] from comment #4)
> > Leo, can you check if attachment 767064 [details] [diff] [review] works for
> > the problem?
> 
> Your patch affects youtube performance somehow.
> I cannot see fast play symptom and symptoms in Bug 883814 is also removed.
> 
> However, one problem occur.
> 1. Start buffering with A/V stopping
> 2. After a while, Audio is resumed.
> 3. Video freezes even though audio is played well (maybe...freezing duration
> is affected by buffering duration) - I think this is because player is
> dropping old video frames.
> 4. After that, video frame suddenly jump to proper timestamp

To solve the above problem, MediaCache should always returns the data to OMXCodec immediately when the data is requested by the OMXCodec. If the MediaCache do not have the data as cache, nsBuiltinDecoderStateMachine should transition to DECODER_STATE_BUFFERING and do buffing until enough data is cached in MediaCache.

Comment 9

[Sotaro Ikeda [:sotaro]]

(In reply to Sotaro Ikeda [:sotaro] from comment #8)
> 
> To solve the above problem, MediaCache should always returns the data to
> OMXCodec immediately when the data is requested by the OMXCodec. If the
> MediaCache do not have the data as cache, nsBuiltinDecoderStateMachine
> should transition to DECODER_STATE_BUFFERING and do buffing until enough
> data is cached in MediaCache.

It seems that Current MediaCache does not provide such capability.

Comment 10

[leo.bugzilla.gecko]

First, I changed HasLowDecodedData() function, because it consider only audio data.
I thought that audio keeps being consumed even though there is no video data to be decoded.
And that make symptom here and Bug 883814. ( Just my thought..)

However, nothing works because I think HasLowUndecodedData() cannot calculate remained cache data correctly.
It seems like there is not enough cached data, however the function returns false and StateMachine cannot enter to  DECODER_STATE_BUFFERING.

Sotaro: Is this same as your opinion?

Comment 11

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Comment on attachment 767064 [details] [diff] [review]
patch - Do not render too late video frames

Review of attachment 767064 [details] [diff] [review]:
-----------------------------------------------------------------

This patch would break videos with very slow frame rates. I'm also worried we might have problems with streams where the audio lasts longer than the video; we might discard the last video frame in this case.

Comment 12

[Robert O'Callahan (:roc) (email my personal email if necessary)]

The patches in bug 883814 might help here (by getting us into the BUFFERING state when needed and reducing the likelihood of the video codec being blocked on I/O).

Comment 13

[Sotaro Ikeda [:sotaro]]

(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #11)
> Comment on attachment 767064 [details] [diff] [review]
> patch - Do not render too late video frames
> 
> Review of attachment 767064 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> This patch would break videos with very slow frame rates. I'm also worried
> we might have problems with streams where the audio lasts longer than the
> video; we might discard the last video frame in this case.

Yeah, attachment 767064 [details] [diff] [review] not a correct fix.

Comment 14

[Sotaro Ikeda [:sotaro]]

Leo, can we close the bug from Bug 883814 comment #22?

Comment 15

[leo.bugzilla.gecko]

(In reply to Sotaro Ikeda [:sotaro] from comment #14)
> Leo, can we close the bug from Bug 883814 comment #22?

Yes, the patches in Bug 883814 also remove symptom here.