Closed Bug 1557994 Opened 6 months ago Closed 4 months ago

Picture-in-Picture flickering on animelab.com

Categories

(Toolkit :: Video/Audio Controls, defect, P2)

69 Branch
defect

Tracking

()

VERIFIED FIXED
mozilla70
Tracking Status
firefox70 --- verified
firefox71 --- verified

People

(Reporter: winson.wen1, Assigned: mconley)

References

(Blocks 1 open bug)

Details

Attachments

(4 files)

Attached video flicker.mp4

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0

Steps to reproduce:

Sign in to animelab.com
Start playing a show
Switch to picture-in-picture

Animelab might be geoblocked to Australia

Actual results:

Video flickers between currently playing point and the beginning of the episode

Expected results:

Video plays without flicker

Component: Untriaged → Video/Audio Controls
Product: Firefox → Toolkit

ni mconley for PiP triage

Flags: needinfo?(mconley)
Blocks: 1527926
Component: Video/Audio Controls → Audio/Video: Playback
Flags: needinfo?(mconley)
Product: Toolkit → Core

Hi jya,

I suspect something's going wrong in the visual cloning we're doing here... does the video capture from the reporter give you any insight into what we might be doing wrong here?

Flags: needinfo?(jyavenard)

(In reply to Mike Conley (:mconley) (:⚙️) from comment #2)

Hi jya,

I suspect something's going wrong in the visual cloning we're doing here... does the video capture from the reporter give you any insight into what we might be doing wrong here?

That's frames being displayed out of order and the destination frame being re-used by the decoder before it got painted.

Sounds like a refcount being cleared too early.

When in the case of PiP, how do we make sure frames aren't being recycled before they are rendered?

We had a similar case occurring on mac in the past because we didn't increase the refcount on the underlying IOSurface.

Matt, how is this handled on Windows?

Flags: needinfo?(jyavenard) → needinfo?(matt.woodrow)

This sounds like it could be a bug in the recycling code we use for D3D11 video textures. I'm not sure if we correctly handle the case where there are two consumers that might need to hold references.

Any ideas Sotaro?

Flags: needinfo?(matt.woodrow) → needinfo?(sotaro.ikeda.g)

Winson Wen, can you upload about:support after the problem happened?
Open about:support, click on "Copy text to clipboard", paste it into a text file and attach it to the report. Thanks!

Flags: needinfo?(winson.wen1)

I live in Japan, then it seems that I could not watch the video.
Winson Wen, is there another video service that cause the same problem?

(In reply to Matt Woodrow (:mattwoodrow) from comment #4)

This sounds like it could be a bug in the recycling code we use for D3D11 video textures. I'm not sure if we correctly handle the case where there are two consumers that might need to hold references.

Any ideas Sotaro?

Recycling should be handled correctly even in this use case. If recycling happened too early, a bit future frame is shown like flickering. But in this case, the old same frame was shown...

Attached file about:support

I've attached the about:support text.
Haven't encountered this problem anywhere else so I don't know what other websites you could try, sorry.
The weird thing about the bug is though, the flickering always shows frames from the beginning of the video, even if I start playing the video from the middle. And every time I go into picture-in-picture, the flickering frames reset to the beginning.

Flags: needinfo?(winson.wen1)
Flags: needinfo?(sotaro.ikeda.g)

The priority flag is not set for this bug.
:drno, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(drno)

Hi jya,

We were talking about this at Whistler, and I recall you mentioning that you were interested in whether or not the user experienced the flickering after the PiP player window was closed. Looking at the posted recording at https://bugzilla.mozilla.org/attachment.cgi?id=9070817, it does appear as if the flicker goes away once the player window is closed.

Does that help box the problem in a bit?

Flags: needinfo?(jyavenard)

I confirm the same issue on seasovar.ru
Appears that picture-in-picture mode begins to autoplay video without sound and without any synchronisation with the main video frame. This causes flickering - two video streams being played at the same time of the same video but on different time position.

(In reply to Mike Conley (:mconley) (:⚙️) from comment #10)

Hi jya,

We were talking about this at Whistler, and I recall you mentioning that you were interested in whether or not the user experienced the flickering after the PiP player window was closed. Looking at the posted recording at https://bugzilla.mozilla.org/attachment.cgi?id=9070817, it does appear as if the flicker goes away once the player window is closed.

Does that help box the problem in a bit?

Nope.

In the other bug, the issue was resolved by enabling webrender.
But here webrender is enabled.

I have no explanation on why this is happening here.
The only explanation is that one of the recycled surface is incorrectly used, or used at the wrong time.

Flags: needinfo?(jyavenard)
Attached video weirdness.mp4

So I've experimented a bit more and found that with hardware acceleration turned off in about:preferences, the flickering reduces dramatically, only flickering when first entering picture-in-picture, and around the control buttons when hovering in and out or the window.

Also, pausing the video while in picture-in-picture only pauses the video playing the current time, the parts playing from earlier continue and also stop flickering. Attached a video showing what I mean.

So the reduced flickering with hardware acceleration disabled feels very temperamental, having obs open seems to break it. Pausing and playing while in pip with h/w acceleration off also seems to randomly switch between 3 states, the flicker-y mess from the bug description, playing the current time of the video with little flicker, or playing the earlier part with audio from the current part.

So testing a bit more, even with hardware acceleration it appears to switch randomly between 3 states when pausing and playing in pip, flickering evenly between the early and current part of the video, playing mostly the current part with flickers from the early part, and playing mostly the early part with flickers of the current part.

Also, sorry for all the comment spam, I'm kinda writing things as I find them. I will try batching stuff up more before commenting next If I find anything else.

Matt does any of the new information provided by Winson in comments #13-15 provide any more insight into what could be going on here?

Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: needinfo?(drno)
Priority: -- → P2

Forgot to set NI: see above ^

Flags: needinfo?(matt.woodrow)

I can confirm identical problem with all tvn24.pl videos.
e.g. https://tvnmeteo.tvn24.pl/informacje-pogoda/prognoza,45/pogoda-na-16-dni-niedlugo-ochlodzenie-przerwie-upal,296088,1,0.html
Tested on Windows 10 with Geforce 1070 and Ubuntu 19.04 and Intel HD 520.

(In reply to igor.zuk from comment #18)

I can confirm identical problem with all tvn24.pl videos.
e.g. https://tvnmeteo.tvn24.pl/informacje-pogoda/prognoza,45/pogoda-na-16-dni-niedlugo-ochlodzenie-przerwie-upal,296088,1,0.html
Tested on Windows 10 with Geforce 1070 and Ubuntu 19.04 and Intel HD 520.

Thanks for the report! I also confirmed the problem.

(In reply to igor.zuk from comment #18)

I can confirm identical problem with all tvn24.pl videos.
e.g. https://tvnmeteo.tvn24.pl/informacje-pogoda/prognoza,45/pogoda-na-16-dni-niedlugo-ochlodzenie-przerwie-upal,296088,1,0.html
Tested on Windows 10 with Geforce 1070 and Ubuntu 19.04 and Intel HD 520.

When the problem happened, VisualCloneTarget HTMLVideoElement also created MediaDecoder and playbacked a video. Then, VideoSink of VisualCloneTarget also rendered video. Then VideoFrameContainer of SecondaryContainer received video frames from two VideoSinks.

When I disabled to create MediaDecoder at VisualCloneTarget HTMLVideoElement , the problem did not happen.

:mconley, can you comment to it?

Flags: needinfo?(mconley)

Thanks, sotaro! That helps a lot.

The original <video> HTML node is cloned, and inserted into the player video, and then becomes the "visual clone target". The reason that we clone the original node is to get the same MediaInfo of the <video>, which includes things like its dimensions.

We can halt the decoder on the cloned video by pausing it after it's cloned.

Assignee: nobody → mconley
Component: Audio/Video: Playback → Video/Audio Controls
Flags: needinfo?(mconley)
Flags: needinfo?(matt.woodrow)
Product: Core → Toolkit

Here are some try builds with the patch to try:

Windows 32-bit: https://queue.taskcluster.net/v1/task/b9nf-9qxROG9qWd19p-9tw/runs/0/artifacts/public/build/target.zip
Windows 64-bit: https://queue.taskcluster.net/v1/task/cp5N4XrQTA6mq07xvmrVOg/runs/0/artifacts/public/build/target.zip

Hey Winson, if you have a moment, can you confirm that the patch fixes the issue with one of these builds?

Flags: needinfo?(winson.wen1)

Everything's working fine, looks like the problem's fixed!

Flags: needinfo?(winson.wen1)
Pushed by mconley@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/dbb8e4a9609d
Picture-in-Picture player video element should not try decoding its own version of the originating video. r=JSON_voorhees
Status: NEW → RESOLVED
Closed: 4 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla70

Build ID 20190902191027
User Agent Mozilla/5.0 (Windows NT 10.0; rv:70.0) Gecko/20100101 Firefox/70.0

Verified as fixed on the latest Beta version (v70b3) and on the latest Nightly version.

Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.