1174419 - Don't decode videos that are hidden

Status: RESOLVED DUPLICATE of bug 1224973

(Core :: Audio/Video: Playback, defect)

Description

[Caspy7]

We shouldn't decode videos that are hidden.  Notably videos in background tabs and hopefully those which are off screen (in the current tab).

Obviously the audio should continue.

This will help with power usage for mobile situations like phones, tablets and unplugged laptops.  But also potentially with other situations related to performance I haven't thought of.

Comment 1

[Caspy7]

On a personal note I would also add that currently my GPU is blacklisted so all videos run through the CPU. This makes things quite hot very quickly on my laptop.  
There are times that I would like to pull up a music video in order to simply listen to the music, but I'm less likely to do so with the full video being decoded as it currently is.

Comment 2

[Jean-Yves Avenard [:jya]]

How could that work while keeping uninterrupted playback?

What if you suddenly unhide the video (either by making the tab active, uncovering the window etc...)

you would want the video to appear instantly right? and be exactly at the position it was already before being unhidden. To do so, you need to seek into the video (and as such seek audio to have proper A/V sync)

Doing so without any hiccups or interruptions or the audio skipping backward is far from trivial.

Comment 3

[Randell Jesup [:jesup] (needinfo me)]

There are user expectations to deal with too:

If audio continues, when you switch back the video needs to be seeked - and really, since the stream has continued, it's not really 'seeked, video decoding has to start again and "catch up".  In practice, the way most likely to succeed would be to just start decoding frames (and because you're missing the keyframe, throwing them away until you get the next keyframe).  This is doable - but will cause a visual glitch, though *probably* not a very objectionable one.  Fancier schemes (cache undecoded data starting at the last keyframe and fast-forward, etc) are likely to be complex and error-prone (with nasty edge cases like 2 minutes between keyframes).

Note that this is not true: "and be exactly at the position it was already before being unhidden." -- that wouldn't match "audio continues".  Having a user pref to pause videos (including audio) when the tab is hidden might be viable, at the cost of yet-more-UI.  Obviously that could be part of an extension (especially if it was available as an about:config pref).

I will say that this could have a large impact on system load especially as more ads move to HTML video and away from flash....

roc: thoughts?

Comment 4

[Robert O'Callahan (:roc) (email my personal email if necessary)]

I think we should do something to, by default, stop videos from automatically playing in tabs that have never been visible, e.g. by simply stalling videos in those pages until the tab becomes visible. That's user-visible but also user-desirable, I think, and would take care of some of this problem.

I think it would also be fine to stall playback for any video that's never been visible (e.g. because it's below the fold), and that has no audio or has muted audio.

To be really conservative we could apply those limitations only to videos in cross-origin IFRAMEs.

Then the question is how much of an issue remains, and I don't know the answer to that. Maybe not enough to worry about.

If we need to go beyond that, I'm not sure what to do. We could try modifying CPU decoders to have a mode where they don't produce output frames, which should lead to some savings. That would be work for VPx but not for H264 where we don't ship the decoder. Also, I guess it would be valuable to limit the use of hardware decoders since they're a scarce resource, and the only way to do that is to not decode at all.

Comment 6

[Chris Pearce [:cpearce (Not reading bugmail)]]

(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) (offline July 8-10) from comment #4)
> I think we should do something to, by default, stop videos from
> automatically playing in tabs that have never been visible, e.g. by simply
> stalling videos in those pages until the tab becomes visible. That's
> user-visible but also user-desirable, I think, and would take care of some
> of this problem.

I added this behaviour behind the media.block-play-until-visible pref in bug 1180563. It should be in Nightly tomorrow.

Comment 7

[Jonas Thiem]

The image and gif hoster imgur.com has started serving most gifs as embedded .webm videos without audio. Opening many tabs with those "gifs" seems to result in a really sluggish firefox.

Therefore, a solution to this that works for videos that have once been shown already would be highly desirable, especially if they don't even have an audio track.

Why not simply use the offset seek solution for a video without audio? I doubt anyone minds the brief pause it needs to seek if the video doesn't even have sound. Of course re-syncing with sound would be hard, but nobody will have multiple tabs with videos _with sound_ open anyway - while many such tabs with videos WITHOUT sound is very realistic (image/gif sites, ads, ...).

Comment 8

[Randell Jesup [:jesup] (needinfo me)]

(In reply to Jonas Thiem from comment #7)
> The image and gif hoster imgur.com has started serving most gifs as embedded
> .webm videos without audio. Opening many tabs with those "gifs" seems to
> result in a really sluggish firefox.
> 
> Therefore, a solution to this that works for videos that have once been
> shown already would be highly desirable, especially if they don't even have
> an audio track.
> 
> Why not simply use the offset seek solution for a video without audio? I
> doubt anyone minds the brief pause it needs to seek if the video doesn't
> even have sound. Of course re-syncing with sound would be hard, but nobody
> will have multiple tabs with videos _with sound_ open anyway - while many
> such tabs with videos WITHOUT sound is very realistic (image/gif sites, ads,
> ...).

"nobody will have multiple tabs with video w/sound".... Have you met the modern web advertising industry?  Or even random sites that love to roll a video with sound on an auto-reload (leading to the "what bloody tab is the sound coming from!??" case).  Though your point is taken.

Comment 12

[Djfe]

my ideas to this (copy pasted from my bug report, so some things are repetitions from above):
Firefox doesn't need to decode video, only the audio, if you are browsing in a different tab.
This could potentially save a lot of CPU time or power (even with hardware acceleration enabled)
Videos on a webpage that aren't visible fall in the same category (whether by a css rule or caused by you scrolling down to let's say the video's comments)


The same thing could be applied for audio if the speaker icon on a tab is clicked to make it silent.

The Demuxer would still keep on running but the decoder would be disabled.

I'm not sure if this is really possible, especially since Videos contain lots of reference frames.
And if the pros outweigh the odds (calculations whether the video is visible)

But maybe it's enough already to use the Demuxer's cache to get everything till the last I-Frame (or IDR depending on the Codec)

Nobody puts videos on the web with 2 minutes between reference frames...
But for such edge-cases you could simply tell the decoder to start running once the cache gets too big

Since the demuxer is running all the time, audio and video should stay in sync
the decoder would just need some time to catch up to the current image
He will either wait for the next I-Frame, or he will try to decode till the current Frame but drop everything if a new I-Frame arrives first and then decode from there

That's exactly how it's on TV, too
you can hear the audio for a second or too until the black screen gets replaced with the first decoded frame

I'm not sure whether this is doable for H.264 since it's implemented with a plugin
But if the demuxer is not part of that plugin, then it should be doable

for webm/vp9 it makes the most sense, because it's so CPU intensive (before Skylake)
for audio it should be easier to implement and might even work flawlessly (because it's not different from choosing one audio track from a video containing several ones, except that you play none)

but I have never developed anything similar, so I'm looking forward to the devs' oppinions
most of all I'm unsure whether it is even possible to combine this idea with DASH/MSE (YouTube)

Comment 13

[Jean-Yves Avenard [:jya]]

(In reply to Djfe from comment #12)
> The Demuxer would still keep on running but the decoder would be disabled.

there's no point to keep demuxing, and actually you certainly don't want to.
Typically video codec requires the previous frames to be decoded to decode the next one.

So the only way to display the proper frame when the tab is visible is to decode all the frames from the previous keyframe until you get to the right frame.


> 
> I'm not sure if this is really possible, especially since Videos contain
> lots of reference frames.

the problem is actually the opposite: what if the videos do not have lot of reference frame (and sometimes only one), which would require to decode a lot of frames until you get to the right one.
> Since the demuxer is running all the time, audio and video should stay in
> sync

the demuxing part is really secondary to the problem, it doesn't take long to demux a frame, plus frames demuxed are in decode order and doesn't give you much information on how they should actually be displayed.

> but I have never developed anything similar, so I'm looking forward to the
> devs' oppinions
> most of all I'm unsure whether it is even possible to combine this idea with
> DASH/MSE (YouTube)

we already have a working solution in bug 1224973.

The only issue we have right now is for small videos that only have a single keyframe at the start

Comment 14

[Djfe]

thx for your answer
it's good to know that you/someone is working on it :)

>the problem is actually the opposite: what if the videos do not have lot of reference frame (and sometimes only one), which would require to decode a lot of frames until you get to the right one.

actually I meant to say lots of referencing frames, ups ^^
(frames that refer to others)

anyways doesn't matter

what do you think about my suggestion to have a maximum number (a counter) of frames that aren't an I-Frame till the decoder has to run in the background regardless of the tab being invisible?

and do you think it's worth it to stop decoding audio? (it's not clear to me whether that is mentioned in the bug report you linked or not, I think when they talk about "muted", then the word is used in another context, right?)