1038527 - Reduce resource consumption <video> elements in Desktop Firefox

Status: NEW

(Core :: Audio/Video: Playback, defect, P3)

Description

[Chris Pearce [:cpearce (Not reading bugmail)]]

We need to reduce the memory and GPU resources consumed/retained by <video> elements when they're not playing.

This blocks enabling HTML5 Media Source Extensions <video> on YouTube, Encrypted Media Extensions, and widespread adoption of HTML5 <video> in general.

This is a spin off of bug 963922 and big 962986. I'm filing here to keep the noise of the other bugs out of the technical discussion.

This work is to be carried out primarily with Desktop Firefox on Windows in mind, but also other tier-1 desktop platforms too.

Use cases we want to support:

1. We should continue to support multiple <video> elements playing concurrently in a single page; we don't want to limit desktop to a single active <video> element in the foreground tab like we do on FirefoxOS.
2. Support the user opening dozens of tabs with HTML5 videos in them and waiting for them to download background tabs. Video ads loading in background tabs also applies to this use case.
3. Support the 4chan forums use case; 4chan now allow WebM <video>s instead of animated gifs on forum posts, and this may result in a single web page with hundreds of videos playing in a loop. We don't want videos which aren't visible as the user scrolls down the page to be consuming resources.
4. Don't crash on webpages that create a thousand 480p <video> elements.

We want the above use cases supported for WebM and MP4 in desktop Firefox, in traditional HTML5 <video>, as well as Media Source Extensions <video>.

This will also help with reducing the load on TBPL mochitests.

We need the above working in the WebMReader and MP4Reader. Long term strategy is to switch to MP4Reader for all MP4 playback, so there's no point worrying about fixing the other desktop MP4 backends. We'll also need this plumbed through in the MediaSourceReader, which operates a *set* of sub-readers, which will complicate things. Don't worry about the other MediaDecoderReaders.

We have code that makes MediaOmxReader "dormant" when the corresponding video element is no longer in the focused tab. Sotaro wrote that code. I assume we'll use or modify that to suit the above use cases, and implement it in MP4Reader and WebMReader.

I propose we achieve the above use cases by doing the following:

* Make WebM/MP4 video elements that reach ended state dormant, possibly after a timeout.
* Make video elements that are paused and not visible (either out-of-view in a foreground tab, or in a background tab) dormant. They don't need to be made dormant immediately, we can set a 60s timer to make them dormant when a video becomes paused and/or not visible.
* Create some kind of measure to determine whether we're "low on video decoding resources".
* When we detect that we're running low on resources, make existing paused video elements dormant. If we're still low on resources, we'll need to more aggressively make dormant, or even terminate, playing video elements. This is usecase 3 and 4 above.

Comment 1

[Chris Pearce [:cpearce (Not reading bugmail)]]

We should also ensure that video elements that go into the bfcache are made dormant.

Comment 2

[Milan Sreckovic [:milan] (needinfo for best results)]

Figuring out how to break this, :jet's suggestion was:
1. Apply the frame visibility checks in bug 1002992 to <video> elements. 
2. Apply throttling to non-visible <video> elements 
3. UX bug: define throttle behavior for non-visible <video> elements

any other suggestions for a breakdown?

Comment 3

[Chris Pearce [:cpearce (Not reading bugmail)]]

We could get most of the benefits that we need by cutting scope to instead make a media element dormant when any of the following are true:

1. The media element is a video element and is in a background tab and has been paused or ended for at least 60s.
2. The media element is a video or audio element and has moved into the bfcache (i.e. made inactive).

That drops the stress-test and visibility issues from our use cases. It should be quite easy to do this.

When a video element is dormant on desktop, we should retain the current Image in the VideoFrameContainer, so we can still render it when the tab is brought into the foreground.

Comment 4

[Benjamin Smedberg]

What is the user-visible impact of making a video element dormant? Does it just take longer to resume playing when you switch back to that tab and click play?

Is "resource consumption" CPU or memory or network bandwidth?

Comment 5

[Chris Pearce [:cpearce (Not reading bugmail)]]

(In reply to Benjamin Smedberg  [:bsmedberg] Away 19-July through 3-Aug from comment #4)
> What is the user-visible impact of making a video element dormant? Does it
> just take longer to resume playing when you switch back to that tab and
> click play?

Yes it would take longer to resume. We'd need to re-start the media load in this situation.
 
> Is "resource consumption" CPU or memory or network bandwidth?

CPU memory is the concern here, but also the DXVA decoder GPU memory in the MP4 case.

Comment 6

[Benjamin Smedberg]

I'm also worried about the memory consumption associated with active videos. But apart from that, can we before making it dormant pre-decode a little bit of video, so that if/when we want to resume, we can do that immediately while we're restarting the load?

Comment 7

[Sotaro Ikeda [:sotaro]]

> enabling HTML5 Media Source Extensions <video> on YouTube

cpearce, I have a very basic question. How can I confirm MES youtube on desktop Firefox?

Comment 8

[Chris Pearce [:cpearce (Not reading bugmail)]]

Disable flash plugin. Set pref media.mediasource.enabled to true. Open youtube.com/html5, opt in.

Set a break point in MediaSourceReader to be extra sure.

Comment 9

[Sotaro Ikeda [:sotaro]]

(In reply to Chris Pearce (:cpearce) from comment #8)
> Disable flash plugin. Set pref media.mediasource.enabled to true. Open
> youtube.com/html5, opt in.
> 
> Set a break point in MediaSourceReader to be extra sure.

Thanks!

Comment 10

[Chris Peterson [:cpeterson]]

These bugs are fit and finish issues that might block EME uplift to Aurora.