1429522 - Avoid using a transparent surface by using CoreAnimation

Status: ASSIGNED

Description

[Jeff Muizelaar [:jrmuizel]]

Transparent windows use too much power. See bug 1422090 and bug 1404042.

Comment 1

[Mark]

I wanted to give some feedback on the transparency & Core Animation issues from some tests I’ve been running. I wanted to measure the performance improvement of "opaque Firefox” and guess at how Core Animation might improve things further, by looking at Chrome with and without Core Animation.

(If you haven’t read bug 1422090 where this started, you’re probably not used to my immensely long posts. Sorry.)

I’ve got a very early Retina MacBook Pro so I presumably have the weakest Retina CPU+iGPU package (excluding MacBooks)

Note 1 gives details about CPU throttling and possible benefit of Core Animation (based on testing Chrome)

Turning off transparency and the tab throbber substantially reduces Firefox’s power consumption and prevents most fan noise even at the highest resolutions. It also prevents the huge page load slowdown at high scaled resolutions which happened when my CPU got drastically power starved by the GPU. But my fan will still spin up on repeated page loads at 1680x1050 because my GPU is still working pretty hard during page load (it seems to paint the whole Firefox window at ~30 fps as the content builds), and the CPU still gets somewhat power starved presumably by the GPU. But the page load slowdown is tens of percent rather than the hundreds of percent I was seeing with transparency enabled.

With transparency & the tab throbber disabled, watching a page load @ 1680x1050 on Intel Power Gadget logging I can see the CPU running at Turbo Boost, 2900 MHz, 12 W power for the first few tenths of a second, until (presumably) first paint, then the GPU power dominates as the page content builds and gets repeatedly painted. During this latter part of the page load the CPU gets power starved to 2400 MHz and ~7 W.  Using Quartz Debug > Flash Screen Updates I can see that the whole Firefox window is being updated like crazy throughout this part of the page load.

Animating at 1680x1050 scaled & 60 fps is still very challenging for my GPU even with "opaque Firefox" so continuous scrolling or on-page animations of any size consume a lot of power and get the jet engine fan going.

“Opaque Firefox" has page load and scroll power consumption broadly similar to Chrome with Core Animation disabled using —disable-mac-overlays. When I use Quartz Debug > Flash Screen Updates I can see Chrome —disable-mac-overlays updating the whole screen like crazy during page load, similar to Firefox. Which leads to power consumption similar to “opaque Firefox".

When I enable Core Animation in Chrome I see a substantial reduction in power consumption during page load. When I use Quartz Debug > Flash Screen Updates I see that the updates are piecemeal, a checkerboard of update flashes as different areas get updated only as needed. Full page scrolling is still very hard work for my GPU, but with Core Animation enabled, Chrome gets better fps with somewhat less GPU power. So I hope the same will be true for a “future Firefox” with Core Animation. (Of course I don’t if your Core Animation solution will be like Chrome’s).

Note 2 gives some data on battery life with and without transparency, and a guess at what Core Animation might do.

At my normal 1440x900 resolution with the 60 fps throbber enabled, "opaque Firefox" is a pleasure to use and doesn’t spin up my fan at all. The GPU rarely goes over 4 W. Transparent Firefox was awful, the GPU went up to 16 W and everything slowed down and got hot. Faster, cooler, battery lif-ier! I measured battery life for a page with a continuously running slideshow carousel and put the results in Note 2. Getting rid of transparency has given a big boost to battery life. I think Core Animation will do even more, particularly at high scaled resolutions. I think that Core Animation might make battery life much less dependent on scaled resolution. And at 1680x1050, based on Chrome with & without Core Animation, “future Firefox” might get double today’s battery life?????? YMMV

Of course WebRender is another story - it’s still a resource monster so turning it on more or less offsets the benefit of the opaque Firefox window!

I think fixing transparency & adding Core Animation to Firefox will make really dramatic improvements to Firefox's power consumption on Mac, prevent page load slowdown at high scaled resolutions, and perhaps improve scrolling fps at high scaled resolutions. Even on my anaemic Retina MBP.  


Note 1:- CPU & GPU power

1680x1050 resolution, full screen browser window, loading www.theguardian.com/uk (a fairly complex page), MacBook Pro 11,1, MacOS 10.13.3

Firefox Nightly; disable transparency and throbber using browser.tabs.hideThrobber TRUE and gfx.compositor.glcontext.opaque TRUE
    loading:- 
      Intel package draws ~27 W, CPU is power starved, page load is somewhat slow:- ~18 W GPU, ~6 W CPU, page load ~3 sec
    scrolling:- 
      ~18 W GPU, 50 fps, some jank

Chrome --disable-mac-overlays to prevent Core Animation (perhaps broadly comparable to Firefox Nightly as above)
    loading:- 
      Intel package draws ~27 W, CPU is power starved, page load is somewhat slow:- ~18 W GPU, ~6 W CPU, page load ~3 sec
    scrolling:- 
      ~18 W GPU, 50 fps, some jank

Chrome default settings, Core Animation is enabled (perhaps broadly comparable to “future Firefox with Core Animation"?)
    loading:- 
      Intel package draws ~17 W, CPU is not starved, page load is fast:  ~6 W GPU, ~11 W CPU, page load ~2 sec
    scrolling:-
      ~15 W GPU, 60 fps, noticeably smoother than above

I used Intel Power Gadget & iStat Menus to get powers; Quartz Debug to get fps.


Note 2:- battery life

Full screen browser window at various resolutions, loading bug 1419079 attachment 1 [details] [diff] [review] and leaving its slideshow carousel running. Screen 50% brightness,  MacBook Pro 11,1, MacOS 10.13.3

Battery life in hours, approx
                                           1440x900     1680x1050

Firefox, default, transparent window         4.5            3
Firefox, opaque window                       6              4.5
Chrome, Core Animation disabled              6              5
Chrome, Core Animation enabled               7              7

YMMV, particularly depending on content being displayed.

Comment 2

[Patrick Walton (:pcwalton)]

Proposed patch attached.

I've verified that this patch works with both regular OMTC layers and WebRender. The main issue remaining is that resizing is ugly. (It works, but it's racy during the actual resize process.)

Feedback is appreciated.

Comment 3

[Mark]

Do you have a Try version with this patch? I would like to have a look at power consumption & performance. Thanks.

Comment 4

[Timothy Nikkel (:tnikkel)]

Try push
https://treeherder.mozilla.org/#/jobs?repo=try&revision=d41538bef16cd7d6e79bdd5f9926a955cd4b0734

Comment 5

[Timothy Nikkel (:tnikkel)]

Try push that actually builds
https://treeherder.mozilla.org/#/jobs?repo=try&revision=7b875fd41f2a488b0f329b9ee4b74e8b2b657330

Comment 6

[Mark]

TL;DR

This Try doesn’t behave at all like I expected. Perhaps I misunderstood or am using it wrong. It’s weird.

Quartz Debug reports the Try window as opaque regardless of gfx.compositor.glcontext.opaque setting, I expected it to toggle opaque/transparent.

But the GPU power consumption is as though the Try build window is always transparent regardless of gfx.compositor.glcontext.opaque setting, I expected low GPU power consumption with the Try window set opaque and high with it set transparent.

And I don’t see the ability for the MacOS compositor to just animate small areas of the Firefox Try window, e.g. throbbers. Quartz Debug shows the whole Firefox Try window updating even when only a small animation (e.g. a throbber) is running. I expected Core Animation to mean that small animations only required only small areas of the screen to be updated, like Chrome. Not the case, so the Try still has very very high GPU power consumption compared to Chrome when animating small screen regions at high scaled resolution. 

This is all with WebRender OFF but I had a quick look and think the same is true with it on.


Long version

1) The Try window seems always opaque regardless of gfx.compositor.glcontext.opaque setting

Quartz Debug > Tools > Show Opaque Regions ticked

gfx.compositor.glcontext.opaque        Nightly 2018-09-12                 Try build
TRUE                                 shows up green - opaque       shows up green - opaque
FALSE                                shops up red - transparent    shows up green - opaque

Not what I expected, I expected gfx.compositor.glcontext.opaque to toggle green/red (opaque/transparent) on the Try build too.

Quartz Debug > Tools > untick Show Opaque Regions


2) But the GPU power consumption is as though the Try build is always transparent regardless of gfx.compositor.glcontext.opaque setting 

scrolling mozilla.org, full screen window, 1440x900 scaled resolution, MacBookPro11,1, GPU power from iStat Menus

gfx.compositor.glcontext.opaque        Nightly 2018-09-12                 Try build
TRUE                                        4 W                               15 W 
FALSE                                      15 W                               15 W

Not what I expected, I expected to see the Try build have low GPU power consumption with gfx.compositor.glcontext.opaque TRUE, like Nightly. The GPU power consumption results also seem to contradict the Quartz Debug results in (1) above:- if the Try window is opaque regardless of gfx.compositor.glcontext.opaque setting, shouldn’t the GPU power consumption always be low??


3) the whole Try window seems to still update however small the animating region

Quartz Debug > Tools > Flash Screen Updates 

whole window flashes while the tab throbber throbs

whole window flashes when the tiny live-pulse throbber is visible on attachment 8962814 [details]

attachment 8962814 [details] on Chrome shows just the tiny rectangle containing the live-pulse animation updating, Firefox Try shows the whole window updating. Chrome GPU power @1440x900 is <<1 Watt vs Firefox Try ~15 W (!!). Likewise the Firefox Try tab throbber still destroys the GPU and throttles my CPU at high scaled resolution, partly I think because issue (2) but also because issue (3) the whole window updating works the MacOS compositor too hard (IIUC?).  

I thought the idea with Core Animation was to allow the MacOS compositor to just update the small changing areas of the screen, not the whole thing?? 

cheers

Comment 7

[Markus Stange [:mstange]]

Thanks for testing this, Mark, but please be aware that this patch is only the first step. Using CoreAnimation will let us update parts of the window eventually, but only once we use different layers for different parts. The first step is just to get something running with a single layer that covers the entire window.

> 1) The Try window seems always opaque regardless of gfx.compositor.glcontext.opaque setting

You're right, the gfx.compositor.glcontext.opaque pref is ignored by the current patch. We should update the patch so that setting the pref causes us to call setContentsOpaque:YES on the CoreAnimation layer.

Another thing that's worth pointing out is that current vanilla Firefox does not have a transparent *window* - the bug summary is inaccurate. Instead, what happens is that Firefox publishes two *surfaces* to the window server:
 1. An opaque window surface (opaque except for the rounded corners), which is mostly black
 2. A transparent OpenGLContext surface on top of that, which contains the actual window contents.

The red that you see in Quartz Debug is the red from the OpenGLContext surface. If the OpenGLContext surface had holes in it you would see green highlighting from the opaque Firefox window underneath.

The power-wasting aspect of this is that the window server blends the OpenGLContext surface into the window surface on every composite for all pixels of the window, even though the OpenGLContext surface mostly contains opaque pixels. Setting gfx.compositor.glcontext.opaque to true eliminates this blending step because it turns the OpenGLContext *surface* completely opaque.

With CoreAnimation, we no longer publish two separate surfaces to the window server. Instead, we publish a CoreAnimation layer tree, which gets treated as one surface by the window server and only gets one highlighting overlay: green for the whole window, because the window itself is considered to be opaque.
This doesn't mean that the blending step is eliminated, though. The CoreAnimation layer tree still consists of multiple layers that need to be blended with one another. In order to eliminate this blending step, it's the *layer* that needs to be marked as opaque, and the window server does not seem to have a way of highlighting opaque or transparent *layers*, as far as I can tell.

Comment 8

[Mark]

Thanks Markus. As usual it's all much much more complicated than I realised. Who knew browers were so complicated??? I appreciate you taking the time to educate me.

I guess my (very blunt) comment would be:- if the aim of the patch is as stated "... a large battery life win" [by reducing GPU power at high scaled resolution] it doesn't work for me. The Try version uses lots of GPU power @1440x900 just like default "transparent" Firefox. Nightly set to "opaque" uses substantially less GPU power.

I expect I've misunderstood in which case apologies for my bluntitude and for taking your valuable time :)

Comment 9

[Mark]

I was just looking at the GPU load using Activity Monitor > Window > GPU usage   - the moving histogram.

If anything the Try build loads up my GPU more than vanilla "transparent" Nightly. Just counting the little blue squares on the histogram while scrolling mozilla.org @1440x900:- the Try uses about 15 squares vs around 6 squares for vanilla "transparent" Nightly.  I am not sure what the connection is between little blue squares and GPU power in watts, it seems quite subtle.

I also looked at FPS in Quartz Debug > Window > FrameMeter while scrolling mozilla.org at 1680x1050 and "transparent window" Nightly managed ~45 fps, the Try only managed ~35 fps. In both cases my GPU was melting at nearly 20 W.

So for me the Try is a step backwards from vanilla "transparent" Nightly in terms of GPU load & power. YMMV I guess.

I've probably misunderstood again, apologies in advance!

Comment 10

[Markus Stange [:mstange]]

Comment on attachment 9008569 [details] [diff] [review]
Proposed patch.

Review of attachment 9008569 [details] [diff] [review]:
-----------------------------------------------------------------

I haven't tested this, but the code looks good, as far as it goes. The commit message is rather inaccurate, though, see my other comment on this bug and Mark's findings.

I've filed a few bugs for the sub-steps that I have in mind for fixing this bug completely. Can you move this patch to bug 1491442 and put the important changes behind a pref check?

Comment 11

[Mark]

Sounds complicated

Can I strongly suggest:-

You guys at Mozilla should implement a Mac power consumption test protocol, particularly focused on the GPU. It sounds like some of the earlier bugs in your sequence won't change power consumption much, but I think you should get in the habit of running some standardized power tests each time you address one of these bugs.

A standardized test might be:- portable Retina Mac, speed scrolling mozilla.org full screen at different scaled resolutions, look at CPU and GPU power in [power app]. Then flip your pref or apply your patch and repeat. 

[power app] could be Intel Power Gadget but iStat Menus is nicer, it breaks out GPU power clearly and always runs in the menu bar.

I think you could do with some more subtle tests too. And across a range of machines say an early Retina MBP like mine; a fanless ULV MacBook; and maybe a high end rMBP with a dGPU; etc?

Don't rely on Some Guy on the Internet (me) to do your power testing. It's too important. I am just a slacker liable to wander off and lose interest. Or I could go under a bus.

And to emphasise:- by power I mean actual watts, not a proxy like ms on the GPU or blue squares in Activity Monitor. They're good too, but this is all about actual power out of the battery and into the heatsink. If a patch doesn't reduce watts, don't implement it. Watts watts watts watts watts.

Comment 12

[Markus Stange [:mstange]]

We're not relying on you to do our power testing, don't worry.

Comment 13

[Mark]

I always worry.

Another good power test would be a small throbber on a static background. But you know that.

Comment 14

[Patrick Walton (:pcwalton)]

I have a new version of the patch that makes the layer opaque and also fixes the resizing issues. Some quick verification seems to show that the power consumption when scrolling is significantly reduced with that.

I will post it once I have it updated to go behind a pref.

Comment 15

[Liz Henry (:lizzard) (use needinfo)]

Any chance of landing this for 64? Or do you think it is better to wait to land it in 65 (and then consider uplift)

Comment 16

[Markus Stange [:mstange]]

This is not going to be in a state to turn it on for users in time for 64.

Comment 17

[Gorjan Jovanovski]

Any updates on this?

Comment 18

[Markus Stange [:mstange]]

Work has begun.

Comment 19

[charlie.siegel]

(In reply to Markus Stange [:mstange] from comment #18)

Work has begun.

Excellent! That's great news. Thank you.

Comment 20

[mb]

Any chance of getting this bumped to P1? Patrick in comment 14 said that there was some verification that he had a patch that worked. Has a pref been created for Nightly users to test out on MacOS since then?

Comment 21

[Markus Stange [:mstange]]

No it has not; I'm currently fixing the glitches from that patch and will provide an update once you can try out the pref.

Comment 22

[Markus Stange [:mstange]]

Rounded bottom corners have been the default since 10.7.

Depends on D22642

Comment 23

[Markus Stange [:mstange]]

Oops, that patch shouldn't have gone to this bug.