646937 - throttle animation in completely obscured windows (virtual desktop, minimised, hidden behind other fullscreened windows)

Status: NEW

(Core :: Widget, defect, P3)

Description

[nemo]

When running compiz, and launching something expensive, like:
http://cjcliffe.github.com/CubicVR.js/cubicvr/tests/gpu_fluid/wavepool_blockers.html

The cpu usage remains high even if on a completely different desktop.
maniac103 of compiz says:
"< maniac103> nemo: when firefox gets the ConfigureNotify, it would need to check if its window rect intersects with the screen rect (the latter being 0, 0 - screen width, screen height)"

My request is that general paints and mozRequestAnimationFrame both be throttled way back if a window is minimised or not on current desktop.

Comment 1

[Karl Tomlinson (:karlt)]

Currently, if a toplevel window is unmapped (including minimized) or viewable and completely obscured (as it would be if outside the screen rect) general paints in response to invalidates are not performed at all.

WebGL and canvas are different because we have to have the canvas ready for when the window does become visible, and the canvas may also be used for non-display purposes.

mozRequestAnimationFrame could be throttled back.
This is not happening for mapped but completely obscured windows, but could be.
I don't know whether we throttle for minimized windows.

Comment 2

[Boris Zbarsky [:bzbarsky]]

> general paints in response to invalidates are not performed at all.

That's not what nemo was observing, iirc; he saw CPU usage drop when the canvas was set to visibility:hidden.

Right now we throttle mozRequestAnimationFrame for only background tabs.  There are requests to throttle for minimized windows and whatnot; what's needed there is a way to have the Firefox UI know when things are in an "invisible" state so it can mark the relevant tabs as background tabs.

Comment 3

[nemo]

Yeah, there were a few different things here.
One was about making Firefox aware that it shouldn't be painting when using compositing in metacity/compiz, and what the criteria for that would be.

The other would be making request animation frame aware of these and other things (like hidden visibility that only reduced the paint part of cpu, not the mozRequestAnimationFrame part) too, so that not only would paints be skipped, but the number crunching as well.

Comment 4

[nemo]

Sorry, to clarify that last comment slightly since I think I was being muddled as usual.

When compositing, in metacity or compiz, CPU is still burned in rendering even when firefox is on another desktop.  In metacity without compositing, it is not, but that's due to the gfx card, not Firefox being clever.  And even in the metacity w/o compositing case, Firefox stil called mozRequestAnimationFrame even though the final results ended up not being used.

Since compositing is pretty common now, it'd be nice to not have things suck up CPU when they are obscured.

For the objection of dimly visible things due to window previews or desktop switchers, I'm pretty sure most people wouldn't mind an approximation of the content for those :)

My linux experience would not be ruined if the Firefox displayed in Expose mode was not animating No Comply.

Comment 5

[Karl Tomlinson (:karlt)]

(In reply to comment #2)
> > general paints in response to invalidates are not performed at all.
> 
> That's not what nemo was observing, iirc; he saw CPU usage drop when the canvas
> was set to visibility:hidden.

When reading comment 1, please consider "general paints" as not including canvas.  But I misassumed that the canvas testcase was the only reason for the cpu use.

Under kwin with compositing too, it seems we don't get VisibilityNotify events when the window is moved outside the screen bounds.  (kwin uses a different approach for multiple desktops, which works well with our general paint suppression code.)

(In reply to comment #0)
> maniac103 of compiz says:
> "< maniac103> nemo: when firefox gets the ConfigureNotify, it would need to
> check if its window rect intersects with the screen rect (the latter being 0, 0
> - screen width, screen height)"

It would be unfortunate to need to modify all client apps to understand which part of the screen is visible in order to deal with changes in window managers.  I wonder why we don't get the VisibilityNotify events in that situation.  Perhaps it is because compiz wants to be able to show up-to-date previews of windows on other desktops, or maybe it is just the way RedirectWindow is implemented.

It sounds like we need a more general means for testing window visibility (VisibilityNotify or a replacement) than just checking screen bounds.

(In reply to comment #3)
> One was about making Firefox aware that it shouldn't be painting when using
> compositing in metacity/compiz, and what the criteria for that would be.

Would you mind checking for me please that you still see sustained cpu use on a different compiz desktop with http://blog.mozilla.com/2008/02/11/capital-n-small-y/
(That example doesn't use canvas.)

> The other would be making request animation frame aware of these and other
> things (like hidden visibility that only reduced the paint part of cpu, not the
> mozRequestAnimationFrame part) too, so that not only would paints be skipped,
> but the number crunching as well.

Yes, there are two issues here.  I guess we can use one bug until we know what the next step will be.

(In reply to comment #4)
> In metacity without compositing, it is
> not, but that's due to the gfx card, not Firefox being clever.

Without compositing, Firefox has VisibilityNotify events.

> And even in the
> metacity w/o compositing case, Firefox stil called mozRequestAnimationFrame
> even though the final results ended up not being used.

Yes.  It seems plausible that animation frames are not required if the window is not visible.

> For the objection of dimly visible things due to window previews or desktop
> switchers, I'm pretty sure most people wouldn't mind an approximation of the
> content for those :)

Yes, I assume previews of minimized windows just show old content.
Does Firefox's paint suppression code work with the animated images on dolske's blog and minimized windows in compiz?

Comment 6

[Boris Zbarsky [:bzbarsky]]

I think there are two pretty separate bugs here:

1)  Whatever is going on with normal painting under compositing window managers.
2)  Adding whatever widget-side hooks we need so that our UI can know when we're minimized or obscured or whatever other conditions we want so it can mark its tabs as hidden when that happens.

Comment 7

[Karl Tomlinson (:karlt)]

(In reply to comment #6)
Yes.

Comment 8

[Boris Zbarsky [:bzbarsky]]

I filed bug 675871 on the compositing window manager issue.

We also already throttle mozRequestAnimationFrame in minimized windows, so this just leaves windows not on the current desktop.

Comment 9

[Boris Zbarsky [:bzbarsky]]

We may need separate per-OS bugs here....

Comment 10

[Jim Mathies [:jimm]]

We send gecko events for size mode changes on all platforms [1], which sets an attribute on xul windows. Is this sufficient for detecting changes in window state in front end code?

[1] http://searchfox.org/mozilla-central/search?q=mWidgetListener-%3ESizeModeChanged&path=
[2] http://searchfox.org/mozilla-central/rev/944f87c575e8a0bcefc1ed8efff10b34cf7a5169/xpfe/appshell/nsXULWindow.cpp#1368

Comment 11

[Boris Zbarsky [:bzbarsky]]

We already handle this for sizemode changes (e.g. minimizing the window).  See http://searchfox.org/mozilla-central/rev/944f87c575e8a0bcefc1ed8efff10b34cf7a5169/browser/base/content/tabbrowser.xml#4906

This bug is about windows that are on non-default virtual desktops, obscured by other windows, etc.

Comment 12

[Jim Mathies [:jimm]]

(In reply to Boris Zbarsky [:bz] (still a bit busy) (if a patch has no decent message, automatic r-) from comment #11)
> We already handle this for sizemode changes (e.g. minimizing the window). 
> See
> http://searchfox.org/mozilla-central/rev/
> 944f87c575e8a0bcefc1ed8efff10b34cf7a5169/browser/base/content/tabbrowser.
> xml#4906
> 
> This bug is about windows that are on non-default virtual desktops, obscured
> by other windows, etc.

Ok, so as a general rule we've not prioritized virtual desktop issues, particularly on Windows. Should this bug really be a Photon P1? That seems a bit extreme considering our stance on these types of bugs in the past.

Comment 13

[nemo]

Yeah, I kinda feel the virtual desktop bit is why this languished for 6 years... But, alt-tab putting the browser behind your MS Outlook or whatnot isn't that uncommon right? In fact it seems like on Windows 10, and maybe due to mobile influence, switching between fullscreen windows is pretty much default.  No one bothers minimising.  So if you guys could detect you weren't foreground, you'd still avoid being a stealth battery suck.  People might not realise that browser 3 apps down the screen stack is what's taking 100% of a core.

Comment 14

[Boris Zbarsky [:bzbarsky]]

Right, "virtual desktop" is not the normal case.  The normal case is the user has three windows, all fullscreen in the usual way, and we shouldn't be painting the two that are not topmost all the time.

In terms of perception this is the "why is Firefox sucking up all my CPU when I'm not even using it?" thing.

Comment 15

[Hsin-Yi Tsai (she/her) [:hsinyi]]

This perhaps could utilize the method introduced in bug 1236512.  Samael is going to investigate first. Thanks!

Comment 16

[Samael Wang [:freesamael] (away for now)]

(In reply to Boris Zbarsky [:bz] (if a patch has no decent message, automatic r-) from comment #14)
> Right, "virtual desktop" is not the normal case.  The normal case is the
> user has three windows, all fullscreen in the usual way, and we shouldn't be
> painting the two that are not topmost all the time.
> 
> In terms of perception this is the "why is Firefox sucking up all my CPU
> when I'm not even using it?" thing.

Consider in bug 1236512 we've concluded that there's not much we can do to detect window occlusion on Windows & GTK, and on macOS we're going to make fully obscured windows to behave as background tabs, do you think there's anything worth further investigation in this bug?

Comment 17

[Samael Wang [:freesamael] (away for now)]

It's worth noting that Edge changes page visibility on Windows 10 tablet mode if it's not the active application (but only in tablet mode). We could probably try to do the same thing if we think tablet mode use case is important enough, but maybe that should be a seperate follow-up of bug 1236512.

Comment 18

[Boris Zbarsky [:bzbarsky]]

> do you think there's anything worth further investigation in this bug?

Probably not.  Making fully obscured windows behave as background tabs is the right call.  Not being able to detect them on non-Mac is really unfortunate, but not much we can do about it.

Comment 19

[Chris Peterson [:cpeterson]]

Windows and GTK don't have a window "occlusionstatechange" API like macOS, but we could still detect when a fullscreen browser window occludes another browser window on the same display. Admittedly this is probably an uncommon use case. Here is a test case:

STR:
1. Load http://daniemon.com/tech/webapps/page-visibility/
2. Play the video, which mutes/unmutes itself when the page visibility changes.
3. Open a new browser window. The video will continue playing audio.
4. Make the new browser window fullscreen (View > Full Screen menu or F11 keyboard shortcut).

RESULT:
The video continues to play audio, but Firefox can determine that the two browser windows are on the same display. If the second window is fullscreen, then other browser windows on the same display must be occluded and should have hidden page visibility.

Comment 20

[Boris Zbarsky [:bzbarsky]]

> then other browser windows on the same display must be occluded

Assuming that we can never have a partially-translucent fullscreen window... How sure are we that we can't?

Comment 21

[Andrew Overholt [:overholt]]

Given recent discussion here I think this should be qf:p2 at best.

Comment 22

[Mike Conley (:mconley) (:⚙️)]

This should probably be retriaged by the qf triage group, and categorized.