1688997 - Add Win32 widget support for occlusion events

Status: NEW

(Core :: Widget: Win32, enhancement, P2)

Description

[(No longer employed by Mozilla) Aaron Klotz]

It looks like the best way to do this would be for nsWindow to use IDXGIFactory2::RegisterOcclusionStatusWindow to register a custom notification message on its HWND.

When we see that message in our window procedure, we would then check whether we're occluded or not fire the event.

(Note that this API isn't implemented until Windows 8, but adequate error handling should be sufficient to deal with that.)

Comment 1

[(No longer employed by Mozilla) Aaron Klotz]

+cc Gijs, FYI. This might need an assist from GFX for an occlusion check from the DXGI swap chain, as this sample code illustrates.

Comment 2

[Christian Holler (:decoder)]

We are investigating this for Windows as part of the power consumption project, as this could be very relevant in particular when decoding video in the background. We already suspend video decoding when minimized or in background tab, but not then the window is completely occluded.

This works for other browsers I've tested on Windows and we can check how it's done there. In addition to the method in comment 0, there seems to be a non-event driven (much older) method here: https://devblogs.microsoft.com/oldnewthing/20030902-00/?p=42693 so we could probably even support this on Windows 7 if that is needed/desirable.

Comment 3

[(No longer employed by Mozilla) Aaron Klotz]

I tried the DirectX variant in a test program locally, and it did not appear to fire events as often as we'd like. I think that the DWM compositor makes the DirectX-generated events occur much less frequently.

I do agree that it would be worth looking into other techniques.

Comment 4

[Christian Holler (:decoder)]

Chromium has a fairly detailed description of their Windows implementation here:

https://chromium.googlesource.com/chromium/src.git/+/master/docs/windows_native_window_occlusion_tracking.md

We should investigate if we can adopt the same technique.

Comment 5

[(No longer employed by Mozilla) Aaron Klotz]

(In reply to Christian Holler (:decoder) from comment #4)

Chromium has a fairly detailed description of their Windows implementation here:

https://chromium.googlesource.com/chromium/src.git/+/master/docs/windows_native_window_occlusion_tracking.md

We should investigate if we can adopt the same technique.

Nice! Due to my GeckoView duties, I probably don't have the cycles to help implement this, however this issue is near and dear to my heart. I'm happy to review this in my role as a Win32 Widget peer. That said, based upon what I've seen in that document and what I know about our current state, I'm going to offer up a brain dump on what we already have, what we partially have, and what we're going to need.

It's pretty clear from both this reading, and my and decoder's research, that checking for occlusion is the easy part; it's doing so in an event-driven way that is hard. We probably should incorporate the DirectX-based events (that I mentioned above) as just another potential event source, along with the various other events that Chromium has already demonstrated as useful.

One dependency that we're absolutely going to need is better awareness of virtual desktops. Some support for this was added to session restore, but its support is incomplete and also has a performance bug. Sometime last year I stumbled across the source code for Microsoft's open-source FancyZones PowerToy, which was written by Microsofties who actually work on the Windows shell team. I don't think we could find a more authoritative reference implementation for managing virtual desktops, so I think we should use some of its concepts.

Speaking of which, one cannot discuss virtual desktops on Windows without discussing cloaking. We'd obviously want to incorporate cloaking events into our occlusion event framework, so maybe we should just dupe bug 1647161 over to this one and make sure cloaking events are added to the list?

Finally, receiving events when the screen is (un)locked: I already added rudimentary support for this in bug 1006765, so we'd just need to wire that into our occlusion event processor.

Comment 6

[Christian Holler (:decoder)]

I think these are all valid points and we need to investigate each of them. However, I think we should not dupe/mix these and instead try to do this somewhat incrementally. Doing the occlusion part for regular (non-virtual) desktops seems to be the most valuable part right now with immediate gains for a large amount of users. But that work can likely be extended later on to cover the other aspects.

Chris, could you look into this (in particular how Chromium does this, see comment 4) and give an estimate on how much work this would be to implement? Thanks!

Comment 7

[Chris Martin [:cmartin]]

Alrighty, I've finally had a chance to investigate this project.

As Christian pointed out to me, we already have infrastructure to handle occlusion events in Firefox - It's just that only Mac OS X actually fires them. So the effort here is generating these events when we're occluded on Windows.

I also tried the DXGI call Aaron talked about in Comment 1, and I agree that it seems like it doesn't report a lot of things, and it's not super-obvious why. This is probably why Chromium does it a different way.

I've had a chance to really go through the Chromium code and understand how it works and how difficult it would be to adapt to Firefox. Their description of it is a solid read, but to summarize:

At a high level it's mostly about installing Win32 global event hooks to monitor all the windows in the system for when they move/size/minimize/maximize/hide/show and periodically scheduling "occlusion checks" any time these things happen. (Chromium does these spaced out by at-least 16ms to avoid doing them more often than the user will be able to actually perceive).

An "occlusion check" is more-or-less just walking through all the windows from front-to-back and tracking the rectangles of the windows you've seen**. If you get to a window you care about and it's completely contained in the region of rectangles you've seen, it's occluded.

(Of course, the devil is in the details about what windows are able to occlude other windows, what windows are being checked, and managing the hooks and checks so you're not doing unnecessary work. A lot of the Chromium code is dealing with those issues. They also do the major calculations in a separate thread to keep it off the UI thread. I think we'd want to do the same.)

Overall though, it's not super-complex, but there's still a fair amount of parts to it. I'd estimate that it would take me about 4 weeks to write the code and have it working, and then a couple more weeks to test, document, debug, review, and polish the code up. So I'd say 6 weeks, but I tend to underestimate things a bit though, so to err on the side of caution maybe 7 or 8 weeks all-in.

Does that seem in the ballpark of what you expected?

** Technically, it's done in a negative-sense. You actually start off with the entire screen being the "unoccluded region" and subtract rectangles of occluding windows

Comment 8

[Sotaro Ikeda [:sotaro]]

Created a diagram around chromium's aura::WindowOcclusionTracker

• https://github.com/sotaroikeda/chromium-diagrams/blob/master/aura_WindowOcclusionTracker.pdf

Comment 9

[Mike Conca [:mconca]]

Sotaro, bug 1712854 looks like a dupe of this one. Can you take a look?

Comment 10

[Sotaro Ikeda [:sotaro]]

bug 1712854 seems basically dup of this bug. But I am not sure that infrastructure to handle occlusion events covers all of bug 1712854 yet.

Comment 12

[Emilio Cobos Álvarez (:emilio)]

Sotaro, make sure to ask QA to test Citrix and similar enterprise-y things, Chrome had a lot of issues with that, see https://bugs.chromium.org/p/chromium/issues/detail?id=1024837

Comment 13

[Emilio Cobos Álvarez (:emilio)]

(But it's great that we're doing this, thanks for working on it! :))

Comment 14

[Sotaro Ikeda [:sotaro]]

Attachment: WIP: Bug 1688997 - wip

Comment 15

[Sotaro Ikeda [:sotaro]]

(In reply to Emilio Cobos Álvarez (:emilio) from comment #12)

Sotaro, make sure to ask QA to test Citrix and similar enterprise-y things, Chrome had a lot of issues with that, see https://bugs.chromium.org/p/chromium/issues/detail?id=1024837

OK, thanks for the advice!

Comment 16

[Sotaro Ikeda [:sotaro]]

Created a diagram of current implementation.

• https://github.com/sotaroikeda/firefox-diagrams/blob/master/widget/widget_WinWindowOcclusionTracker_94.pdf

Comment 17

[Sotaro Ikeda [:sotaro]]

Just enabling window occlusion caused a lot of time out failures. Needs to think about how to run test.
https://treeherder.mozilla.org/jobs?repo=try&revision=16c4bc573147b78f232926b3cd0476986c51770e

Comment 18

[Sotaro Ikeda [:sotaro]]

I tested how requestAnimationFrame() and nsRefreshDriver with moz log "nsRefreshDriver" and with the following sample.
https://mdn.github.io/webgl-examples/tutorial/sample4/

When a window is occluded with enabling window occlusion tracking, period of requestAnimationFrame() and nsRefreshDriver were gradually longer and stopped finally. How they worked was same to a background tab of active window and a front tab of minimized window. It seems to related why a lot of tests were timed out.

• https://searchfox.org/mozilla-central/rev/a82dd14b6bc3d05a6a3facc126607e6d34cc2f53/layout/base/nsRefreshDriver.cpp#939

---

inactive timer next tick in 1000.000000 ms
inactive timer next tick in 2000.000000 ms
inactive timer next tick in 4000.000000 ms
inactive timer next tick in 8000.000000 ms
inactive timer next tick in 16000.000000 ms
inactive timer next tick in 32000.000000 ms
inactive timer next tick in 64000.000000 ms
inactive timer next tick in 128000.000000 ms
inactive timer next tick in 256000.000000 ms
inactive timer next tick in 512000.000000 ms
inactive timer next tick in 1024000.000000 ms
stopped to tick

Comment 19

[Sotaro Ikeda [:sotaro]]

(In reply to Sotaro Ikeda [:sotaro] from comment #17)

Just enabling window occlusion caused a lot of time out failures. Needs to think about how to run test.
https://treeherder.mozilla.org/jobs?repo=try&revision=16c4bc573147b78f232926b3cd0476986c51770e

mochi test "browser/components/search/test/browser/browser_contentSearchUI_default.js" was failed by timeout to wait !content.document.hidden. When window is occluded by other window, document.hidden is set to true.

wpt test "mozilla-central/testing/web-platform/tests/css/css-transitions/Element-getAnimations.tentative.html" seemed to be failed by timeout by slow nsRefreshDriver with InactiveRefreshDriverTimer.

Comment 20

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee didn't login in Bugzilla in the last months and this bug has priority 'P2'.
:spohl, could you have a look please?
For more information, please visit auto_nag documentation.

Comment 21

[Suhaib Mujahid [:suhaib]]

Sorry, there was a problem with the detection of inactive users. I'm reverting the change.