Need a global "user is now actively interacting with the app" state

NEW
Assigned to

Status

()

defect
7 years ago
2 years ago

People

(Reporter: taras.mozilla, Assigned: bsurender)

Tracking

(Blocks 7 bugs)

unspecified
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(blocking-kilimanjaro:-)

Details

(Whiteboard: [Snappy:P1])

(Reporter)

Description

7 years ago
When the user is interacting with Firefox, it would be helpful to try to suspend cycle-collection, limit events on background pages, etc.
(Reporter)

Updated

7 years ago
Blocks: 712731
(Reporter)

Updated

7 years ago
Blocks: 698547

Updated

7 years ago
Blocks: 715584
This is what we did in FF3.x for CC and we have still user-interaction-active/user-interaction-inactive
notifications.
(Reporter)

Updated

7 years ago
Blocks: 715592
Whiteboard: [Snappy] → [Snappy:P1]
(Reporter)

Comment 2

7 years ago
(In reply to Olli Pettay [:smaug] from comment #1)
> This is what we did in FF3.x for CC and we have still
> user-interaction-active/user-interaction-inactive
> notifications.

Is there a way we can poll for this? ie while making a decision on whether to run gc or postpone it.

I think this should be a tri-state:
User-inactive - proceed as normal
User-might-interact - try to avoid scheduling activities in near future(perhaps bump set-timeouts on background tabs)
User-interacting - suspend all possible gc, cc, settimeout/xmlhttprequest/etc on background tabs, etc.
(You can't suspend active xmlhttprequest)

Currently there is no way to poll the state. But that could be added easily.

user-interaction-active/user-interaction-inactive are sent based on whether user has 
caused any key or mouse events during last 5 seconds.

Note, my aim is to drop CC times to way lower than they are now.
I don't know what is happening with GC (which aren't often manually triggered).
I see GC time usually 3x a long as CC time. (well, atm GC is 5x CC )
I don't know how we could not GC so often.

Comment 4

7 years ago
I'd prefer to frame this not as "is the user interacting?" but as "how important is frame rate, right now?"  To ensure we can pick the "least bad" time to GC in a variety of situations, I think we'll need multiple tiers.

T1: User is interacting with something that animates.
* Scrolling
* Dragging tabs

T2: Something is animating.
* A video is playing and visible.
* A tab is closing.

T3: User input is imminent.
* A key is held down.
* A mouse button is held down.
* There has been user input in the last second.

T4: User input is likely.
* There has been user input in the last 5 seconds.

Updated

7 years ago
No longer blocks: 715584

Updated

7 years ago
Blocks: 675015
Maybe I'm over engineering this but perhaps instead of having hard rules like 'Don't do X while Y' with a bit more effort we could try to quantity the cost/benefit of an operation and only do it once we get over a certain threshold.

Whatever we do we need I think we need to be careful not to allow state(s) in which GC will never run because some users may spend a larger time in those states in certain edge cases that we anticipate thus allowing memory to increase dramatically.
I don't think that is a problem with GC since JS eng will just call GC when needed.
With CC it could be a problem. In FF3.x CC just run with lower probability if 
user was interacting with the browser.

But, if the becoming fixes to CC handling work as well as they do for me, we wouldn't, IMO, need to
change CC handling frequency because of user activity.
However we should certainly postpone any mainthread I/O to the times when user is inactive.
From the point of view of the markup parsers, the question isn't "Is it OK to do work now?" The question is "Given that I'm doing some work now, should I keep doing some more pending work or should I return to the event loop early?"

nsContentSink and interruptible reflow currently end up calling nsIWidget::HasPendingInputEvent(), which seems to answer roughly the right question, except it seems it doesn't answer the question on Mac. (From reading the source, it is unclear to me if on Windows mouse moves cause nsIWidget::HasPendingInputEvent() to return true when no mouse button is pressed.)

Maybe nsIWidget::HasPendingInputEvent() is already the right thing although it would be nicer to have the API on nsContentUtils instead of having to obtain nsIWidget from behind a couple of layers of indirection.
Exposing this on nsContentUtils would make a lot of sense to me.
(Reporter)

Updated

7 years ago
Blocks: 716735
I think that a global "is user needs responsiveness" flag would suffice (without higher resolution states).

Since snappy aims at 100ms hangs at most (if I understand correctly), and assuming it's successful at that, I believe that such single global flag would eliminate the vast majority of related issues. Only if it doesn't, then a higher resolution (and complexity) approach should be considered IMHO.

Few suggestions:

1. Expose this functionality to Addons (both to poll/notified of this flag, and to affect it).

2. To not accidentally release this flag just when a user starts interaction again, release it on a timer (750ms? 7500ms?) after last known activity (and this timer gets extended on any new activity).

3. A simple API might be SetInteractingNow() (possibly called from many UI-related modules) and IsInteracting() (and possibly also a notification), combined with a timer from #2 above.

Modules who watch this flag can have for themselves a domain-specific threshold, after which the action will be taken regardless of this flag (however, I'd guess that on most cases this could hopefully be quite long).
Addons can use existing user-interaction-inactive/user-interaction-active notifications.
Re-assigning this to Bonnie, she's working on implementing a web-exposed idle API, and in doing that work she already has a helper on nsContentUtils that exposes whether the user is currently idle or not. My guess is that once bug 715041 is landed, we're done here as well.
Assignee: nobody → bsurender
Depends on: 715041
(Assignee)

Comment 12

7 years ago
There is an IsUserIdle() function in nsContentUtils.cpp. This helper indicates whether the user is idle or not and will be available once the code has been committed.
(Reporter)

Updated

7 years ago
Blocks: 733293
(Reporter)

Comment 13

7 years ago
For a proof of concept we'll hang a the flag off nsIAppStartup and have scrolling + full screen invoke the interactive state.

As initial consumers of the service we'll look into session-store, idle* observers, thumbnail capture.
nsIAppStartup sounds very strange place to add such flag.
(Reporter)

Updated

7 years ago
blocking-kilimanjaro: --- → ?

Comment 15

7 years ago
We are going to minus for now but please renom if we find it's a problem on real HW.
blocking-kilimanjaro: ? → -

Updated

6 years ago
Duplicate of this bug: 824204

Comment 17

6 years ago
GC? I thought this was a problem with I/O.

In my experience, FF hangs most when fetching content from the cache.

Also will this only be fixed on Windows 7 on x86 arch?

Updated

6 years ago
OS: Windows 7 → All
Hardware: x86 → All
(In reply to dE from comment #17)
> GC? I thought this was a problem with I/O.
> In my experience, FF hangs most when fetching content from the cache.
> Also will this only be fixed on Windows 7 on x86 arch?

Probably both. Cache issues are already greatly improved with Cache v2 which is already enabled by default on nightlies and should reach the release channel soon. I think it applies to all platforms.


As for GC/CC, I think I'll try to take this bug at some stage, and maybe also a derivative of it which will be pausing GC/CC during such activity if possible (i.e. as long as GC/CC can live with the lack of collections).

(In reply to Taras Glek (:taras) from comment #0)
> When the user is interacting with Firefox, it would be helpful to try to
> suspend cycle-collection, limit events on background pages, etc.

"user is interacting" is probably not the best correlation to what we care about here.

The reason we want to pause GC/CC is to increase responsiveness AND smoothness of stuff which happens on screen. The latter doesn't necessarily involve interactivity e.g. it also applies when there's some animation which doesn't involve user inputs.

I'll probably approach the "flag" aspect of this bug with some heuristics on the recent frequency of the refresh driver iterations. The reasons being:

1. GC/CC are main thread and therefore only hang the refresh driver and not off-main-thread animations (some CSS animations, video playback etc, async pan-zoom, etc)

2. As long as the refresh driver iterates frequently, it means that GC/CC cycles may hurt smoothness.

3. In case there are some user inputs which don't result in refresh driver iteration (e.g. maybe some mouse move or possibly APZC scroll, etc), the user won't notice GC/CC hangs anyway because there are no screen updates via the main thread.

So right now, recent refresh driver frequency (say, during the last 1-2-3 seconds) sounds to me like it could cover all the things we care about as far as GC/CC hangs go, and ignore exactly the things which we don't care about (off main thread display updates).

I've kickstarted some telemetry collection on GC/CC hangs during UI animations at bug 1017055, though right now that's on my backlog.
Blocks: 1031011
You need to log in before you can comment on or make changes to this bug.