Closed Bug 712478 Opened 12 years ago Closed 2 years ago

Need a global "user is now actively interacting with the app" state

Categories

(Core :: General, defect)

defect
Not set
normal

Tracking

()

RESOLVED INCOMPLETE
blocking-kilimanjaro -

People

(Reporter: taras.mozilla, Unassigned)

References

(Blocks 3 open bugs)

Details

(Whiteboard: [Snappy:P1])

When the user is interacting with Firefox, it would be helpful to try to suspend cycle-collection, limit events on background pages, etc.
Blocks: 712731
Blocks: 698547
Blocks: 715584
This is what we did in FF3.x for CC and we have still user-interaction-active/user-interaction-inactive
notifications.
Blocks: 715592
Whiteboard: [Snappy] → [Snappy:P1]
(In reply to Olli Pettay [:smaug] from comment #1)
> This is what we did in FF3.x for CC and we have still
> user-interaction-active/user-interaction-inactive
> notifications.

Is there a way we can poll for this? ie while making a decision on whether to run gc or postpone it.

I think this should be a tri-state:
User-inactive - proceed as normal
User-might-interact - try to avoid scheduling activities in near future(perhaps bump set-timeouts on background tabs)
User-interacting - suspend all possible gc, cc, settimeout/xmlhttprequest/etc on background tabs, etc.
(You can't suspend active xmlhttprequest)

Currently there is no way to poll the state. But that could be added easily.

user-interaction-active/user-interaction-inactive are sent based on whether user has 
caused any key or mouse events during last 5 seconds.

Note, my aim is to drop CC times to way lower than they are now.
I don't know what is happening with GC (which aren't often manually triggered).
I see GC time usually 3x a long as CC time. (well, atm GC is 5x CC )
I don't know how we could not GC so often.
I'd prefer to frame this not as "is the user interacting?" but as "how important is frame rate, right now?"  To ensure we can pick the "least bad" time to GC in a variety of situations, I think we'll need multiple tiers.

T1: User is interacting with something that animates.
* Scrolling
* Dragging tabs

T2: Something is animating.
* A video is playing and visible.
* A tab is closing.

T3: User input is imminent.
* A key is held down.
* A mouse button is held down.
* There has been user input in the last second.

T4: User input is likely.
* There has been user input in the last 5 seconds.
No longer blocks: 715584
Blocks: 675015
Maybe I'm over engineering this but perhaps instead of having hard rules like 'Don't do X while Y' with a bit more effort we could try to quantity the cost/benefit of an operation and only do it once we get over a certain threshold.

Whatever we do we need I think we need to be careful not to allow state(s) in which GC will never run because some users may spend a larger time in those states in certain edge cases that we anticipate thus allowing memory to increase dramatically.
I don't think that is a problem with GC since JS eng will just call GC when needed.
With CC it could be a problem. In FF3.x CC just run with lower probability if 
user was interacting with the browser.

But, if the becoming fixes to CC handling work as well as they do for me, we wouldn't, IMO, need to
change CC handling frequency because of user activity.
However we should certainly postpone any mainthread I/O to the times when user is inactive.
From the point of view of the markup parsers, the question isn't "Is it OK to do work now?" The question is "Given that I'm doing some work now, should I keep doing some more pending work or should I return to the event loop early?"

nsContentSink and interruptible reflow currently end up calling nsIWidget::HasPendingInputEvent(), which seems to answer roughly the right question, except it seems it doesn't answer the question on Mac. (From reading the source, it is unclear to me if on Windows mouse moves cause nsIWidget::HasPendingInputEvent() to return true when no mouse button is pressed.)

Maybe nsIWidget::HasPendingInputEvent() is already the right thing although it would be nicer to have the API on nsContentUtils instead of having to obtain nsIWidget from behind a couple of layers of indirection.
Exposing this on nsContentUtils would make a lot of sense to me.
Blocks: 716735
I think that a global "is user needs responsiveness" flag would suffice (without higher resolution states).

Since snappy aims at 100ms hangs at most (if I understand correctly), and assuming it's successful at that, I believe that such single global flag would eliminate the vast majority of related issues. Only if it doesn't, then a higher resolution (and complexity) approach should be considered IMHO.

Few suggestions:

1. Expose this functionality to Addons (both to poll/notified of this flag, and to affect it).

2. To not accidentally release this flag just when a user starts interaction again, release it on a timer (750ms? 7500ms?) after last known activity (and this timer gets extended on any new activity).

3. A simple API might be SetInteractingNow() (possibly called from many UI-related modules) and IsInteracting() (and possibly also a notification), combined with a timer from #2 above.

Modules who watch this flag can have for themselves a domain-specific threshold, after which the action will be taken regardless of this flag (however, I'd guess that on most cases this could hopefully be quite long).
Addons can use existing user-interaction-inactive/user-interaction-active notifications.
Re-assigning this to Bonnie, she's working on implementing a web-exposed idle API, and in doing that work she already has a helper on nsContentUtils that exposes whether the user is currently idle or not. My guess is that once bug 715041 is landed, we're done here as well.
Assignee: nobody → bsurender
Depends on: 715041
There is an IsUserIdle() function in nsContentUtils.cpp. This helper indicates whether the user is idle or not and will be available once the code has been committed.
Blocks: 733293
For a proof of concept we'll hang a the flag off nsIAppStartup and have scrolling + full screen invoke the interactive state.

As initial consumers of the service we'll look into session-store, idle* observers, thumbnail capture.
nsIAppStartup sounds very strange place to add such flag.
blocking-kilimanjaro: --- → ?
We are going to minus for now but please renom if we find it's a problem on real HW.
blocking-kilimanjaro: ? → -
GC? I thought this was a problem with I/O.

In my experience, FF hangs most when fetching content from the cache.

Also will this only be fixed on Windows 7 on x86 arch?
OS: Windows 7 → All
Hardware: x86 → All
(In reply to dE from comment #17)
> GC? I thought this was a problem with I/O.
> In my experience, FF hangs most when fetching content from the cache.
> Also will this only be fixed on Windows 7 on x86 arch?

Probably both. Cache issues are already greatly improved with Cache v2 which is already enabled by default on nightlies and should reach the release channel soon. I think it applies to all platforms.


As for GC/CC, I think I'll try to take this bug at some stage, and maybe also a derivative of it which will be pausing GC/CC during such activity if possible (i.e. as long as GC/CC can live with the lack of collections).

(In reply to Taras Glek (:taras) from comment #0)
> When the user is interacting with Firefox, it would be helpful to try to
> suspend cycle-collection, limit events on background pages, etc.

"user is interacting" is probably not the best correlation to what we care about here.

The reason we want to pause GC/CC is to increase responsiveness AND smoothness of stuff which happens on screen. The latter doesn't necessarily involve interactivity e.g. it also applies when there's some animation which doesn't involve user inputs.

I'll probably approach the "flag" aspect of this bug with some heuristics on the recent frequency of the refresh driver iterations. The reasons being:

1. GC/CC are main thread and therefore only hang the refresh driver and not off-main-thread animations (some CSS animations, video playback etc, async pan-zoom, etc)

2. As long as the refresh driver iterates frequently, it means that GC/CC cycles may hurt smoothness.

3. In case there are some user inputs which don't result in refresh driver iteration (e.g. maybe some mouse move or possibly APZC scroll, etc), the user won't notice GC/CC hangs anyway because there are no screen updates via the main thread.

So right now, recent refresh driver frequency (say, during the last 1-2-3 seconds) sounds to me like it could cover all the things we care about as far as GC/CC hangs go, and ignore exactly the things which we don't care about (off main thread display updates).

I've kickstarted some telemetry collection on GC/CC hangs during UI animations at bug 1017055, though right now that's on my backlog.
Blocks: 1031011

The bug assignee didn't login in Bugzilla in the last 7 months, so the assignee is being reset.

Assignee: bsurender → nobody

This is a fairly vague proposal, so I'll just close this. Hopefully there's been more recent work along these lines if it is useful.

Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.