1851288 - High CPU usage in idle state even when starting in Troubleshoot Mode since Firefox 118

Status: RESOLVED FIXED

(Core :: Disability Access APIs, defect)

Description

[[:FSpark]]

Attachment: HIgh CPU Usage

Steps to reproduce:

Simply open Firefox Developer Edition 118.0b3 on Windows 10 21H2 (my enviorment).

It has never occurred before upgrading to version 118. The issue persists even when starting Firefox in Troubleshoot Mode, with all extensions and personal configurations temporarily disabled.

I'm not sure how to debug or pinpoint the issue. I have used Firefox Profiler and examined the flame graph, and it seems that I have identified functions like ZwQuerySystemInformation that are causing high usage. Link is here: https://share.firefox.dev/3R5EcUJ

Please let me know if there's anything else I can assist you with.

Actual results:

1. Observe the Process Manager (about:processes) where the CPU usage of the Firefox main process remains consistently around 100%.
2. Attempt to browse webpages, experiencing significant lag.
3. When the focus is shifted away from the Firefox window, CPU usage immediately drops back to normal levels.

Expected results:

There is no longer high CPU usage even during idle periods.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

This bug was moved into the Performance component.

:FSpark, could you make sure the following information is on this bug?

• ✅ For slowness or high CPU usage, capture a profile with http://profiler.firefox.com/, upload it and share the link here.
• For memory usage issues, capture a memory dump from about:memory and attach it to this bug.
• Troubleshooting information: Go to about:support, click "Copy raw data to clipboard", paste it into a file, save it, and attach the file here.

If the requested information is already in the bug, please confirm it is recent.

Thank you.

Comment 2

[[:FSpark]]

Attachment: support.json

Comment 3

[[:FSpark]]

Attachment: memory-report.json.gz

Comment 4

[[:FSpark]]

updated to 118b4, same issue, latest profiler: https://share.firefox.dev/3sLwzIX

Comment 5

[Sean Feng [:sefeng]]

Hi, looks like it's related to a11y features.

If this is an regression and you can reproduce this reliably, do you mind to use https://mozilla.github.io/mozregression/ to bisect this? It'd be extremely helpful.

Thanks

Comment 6

[[:FSpark]]

Hi, thank you. I have successfully located the version using bisect, and it does seem to be related to a11y, especially this bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1847489 .

Here are the logs. This bug really impacts my usage of Firefox😢.

build_date: 2023-08-10 02:53:58.812000
build_file: b2b54f91567d--autoland--target.zip
build_type: integration
build_url: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/aqxmoYmNTE6EfmwwKr7vqw/runs/0/artifacts/public%2Fbuild%2Ftarget.zip
changeset: b2b54f91567d2de224314e03fc81ba6e815e2fa7
pushlog_url: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=b2b54f91567d2de224314e03fc81ba6e815e2fa7&tochange=3dace62646004f0ac659dab97b9e1e139f854211
repo_name: autoland
repo_url: https://hg.mozilla.org/integration/autoland
task_id: aqxmoYmNTE6EfmwwKr7vqw

2023-09-09T13:52:09.305000: INFO : Narrowed integration regression window from [f2506aaa, 3dace626] (3 builds) to [b2b54f91, 3dace626] (2 builds) (~1 steps left)
2023-09-09T13:52:09.327000: DEBUG : Starting merge handling...
2023-09-09T13:52:09.327000: DEBUG : Using url: https://hg.mozilla.org/integration/autoland/json-pushes?changeset=3dace62646004f0ac659dab97b9e1e139f854211&full=1
2023-09-09T13:52:09.328000: DEBUG : redo: attempt 1/3
2023-09-09T13:52:09.328000: DEBUG : redo: retry: calling _default_get with args: ('https://hg.mozilla.org/integration/autoland/json-pushes?changeset=3dace62646004f0ac659dab97b9e1e139f854211&full=1',), kwargs: {}, attempt #1
2023-09-09T13:52:09.338000: DEBUG : urllib3.connectionpool: Resetting dropped connection: hg.mozilla.org
2023-09-09T13:52:10.928000: DEBUG : urllib3.connectionpool: https://hg.mozilla.org:443 "GET /integration/autoland/json-pushes?changeset=3dace62646004f0ac659dab97b9e1e139f854211&full=1 HTTP/1.1" 200 None
2023-09-09T13:52:11.158000: DEBUG : Found commit message:
Bug 1847489: Detect UIA clients in Windows 10. r=nlapre

The new detection code introduced in bug 1838123 doesn't work on Windows 10.
This patch:

1. Splits the Windows 11 code into its own function.
2. Refactors the system handle enumeration code into its own function which can be called with a lambda, since it is needed for both Windows 11 and Windows 10.
3. Adds code to detect clients on Windows 10 based on the old detection code before bug 1838123, with some noteworthy changes:
    - Hooking the UIA window message doesn't work; our hook runs too late. It also doesn't work well for blocking; some clients will very likely poke us more than the maximum attempts in the old code (5 times).
    - Instead, we run this code as part of LazyInstantiator::ShouldInstantiate, just as we do for all other client detection.
    - This means we use the same UIA detection caching strategy; i.e. reset on foreground changes.
    - It also means we reuse the same instantiator setting and block listing code in LazyInstantiator.

Differential Revision: https://phabricator.services.mozilla.com/D185627

2023-09-09T13:52:11.158000: DEBUG : Did not find a branch, checking all integration branches
2023-09-09T13:52:11.161000: INFO : The bisection is done.
2023-09-09T13:52:11.162000: INFO : Stopped

Comment 7

[James Teh [:Jamie]]

[Tracking Requested - why for this release]: Severe performance problems for some users.

Comment 8

[James Teh [:Jamie]]

I don't know why this is happening, but you're definitely correct that this is caused by bug 1847489. Thank you for your investigation efforts.

One thing I can say for sure is that it seems you have some software on your system which is repeatedly trying to access Firefox using the UI Automation API. UI Automation is primarily used by accessibility tools, but it is increasingly used by other things as well. Do you have any accessibility tools, enterprise SSO tools, etc. installed that might be trying to access the browser UI or web content?

Comment 9

[James Teh [:Jamie]]

I also see that you have the accessibility.force_disabled preference set to 1. Can you explain why? Have you had issues with something triggering accessibility in the past?

I suspect this might be due to an unanticipated interaction between the code in bug 1847489 and the force_disabled preference.

Comment 10

[James Teh [:Jamie]]

Note that setting accessibility.force_disabled to 0 and restarting Firefox might allow Firefox to detect which is the problematic UI Automation client. If it can, that information will be available in the Accessibility section of about:support under "Accessibility instantiator".

Comment 11

[James Teh [:Jamie]]

Attachment: Bug 1851288: Cache when a UIA client is not on the block list.

Previously, we cached when a UIA client was blocked or when there were no UIA clients.
However, we did not cache the result when a UIA client was present but not blocked.
This isn't normally a problem because a11y is normally instantiated in this case, which means we won't try to do any client detection again this session.
However, if a11y is force disabled via the pref, we still do detection, but we don't instantiate.
This meant that a UIA client which hammered us with queries would keep triggering the detection code, since we weren't caching the result.
That resulted in severe performance degradation for impacted users.
To fix this, cache the UIA detection result even if we do allow a11y instantiation.

Comment 12

[James Teh [:Jamie]]

Triage note: This is not an s1 because it only impacts users with a11y force disabled (which isn't really an officially supported configuration). It also only impacts users running UI Automation clients which deliberately and continually hammer us with queries. Given that, we could perhaps even downgrade this to an s3, but I'll leave it as s2 for now in case there are more users with such configurations than we think.

Comment 13

[BugBot [:suhaib / :marco/ :calixte]]

The bug has a release status flag that shows some version of Firefox is affected, thus it will be considered confirmed.

Comment 14

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1847489

Comment 15

[[:FSpark]]

Hi, as you mentioned, setting accessibility.force_disabled to 0 and restarting Firefox resolves the issue! Thank you very much!

There is only one Accessibility instantiator, which is "UIAUTOMATION|D:\Program Files\ImTip.exe" from https://github.com/aardio/ImTip. It is a universal input method status tracking and prompt tool.

As for why accessibility.force_disabled needs to be set to 0, it's because this practice has been popular for several years to prevent Firefox memory leaks by disabling accessibility features. You can easily find this information on Google like https://www.ghacks.net/2021/08/25/firefox-tip-turn-off-accessibility-services-to-improve-performance . Based on this, I believe there are still many users who use this configuration.

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d165459bd2e8
Cache when a UIA client is not on the block list. r=nlapre

Comment 17

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/d165459bd2e8

Comment 18

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:Jamie, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
• If no, please set status-firefox118 to wontfix.

For more information, please visit BugBot documentation.

Comment 19

[James Teh [:Jamie]]

Comment on attachment 9352422 [details]
Bug 1851288: Cache when a UIA client is not on the block list.

Beta/Release Uplift Approval Request

• User impact if declined: Severe performance degradation for some users.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Straightforward patch which only impacts UI Automation client detection.
• String changes made/needed:
• Is Android affected?: No

Comment 20

[Pascal Chevrel:pascalc]

Comment on attachment 9352422 [details]
Bug 1851288: Cache when a UIA client is not on the block list.

Approved for 118.0b9 (last beta) thanks.

Comment 21

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/6896b3825e1f

Comment 22

[Pascal Chevrel:pascalc]

https://hg.mozilla.org/releases/mozilla-beta/rev/6896b3825e1f