1418290 - [meta] Reports about serious performance regressions after the 57 update from users with a11y turned on

Status: RESOLVED FIXED

(Core :: Disability Access APIs, defect, P3)

Description

[[:philipp]]

[Tracking Requested - why for this release]:
we're seeing reports from users on various support channels that they are experiencing serious performance problems after the update to firefox 57, like multi-second freezes/hangs/lags after all sorts of user actions (opening tabs, scrolling, typing).

we have performance profiles from three affected users - i am no expert in interpreting those but as it seems to me, the lag there is mainly taking place in functions relating to accessibility:
*"Opening new tab causes intense lag": https://perfht.ml/2AWi9GO - https://support.mozilla.org/en-US/questions/1186701
*"FF57 lagging on everything?": https://perfht.ml/2APLjX7 - https://www.reddit.com/r/firefox/comments/7dec8t/ff57_lagging_on_everything/
* https://perfht.ml/2zA5ifX - https://www.camp-firefox.de/forum/viewtopic.php?f=1&t=122400

the amount of questions we get about this problem is worryingly high i'd say and most users probably aren't aware that accessibility functions are enabled in their profile, which may be a separate issue...

Comment 1

[Marco Zehe (:MarcoZ)]

Note that perf problems are documented here, among others: https://support.mozilla.org/en-US/kb/can-i-use-my-screen-reader-new-firefox

So the main question is, what client is triggering accessibility for these users. Note also that there's a checkbox in Advanced Settings telling Firefox to block accessibility from being turned on.

Comment 2

[[:philipp]]

i'll be tagging all the user questions at sumo that i can find that have information on them that accessibility is turned on and they experience serious lags at https://support.mozilla.org/en-US/questions/firefox?owner=all&tagged=bug1418290&show=all

Comment 3

[Jim Mathies [:jimm]]

Phillip, fyi there's an option under privacy for this now. You don't have to tell users to manipulate about:config.

https://support.mozilla.org/en-US/questions/1185780

Comment 4

[[:philipp]]

ok, i see (& will adapt the reply in future responses). i mainly ni'd you as the regression engineering "boss" for 57, because you may know better who this should be escalated to.

Comment 5

[Jim Mathies [:jimm]]

Two of the users who posted about:support text have accessibility enabled. Both are unknown injected clients.

Our support approach here is mostly correct, ask users to disable and see if it fixes the issue. However we should be careful about suggesting this to users we detect are running a known screen reader - disabling a11y for these users will complete break their ability to interact with Firefox. This information would be under the accessibility about:support section.

Also we should follow these support incidents up with a question about what accessible software they run.

Comment 6

[David Bolter [:davidb] (NeedInfo me for attention)]

Let's try to get STR of course.

In particular the about:support accessibility section as jimm mentions in comment 5 might help.

Similar information can come from "menu ≡ > help ? > troubleshooting information" which philipp mentioned in a support incident: https://support.mozilla.org/en-US/questions/1185780.

The relevant section is:
 "accessibility": {
   "isActive": true,
   "forceDisabled": 0,
   "handlerUsed": true,
   "instantiator": "UIAUTOMATION|"
 }

The important bit is "instantiator". We don't know all the things that cause UIAUTOMATION to kick in, but it sounds like a lot of users are hitting this so hopefully we can discover a case locally soon.

Comment 7

[David Bolter [:davidb] (NeedInfo me for attention)]

Put another way, we need to know what unexpected software is causing accessibility to instantiate and use the API so heavily that these users are noticing such a drastic regression.

Comment 8

[(No longer employed by Mozilla) Aaron Klotz]

Looks like the right info is in place for this, clearing ni

Comment 9

[Liz Henry (:lizzard) (relman/hg->git project)]

Tracking for 57.

Comment 10

[[:philipp]]

a user at https://support.mozilla.org/en-US/questions/1187076 is getting fortinet av software as the "instantiator". incidentally he is also the 3rd user with this problem explicitly referencing having a surface 4 pro device...

Comment 11

[Jim Mathies [:jimm]]

(In reply to [:philipp] from comment #10)
> a user at https://support.mozilla.org/en-US/questions/1187076 is getting
> fortinet av software as the "instantiator". incidentally he is also the 3rd
> user with this problem explicitly referencing having a surface 4 pro
> device...

Unfortunately there are a lot of these types of applications out there. We're planning on using various sticks and carrots to try and persuading these vendors to move to a more appropriate extension mechanism.  The new browser window 'accessibility indicator', which should be rolling out in 58 or 59, will be the first step in this process.

Comment 12

[Kohei Yoshino]

Some people on Twitter say RealPlayer is the culprit. One person says disabling a11y services disabled the RealPlayer add-on and solved the perf issue.

Comment 13

[Jim Mathies [:jimm]]

(In reply to Kohei Yoshino [:kohei] from comment #12)
> Some people on Twitter say RealPlayer is the culprit. One person says
> disabling a11y services disabled the RealPlayer add-on and solved the perf
> issue.

Interesting! Thanks for the data point. I'll file a bug on that.

Comment 14

[[:philipp]]

would https://mzl.la/2zOShhS indicate that this is affecting 7%+ of the userbase?

Comment 15

[Kohei Yoshino]

In that case we should remotely disable the a11y service right away by flipping the pref...

Comment 16

[Ben Kelly [:bkelly, not reviewing]]

I got an about:support from one user affected by this in bug 1419028.

Comment 18

[Ben Kelly [:bkelly, not reviewing]]

One user that ran into this problem said it only affected their newer model HP laptop.  Their old laptop was not affected.  I wonder if we could test drive one of those at a store to see if it reproduces and get it in house to diagnose the issue.

Comment 19

[DDM]

On bug1418749 -- Problem was with RealPlayer / RealDownloader .From Kohei Yoshino [:kohei]Comment 12 ; Turning off accessibility services disables the downloader and resolves the performance issue. 
Will there be a fix for this ? RealDownloader is the only one I have found that can perform almost all of my needs.

Comment 20

[Kohei Yoshino]

Please ask RealNetworks.

Comment 21

[DDM]

OK . Thanks I will try to contact them .

Comment 22

[Ben Kelly [:bkelly, not reviewing]]

Here is a user with a slightly different report, but solved by disabling a11y in options:

https://twitter.com/sigfig/status/933378222594646018

They report sites would not complete loading until the defocused the window.

Comment 23

[Dan]

58 (Nightly) currently also affected by this on my PC. (also Beta). We need hotfix asap, this really ruins YouTube browsing experience.

Comment 24

[James Teh [:Jamie]]

(In reply to Daniel from comment #23)
> 58 (Nightly) currently also affected by this on my PC. (also Beta). We need
> hotfix asap, this really ruins YouTube browsing experience.

Does disabling accessibility help in your case? Are you able to provide any insight as to what software might be running on your system that might be using accessibility services? This could include any software which searches for content on web pages. One example is RealPlayer; see bug 1418535.

Comment 25

[Dan]

No it didn't help. Btw I also ran Gecko Profiler in Nightly, disabled the accessibility, this is my result: https://perfht.ml/2BgfpEo I will post this result also in my bug report ( https://bugzilla.mozilla.org/show_bug.cgi?id=1419833 ).

Comment 26

[James Teh [:Jamie]]

(In reply to Daniel from comment #25)
> No it didn't help.

Okay. That would suggest this particular case is not accessibility related.

Comment 27

[Dan]

James the only other software I run on my PC are AVG free antivirus and Windows Defender (Microsoft Antivirus). My system is Win 10 Fall Creators update, 64 bit. Or could windows defender smartscreen suddenly cause problems? I really hope the Gecko profile will help understand the problem :)

Comment 28

[(No longer employed by Mozilla) Aaron Klotz]

The profile helps a lot, thank you. Your problem is actually not related to accessibility, but is rather a layout problem.

I'll update your bug; this one is a different issue.

Comment 29

[Dan]

Thanks very much Aaron! :)

Comment 30

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

This issue has been actively investigated since unthrottling updates. We have put some mitigations in place and continue to find other low risk/high reward fixes.

Comment 31

[Kohei Yoshino]

(In reply to [:philipp] from comment #14)
> would https://mzl.la/2zOShhS indicate that this is affecting 7%+ of the
> userbase?

This data still indicates 6.83% as of today. According to Bug 1421010, only 0.3% of users have RealPlayer installed. I see people on Twitter still say Firefox is so slow after upgrading. Maybe whitelisting common a11y services would be better than blacklisting some rogue external apps?

Comment 32

[Jim Mathies [:jimm]]

(In reply to Kohei Yoshino [:kohei] from comment #31)
> (In reply to [:philipp] from comment #14)
> > would https://mzl.la/2zOShhS indicate that this is affecting 7%+ of the
> > userbase?
> 
> This data still indicates 6.83% as of today. According to Bug 1421010, only
> 0.3% of users have RealPlayer installed. I see people on Twitter still say
> Firefox is so slow after upgrading. Maybe whitelisting common a11y services
> would be better than blacklisting some rogue external apps?

White listing isn't a short term solution here but it is something we're looking at. We're also shipping the a11y indicator and working on general performance of the a11y code base to address.

FYI we have telemetry evidence that this is limited to a small subset of users. We're still researching to fully understand the scope though.

Comment 33

[Liz Henry (:lizzard) (relman/hg->git project)]

Tracking for 58 and 59. It is likely too late for a fix in 57 based on the number of dependent issues here.

Comment 34

[Liz Henry (:lizzard) (relman/hg->git project)]

This was relnoted for 57.0 with advice for screen reader users to switch to ESR 52: https://www.mozilla.org/en-US/firefox/57.0/releasenotes/

Comment 35

[Todd]

(In reply to Kohei Yoshino [:kohei] from comment #12)
> Some people on Twitter say RealPlayer is the culprit. One person says
> disabling a11y services disabled the RealPlayer add-on and solved the perf
> issue.

+1

Comment 36

[James Teh [:Jamie]]

(In reply to Todd from comment #35)
> > Some people on Twitter say RealPlayer is the culprit. One person says
> > disabling a11y services disabled the RealPlayer add-on and solved the perf
> > issue.
> +1

Todd, RealPlayer was blocked in Firefox 57.0.1. Are you still seeing it run on your system?

Comment 37

[Todd]

Real Player is installed, but not showing in Firefox.  I was experiencing performance issues after update to Quantum and disabling/preventing accessibility services from accessing browser in Privacy & Security seems to have remedied that (symptom).  The only real software difference between this machine and other endpoints on the network is Real Player.  Still researching underlying cause of the performance issue.  See  https://bugzilla.mozilla.org/show_bug.cgi?id=1437354

Comment 38

[Liz Henry (:lizzard) (relman/hg->git project)]

Too late for a fix for 59 here. But it sounds like the issue may have improved since we blocked RealPlayer.

Comment 39

[Julien Cristau [:jcristau]]

This has turned into a meta bug, so I'll unset tracking flags for 60 as there doesn't seem to be anything directly actionable for the release here.

Comment 40

[BugBot [:suhaib / :marco/ :calixte]]

Moving to p3 because no activity for at least 24 weeks.
See https://github.com/mozilla/bug-handling/blob/master/policy/triage-bugzilla.md#how-do-you-triage for more information

Comment 41

[BugBot [:suhaib / :marco/ :calixte]]

In the process of migrating remaining bugs to the new severity system, the severity for this bug cannot be automatically determined. Please retriage this bug using the new severity system.

Comment 42

[James Teh [:Jamie]]

Severe performance problems caused specifically by e10s should have been addressed by the Cache the World project, which has now been shipping for some time.