2020386 - Fenix navigation performance regresses more than Chrome when accessibility is enabled in raptor tests

Status: NEW

(Core :: Disability Access APIs, defect)

Description

[Andrew Creskey [:acreskey]]

We discovered in bug 2017264 that Accessibility mode was unintentionally enabled on all Samsung A55s running in CI at Lambda Test.

As a result we can measure the performance impact of disabling Accessibility mode in various tests (by disabling via adb commands).

As far as I can tell, in our raptor tests, Fenix navigation performance regresses more than Chrome when accessibility is enabled.

Or, looking at it from the perspective of these tests, Fenix navigation performance improves more than Chrome when accessibility is disabled.

PerfCompare - base is a117 enabled, new is disabled

Looking at the improvements from disabling Accessibility, we see numerous 10%, 15%, 20% improvements in Fenix.
e.g.

wikipedia fcp optwarmwebrender
Android a55
143.58 ms > 112.86 ms
Improvement -21.39 %
Confidence: High
B:18N:21

wikipedia FirstVisualChange optwarmwebrender
Android a55
144.29 ms > 124.22 ms
Improvement -13.91 %
Confidence:  High
B:18N:21

amazon-search largestContentfulPaint optwarmwebrender
Android a55
197.34 ms > 173.01 ms
Improvement -12.33 %
Confidence: High

google-search-restaurants fcp optwarmwebrender
Android a55
122.38 ms > 109.29 ms
Improvement -10.69 %
Confidence: High

allrecipes FirstVisualChange optwarmwebrender
Android a55
165.18 ms > 137.62 ms
Improvement -16.69 %
Confidence: High

Apologies for the include of geckoview and ChromeAsRelease in this compare -- that was not intended.

However in Chrome the improvements are fewer and tend to be of lower magnitude (but are still clearly present)
e.g.

wikipedia loadtime optwarm
Android a55
220.86 ms (chrome-m) > 193.12 ms (chrome-m)
Improvement -12.56 %
Confidence: High
B:13N:13

allrecipes LastVisualChange optcold
Android a55
578.9 ms (chrome-m) > 521.21 ms (chrome-m)
Improvement -9.97 %
Confidence: Medium
B:14N:12

reddit loadtime optwarm
Android a55
781.95 ms (chrome-m) > 716.45 ms (chrome-m)
Improvement -8.38 %
Confidence: High
B:13N:14

PerfCompare Chrome - base is a117 enabled, new is disabled

Not sure the best way to handle this -- it might be to make this a meta and spin off specific bugs for sites where we see a larger Accessibility impact in Fenix than Chrome, e.g. wikipedia

Profile from Fenix on wikipedia
https://share.firefox.dev/4qX4Ddu

Comment 1

[James Teh [:Jamie]]

One big difference between Gecko and Chromium accessibility on Android is that Gecko runs on a different thread to the UI thread, whereas Chromium runs on the UI thread. Because all accessibility calls and events must run on the UI thread, we need a lock for thread safety. We must acquire this lock for every accessibility query on the UI thread, as well as any update to the accessibility tree in the Gecko thread, which incurs some performance penalty, though it's unclear how much. If there happens to be an accessibility query on the UI thread while we are also handling a tree update on the Gecko thread, there will be lock contention, which would incur a larger performance penalty. There isn't really any way to fix this without re-architecting GeckoView to run Gecko on the UI thread like Chromium does.

Note that we don't have this particular problem on other platforms because Gecko runs on the UI thread on other platforms.

Comment 2

[Eitan Isaacson [:eeejay]]

I can't be 100% certain this is related to our thread topography. I am curious what the delta is in desktop raptor tests with and without accessibility.

Comment 3

[Andrew Creskey [:acreskey]]

(In reply to James Teh [:Jamie] from comment #1)

One big difference between Gecko and Chromium accessibility on Android is that Gecko runs on a different thread to the UI thread, whereas Chromium runs on the UI thread. Because all accessibility calls and events must run on the UI thread, we need a lock for thread safety. We must acquire this lock for every accessibility query on the UI thread, as well as any update to the accessibility tree in the Gecko thread, which incurs some performance penalty, though it's unclear how much. If there happens to be an accessibility query on the UI thread while we are also handling a tree update on the Gecko thread, there will be lock contention, which would incur a larger performance penalty. There isn't really any way to fix this without re-architecting GeckoView to run Gecko on the UI thread like Chromium does.

Note that we don't have this particular problem on other platforms because Gecko runs on the UI thread on other platforms.

Thank you for the context.

Comment 4

[Andrew Creskey [:acreskey]]

(In reply to Eitan Isaacson [:eeejay] from comment #2)

I can't be 100% certain this is related to our thread topography. I am curious what the delta is in desktop raptor tests with and without accessibility.

Yes, now that I have an understanding of the threading model, I think that we might still be able to find improvements because we have this rare glimpse of a11y running in the raptor tests in CI.
(Of course we could force enable this at any time).

Maybe comparing against Chrome is less useful right now, but comparing Fenix with and without a11y could indeed be useful.

We can generate profiles from any of the raptor test jobs, e.g. wikipedia by pressing Generate performance profile.

The jobs with that end with -p , like wikipedia-p are the result and they include an Open in Firefox Profiler button

Wikipedia, a11y

Wikipedia, a11y disabled

The warm pageload tests look to be the most affected.

Comment 5

[Andrew Creskey [:acreskey]]

This is a quick flame graph view of Wikipedia warm load with a11y enabled. None of the filtered code appears without a11y.
https://share.firefox.dev/4u7HXu7

As an aside, the profiles can always be brought to https://matrix.to/#/#joy-of-profiling:mozilla.org for "free analysis" :)