2017264 - A55s in Lambda test environment are running with Accessibility mode enabled

Status: NEW

(Testing :: Performance, defect, P2)

Description

[Andrew Creskey [:acreskey]]

As far as I can tell the A55's in the lambda test environment have Accessibility mode enabled

See a11y calls and construction of the accessibility tree in this ci profile:
https://share.firefox.dev/4qHbp7d

This introduces additional performance overhead which I don't believe we intended for these tests.
(Noting that a11y performance relative to Chrome is still important, but should be captured in other tests, bugs).

Comment 1

[Andrew Creskey [:acreskey]]

Claude suggested disabling accessibility via these settings

https://hg-edge.mozilla.org/try/rev/e61891679f29ed60792d1ce38b66226cd1adf8d8

+        self.device.shell("settings put secure enabled_accessibility_services \"\"")
+        self.device.shell("settings put secure accessibility_enabled 0")

Which is yielding "significant" improvements in applink cpu timing, and possible 20-30ms on applink times, and possible 11-30ms improvements to newssite, (especially on pixel 6)
perf compare

Comment 2

[Kash Shampur [:kshampur] ⌚EST]

hi :aerickson, wanted to get your thoughts/ if you'd know if this is something that should be handled on the relops/lambda test side or our side?

Comment 3

[Andrew Erickson [:aerickson]]

We can do either/both (similar to device rotation... Bitbar had done it in the past, but (if possible) it's good to have it in our code to explicitly define the test requirements and it makes porting easier). I've added this to our 'Android HW Testing Requirements' doc (https://docs.google.com/document/d/1H0oQYkxWBrYQTWb5BFIrShrtcm0_VOB-cSInjLPx-tM/edit?tab=t.0).

It's strange that accessibility is on by default (unless we have test code that turns it on?).

I will bring this up with LT.

Comment 4

[Andrew Creskey [:acreskey]]

The results will shift as I'm adding more jobs, but I'm comparing the impact of our current CI setup (running A55s in Accessibility mode) to disabling accessibility.

Outside of applink, this looks to have a large impact on our browsertime pageload tests

https://perf.compare/compare-lando-results?baseLando=181700&newLando=181701&baseRepo=try&newRepo=try&framework=13&test_version=student-t&filter_confidence=medium%2Chigh

Numerous high confidence performance improvements of 8% to 15% on top sites.
(Still early data, will collect profiles to verify).

Comment 5

[Andrew Creskey [:acreskey]]

Just confirming via performance profile that Accessibility is running on the browsertime tests

job

Profile
https://share.firefox.dev/4qX4Ddu

perf compare

Impact from disabling Accessibility

wikipedia fcp optwarmwebrender, Android a55 
Improvement -21.39 %

Alexandru, just letting you know to expect a shift in the browsertime A55 pageload numbers once this is fixed at LambdaTest.

Comment 6

[Andrew Erickson [:aerickson]]

LT will be adding code to disable accessibility in their startup scripts. They said they don't normally turn it on.

Will post an update when I get confirmation that it's in place.

Comment 7

[Alex Finder]

(In reply to Andrew Creskey [:acreskey] from comment #5)

Just confirming via performance profile that Accessibility is running on the browsertime tests

job

Profile
https://share.firefox.dev/4qX4Ddu

perf compare

Impact from disabling Accessibility

wikipedia fcp optwarmwebrender, Android a55 
Improvement -21.39 %

Alexandru, just letting you know to expect a shift in the browsertime A55 pageload numbers once this is fixed at LambdaTest.

Thanks for the ping Andrew, I notified the sheriffs, we'll keep a look out for any alerts.

Comment 8

[Markus Stange [:mstange]]

Jamie, how can we find out what might cause Firefox to run with accessibility enabled in these performance tests?

Comment 9

[James Teh [:Jamie]]

There was some discussion about accessibility being enabled after some CI update in bug 1978133 comment 5. Beyond that, I'm not sure how to figure out what is causing it to be enabled.

Eitan, any thoughts here?

Comment 10

[Markus Stange [:mstange]]

(In reply to Andrew Erickson [:aerickson] from comment #6)

LT will be adding code to disable accessibility in their startup scripts. They said they don't normally turn it on.

Will post an update when I get confirmation that it's in place.

It might be worth leaving one or two devices around with the current configuration, so that we can determine a root cause for why Firefox was enabling accessibility in this configuration, in case there's a bug that affects users as well. Is that an option?

Comment 11

[Andrew Erickson [:aerickson]]

That's not super easy if LT adds this to their startup/init scripts (it would be an if block with specific device IDs, not the most durable).

They haven't implemented management of Accessibility yet. Do we just want LT to leave Accessibilty unmanaged (or pause rollout of management)?

I hadn't considered that If we have LT manage Accessibilty we lose the ability to detect bugs of this type.

:sparky, thoughts on this? Ideas on how we should decide issues like this (should we set, should we have vendor set, should we set in tree, set in our Relops setup code)? We started setting device rotation recently (decided to set in-tree (implemented) and in Relops setup code (in-progress).

Comment 12

[Andrew Creskey [:acreskey]]

This try push logs via adb what accessibility services are enabled via Android system command.

i.e.

testing/performance/mobile-startup/android_startup_videoapplink.py
+        a11y_enabled = self.device.shell_output("settings get secure accessibility_enabled")
+        a11y_services = self.device.shell_output("settings get secure enabled_accessibility_services")
+        print(f"A11Y STATE: accessibility_enabled={a11y_enabled}, services={a11y_services}")

Looking at logs from the A55 in Lambda Test we can see that it's LambdaTest's own service that's triggering a11y mode:

[task 2026-03-06T14:19:29.228+00:00] A11Y STATE: accessibility_enabled=1, services=com.lambdatest.lambdatestservice/com.lambdatest.lambdatestservice.watcher.services.Watcher

Claude's analysis:

That's the CI provider's own accessibility service. LambdaTest (the device farm) installs a service called Watcher that uses Android's accessibility APIs to monitor/control the device — likely for things like detecting UI state, capturing screenshots, handling      
  dialogs, or coordinating test execution.
                                                                                                                                                                                                                                                                            
  This is almost certainly what's causing your accessibility-related issues. It's not TalkBack or any user-facing a11y service — it's infrastructure tooling that happens to use the accessibility framework as a hook into the UI. 

Comment 13

[Markus Stange [:mstange]]

(In reply to Andrew Creskey [:acreskey] from comment #12)

services=com.lambdatest.lambdatestservice/com.lambdatest.lambdatestservice.watcher.services.Watcher

Great finding!

It's not good to have an accessibility service active on a device we use for performance testing. This will need to be addressed.

Comment 14

[Greg Mierzwinski [:sparky]]

:aerickson, if it's LT software that's enabling it (like :acreskey found), we should ask them to disable it since it's not a standard configuration for the device. We can also add something in-tree to try to disable it if need be (but maybe that'll break whatever tooling LT has?).

If there's a need to have some performance numbers with accessibility mode enabled, we should explore making some separate tests for it rather than splitting up our device pool.

Comment 15

[Andrew Erickson [:aerickson]]

I'm working with LT to get this disabled. Tracking in https://mozilla-hub.atlassian.net/browse/RELOPS-2219.

Comment 16

[Andrew Creskey [:acreskey]]

We will need additional patches to disabled LT's a11y service on other test frameworks (e.g. raptor-browsertime).

Sharing one of Kash's findings here:

The a11y service and a11y mode reduce both Chrome and Fenix speedometer scores by 18% to 28%, high confidence
Performance compare: Impact of Accessibility Mode on Speedometer 3, LambdaTest