1926214 - Accessibility focus should be set to text leaf when scrolling to text fragment on Android

Status: RESOLVED FIXED

(Core :: Disability Access APIs, defect, P3)

Description

[James Teh [:Jamie]]

STR (with Talkback):

1. Open this test case:
   data:text/html,<h1>Begin</h1><hr style="height: 100vh;"><div id="content"><h1>Content</h1><hr style="height: 200vh;">foo bar#:~:text=bar
   • Expected: The Talkback cursor should move to "foo bar"
   • Actual: The Talkback cursor moves to "content"

The target text is a child of a much larger container. The top of the container is a long way away from the target text, but currently, accessibility fires scrolling start events on the container to which we have scrolled (#content), so we lose where we have scrolled inside the container. Our Android code will Pivot to the first node inside #content, but that's far from ideal.

We could do better if we instead fire the scrolling start on the text leaf. On Windows, scrolling start needs to fire on a hypertext, not a text leaf. So, I guess we're going to need to modify the core event to fire on a text leaf and have the Windows code tweak it.

Comment 1

[James Teh [:Jamie]]

Attachment: Bug 1926214 part 2: Fire a11y scrolling start events on a text leaf node when navigating to text fragments.

Previously, scrolling start events always fired on an Accessible for an element, never a text leaf.
However, a text fragment starts in a text leaf node and there may be many other children of the parent element prior to the start of the text fragment.
We want to get a11y clients as close as possible to the fragment, so fire the event on the text leaf node instead.

Unfortunately, we can't fire the Windows event on text leaf nodes, so we have to override this to use the parent in Windows specific code.
Fortunately, we can expose the highlight on Windows via text attributes, so clients can find out exactly what is highlighted, albeit with a bit more work.
This text leaf change is of most benefit on Android, where we can't expose the highlight, but we can move accessibility focus to a text leaf.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ada40589ff0c
Fire a11y scrolling start events on a text leaf node when navigating to text fragments. r=jjaschke,morgan

Comment 3

[agoloman]

Backed out for causing ba failures.

• Backout link
• Push with failures
• Failure Log
• Failure line:

[task 2024-10-28T07:26:51.861Z] 07:26:51     INFO - TEST-START | accessible/tests/browser/windows/ia2/browser_scrolling.js
[task 2024-10-28T07:26:52.179Z] 07:26:52     INFO - GECKO(4256) | must wait for focus in content
[task 2024-10-28T07:27:02.267Z] 07:27:02     INFO - TEST-INFO | started process screenshot
[task 2024-10-28T07:27:02.873Z] 07:27:02     INFO - TEST-INFO | screenshot: exit 0
[task 2024-10-28T07:27:02.873Z] 07:27:02     INFO - Buffered messages logged at 07:26:51
[task 2024-10-28T07:27:02.874Z] 07:27:02     INFO - Entering test bound 
[task 2024-10-28T07:27:02.875Z] 07:27:02     INFO - TEST-PASS | accessible/tests/browser/windows/ia2/browser_scrolling.js | Received document load complete event - 
[task 2024-10-28T07:27:02.876Z] 07:27:02     INFO - Buffered messages logged at 07:26:52
[task 2024-10-28T07:27:02.877Z] 07:27:02     INFO - TEST-PASS | accessible/tests/browser/windows/ia2/browser_scrolling.js | Actually remote browser - 
[task 2024-10-28T07:27:02.877Z] 07:27:02     INFO - Navigating to text fragment: second phrase
[task 2024-10-28T07:27:02.878Z] 07:27:02     INFO - Buffered messages finished
[task 2024-10-28T07:27:02.882Z] 07:27:02     INFO - TEST-UNEXPECTED-FAIL | accessible/tests/browser/windows/ia2/browser_scrolling.js | Uncaught exception in test bound  - at chrome://mochitests/content/browser/accessible/tests/browser/shared-head.js:1024 - Error: Traceback (most recent call last):
[task 2024-10-28T07:27:02.882Z] 07:27:02     INFO -   File "D:\task_173009891857043\build\tests\mochitest\browser\accessible\tests\browser\windows\a11y_setup.py", line 227, in wait
[task 2024-10-28T07:27:02.882Z] 07:27:02     INFO -     ctypes.oledll.ole32.CoWaitForMultipleHandles(
[task 2024-10-28T07:27:02.882Z] 07:27:02     INFO -   File "_ctypes/callproc.c", line 1000, in GetResult
[task 2024-10-28T07:27:02.882Z] 07:27:02     INFO - OSError: [WinError -2147417835] OLE has sent a request and is waiting for a reply
[task 2024-10-28T07:27:02.882Z] 07:27:02     INFO - 
[task 2024-10-28T07:27:02.882Z] 07:27:02     INFO - During handling of the above exception, another exception occurred:
[task 2024-10-28T07:27:02.882Z] 07:27:02     INFO - 
[task 2024-10-28T07:27:02.882Z] 07:27:02     INFO - Traceback (most recent call last):
[task 2024-10-28T07:27:02.882Z] 07:27:02     INFO -   File "<string>", line 87, in web_socket_transfer_data
[task 2024-10-28T07:27:02.882Z] 07:27:02     INFO -   File "<string>", line 3, in run
[task 2024-10-28T07:27:02.883Z] 07:27:02     INFO -   File "D:\task_173009891857043\build\tests\mochitest\browser\accessible\tests\browser\windows\a11y_setup.py", line 232, in wait
[task 2024-10-28T07:27:02.883Z] 07:27:02     INFO -     raise TimeoutError("Timeout before desired event received")
[task 2024-10-28T07:27:02.883Z] 07:27:02     INFO - TimeoutError: Timeout before desired event received
[task 2024-10-28T07:27:02.883Z] 07:27:02     INFO - 
[task 2024-10-28T07:27:02.883Z] 07:27:02     INFO - Stack trace:
[task 2024-10-28T07:27:02.883Z] 07:27:02     INFO - runPython/</gPythonSocket.onmessage@chrome://mochitests/content/browser/accessible/tests/browser/shared-head.js:1024:16
[task 2024-10-28T07:27:02.883Z] 07:27:02     INFO - EventHandlerNonNull*runPython/<@chrome://mochitests/content/browser/accessible/tests/browser/shared-head.js:1017:5
[task 2024-10-28T07:27:02.883Z] 07:27:02     INFO - runPython@chrome://mochitests/content/browser/accessible/tests/browser/shared-head.js:1016:10
[task 2024-10-28T07:27:02.883Z] 07:27:02     INFO - testTextFragment@chrome://mochitests/content/browser/accessible/tests/browser/windows/ia2/browser_scrolling.js:25:11
[task 2024-10-28T07:27:02.883Z] 07:27:02     INFO - async*accessibleTask/</<@chrome://mochitests/content/browser/accessible/tests/browser/shared-head.js:602:15
[task 2024-10-28T07:27:02.883Z] 07:27:02     INFO - async*withNewTab@resource://testing-common/BrowserTestUtils.sys.mjs:121:22
[task 2024-10-28T07:27:02.883Z] 07:27:02     INFO - async*accessibleTask/<@chrome://mochitests/content/browser/accessible/tests/browser/shared-head.js:521:28
[task 2024-10-28T07:27:02.883Z] 07:27:02     INFO - handleTask@chrome://mochikit/content/browser-test.js:1145:26
[task 2024-10-28T07:27:02.883Z] 07:27:02     INFO - _runTaskBasedTest@chrome://mochikit/content/browser-test.js:1217:18
[task 2024-10-28T07:27:02.883Z] 07:27:02     INFO - Tester_execTest@chrome://mochikit/content/browser-test.js:1358:14
[task 2024-10-28T07:27:02.884Z] 07:27:02     INFO - nextTest/<@chrome://mochikit/content/browser-test.js:1134:14
[task 2024-10-28T07:27:02.884Z] 07:27:02     INFO - SimpleTest.waitForFocus/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:1058:13
[task 2024-10-28T07:27:02.884Z] 07:27:02     INFO - Leaving test bound 
[task 2024-10-28T07:27:02.885Z] 07:27:02     INFO - GECKO(4256) | MEMORY STAT | vsize 956MB | vsizeMaxContiguous 1293MB | residentFast 236MB | heapAllocated 88MB
[task 2024-10-28T07:27:02.886Z] 07:27:02     INFO - TEST-OK | accessible/tests/browser/windows/ia2/browser_scrolling.js | took 10758ms

Comment 4

[James Teh [:Jamie]]

Attachment: Bug 1926214 part 1: Call CoInitialize for new threads in a11y Python test runner.

Each browser test file is run with a separate instance of shared-head.js and thus a distinct WebSocket connection to the a11y Python runner.
pywebsocket3 is multi-threaded and may thus choose different threads to handle different WebSocket requests.
Python comtypes implicitly initialises COM when it is imported, but COM initialisation is thread specific and Python modules are cached, so this only applies to the thread on which comtypes was imported.
These factors were causing problems such as event timeouts when running multiple IA2 test files because IA2 relies on COM being initialised.
To fix this, add a setup() function to a11y_setup.py which is called when a request is handled.
For Windows, this calls CoInitialize once in each thread in which it is run.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a0e11f18e325
part 1: Call CoInitialize for new threads in a11y Python test runner. r=nlapre
https://hg.mozilla.org/integration/autoland/rev/3613d93e8727
part 2: Fire a11y scrolling start events on a text leaf node when navigating to text fragments. r=jjaschke,morgan

Comment 6

[amarc]

https://hg.mozilla.org/mozilla-central/rev/a0e11f18e325
https://hg.mozilla.org/mozilla-central/rev/3613d93e8727