1852190 - Windows accessibility API automated testing

Status: RESOLVED FIXED

(Core :: Disability Access APIs, task)

Description

[James Teh [:Jamie]]

Currently, we have pretty solid test coverage for our cross-platform accessibility code. We also have reasonable coverage for Mac and Android specific accessibility code. However, we have no coverage whatsoever for accessibility code specific to Windows and Linux, and this has definitely meant that some fairly obvious regressions have slipped through the cracks for a while.

Our Android accessibility tests are written as Kotlin JUnit tests, as with other GeckoView stuff. For Mac, because the core API is pretty basic (essentially attribute retrieval and notification firing), we're able to do this with some Mac specific XPCOM glue, so we can use browser tests. However, Windows and Linux are more complicated. Both APIs involve many method calls. Windows uses COM, so it's even more complicated in that it involves querying for interfaces and calling methods on specific objects. If we were to instrument these using XPCOM, we'd basically be wrapping every function in the entire API.

I don't think we have a way of accessing COM from JS, nor can we access what we need for ATK. Eitan noted that there is gjs, but I think it'd be a lot of work to get that working inside Firefox, and that wouldn't solve the problem for Windows.

Fortunately, there are solid Python packages which allow for access to everything we need: comtypes on Windows and pyatspi for ATK. We want to use browser tests as a base, which maximises consistency and leverages the solid framework we already have there. Thus, the aim is to implement a bridge which allows browser tests to run snippets of out-of-process Python code.

Comment 1

[James Teh [:Jamie]]

For the record, I considered two other options:

1. Write all the tests in C++. They'd need to be able to load an HTML snippet, run JS, asynchronously wait for events, etc. Ideally, we'd be able to support both in-process content, remote tabs and OOP iframes. I have no idea whether we have the foundations to do such things in C++. Doing this in C++ would also make the tests more complicated to write and maintain.
   • From Joel Maher: "Are there other ways to call WINAPI from the browser?  there probably are a few, but I would rather do it via python than node.js or xpcshell+JS.  C++ would work, but it seems overkill; I do wonder if some calls will be more straightforward to implement in C++, maybe that could be an option for a subset of calls."
   • But can we run C++ code against a full browser with remote documents and all?
2. Write the tests entirely in Python as Marionette tests.
   • I wonder whether this allows us to automate the browser UI (not just content)?
   • We'd need to be able to put markup snippets in the Python tests to avoid lots of clutter in testing/marionette/harness/marionette_harness/www. I guess we can do this by navigating to data: URIs.

Comment 2

[James Teh [:Jamie]]

I'm going to start with Windows, but I'm keeping ATK in mind as I work on this.

One challenge I'm not sure how to address is the IAccessible2 proxy. Thankfully, Windows 10 and later register this system wide, so we can probably deal with this problem later. However, this doesn't include interfaces added to IAccessible2 more recently; e.g. IAccessibleTextSelectionContainer. We won't be able to test those unless we register our own updated proxy. There are a few solutions here:

1. Build this and have it registered in the system registry (or the user registry) on CI machines. Keeping that up to date seems like it might be messy.
2. Build this in Mozilla central and somehow have CI add it to the user registry before running the tests.
3. Build this in Mozilla central and have the Python harness add it to the user registry when it starts and remove it when it stops. I don't love this because it could leave registry keys around if the test exits abnormally, which might be bad on local machines.
4. Build this in Mozilla central. Do what NVDA does and register it in the Python process, as well as injecting an in-process dll into Firefox which also registers it there. This is going to be quite a bit of work, but it does avoid messing with the registry. I'd really prefer to avoid this just because of the complexity.

For this bug, I'm going to rely on the proxy bundled with Windows, knowing that we can't support more recent interfaces for now.

Comment 3

[James Teh [:Jamie]]

Attachment: Bug 1852190 part 1: Provide the ability for a11y browser tests to run Python code.

Our mochitest harness always runs a pywebsocket3 server which supports handlers as files in any test directory.
We use this to implement a WebSocket which takes Python code as input and returns a JSON serialised result or exception.
The Python code is wrapped in a function so that it can return a result by simply using the return statement.
Python code run via the WebSocket has its own globals namespace.
OS specific things will be placed in this namespace for convenience in future patches.
Tests might also use Python globals to persist things they don't want to keep retrieving every call.
A runPython function has been added to shared-head.js to be called by browser tests.
This manages connection to the WebSocket, sending the code, asynchronously handling the response, etc.
It simply takes Python code as a string and returns the deserialised result.

Comment 4

[James Teh [:Jamie]]

Attachment: Bug 1852190 part 2: Reintroduce the IAccessible2 typelib.

We need this so that Python comtypes can generate bindings for IAccessible2.
This is almost identical to what was here before, but it excludes some stuff that was only needed for the COM interceptor.

Comment 5

[James Teh [:Jamie]]

Attachment: Bug 1852190 part 3: Add Windows MSAA/IA2 interfaces, constants and utility functions to the Python environment.

This also adds a simple role test which shows all of this working.
We can support UI Automation in future too.

Comment 6

[James Teh [:Jamie]]

Joel, I'd love your thoughts on comment 2 with regard to the options to register a newer IAccessible2 proxy. There's a lot we can do without this, so there's no rush, but we'll want this at some point.

Comment 7

[James Teh [:Jamie]]

Attachment: Bug 1852190 part 4: Add Windows UIA interfaces, constants and utility functions to the Python environment.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

interesting problem here- overall I think editing the registry should be done at OS configuration time, not test runtime.

How often will there need to be updates? possibly weekly, or most likely 1-2x/year? It is relatively easy to add software/changes to the OS configuration, but it takes up to 2 weeks to get done and rolled out.

For each task we prep the machine (here), so most likely it would be best to have a "cleanup" script that runs and ensures things are reset for every task, and for the a11y tasks that need to adjust settings we can have a different script or built into the task.

There is the ability for windows tasks to run-as-administrator (here), that will give the ability to edit the registry.

If we go that route, then I would prefer if we had a separate pool of machines, not necessarily a shared pool. This would prevent potential issues of tests going bad and interfering with future tasks on the same worker.

Comment 9

[James Teh [:Jamie]]

(In reply to Joel Maher ( :jmaher ) (UTC -8) from comment #8)

How often will there need to be updates? possibly weekly, or most likely 1-2x/year?

More likely once every few years, though that might rarely increase to 1-2x a year if there's some very new accessibility paradigm being developed. The last update gap was about 7 years.

There is the ability for windows tasks to run-as-administrator (here), that will give the ability to edit the registry.

If we go that route, then I would prefer if we had a separate pool of machines, not necessarily a shared pool. This would prevent potential issues of tests going bad and interfering with future tasks on the same worker.

Even though the dll needs to be registered in the registry (either in system/HKLM or user/HKCU), it won't interfere with any other tests. It only impacts marshaling of accessibility interfaces between processes, and since it's only accessibility interfaces, it only has an impact when OS accessibility calls are being made. So, this can be registered permanently; it doesn't need to be registered before tests and unregistered afterwards. FWIW, Windows 10 and later already registers this dll itself on every system. It's just that the version bundled with Windows is slightly outdated and doesn't include a newer interface we implement.

The easiest way to register this dll is to use regsvr32.exe (included with Windows), rather than manually editing the registry. The only catch is that this will register in the system registry (HKLM), not the user registry (HKCU), so this requires administrator privileges.

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

It sounds like we should bake this into the image. I see this a lot like what we did for xperf and a .mof file in bug 1748149

:jamie, can you give a rough idea of what it would take to make this tool, and how we can verify it?

:jmoss, could you chime in with how long it would take to bake into the windows image a specific tool (.dll that needs registering)

Comment 11

[Jonathan Moss]

(In reply to Joel Maher ( :jmaher ) (UTC -8) from comment #10)

It sounds like we should bake this into the image. I see this a lot like what we did for xperf and a .mof file in bug 1748149

:jamie, can you give a rough idea of what it would take to make this tool, and how we can verify it?

:jmoss, could you chime in with how long it would take to bake into the windows image a specific tool (.dll that needs registering)

What timeline are we looking to implement this? We could test by baking in the change on win11-64-2009-alpha.

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

timeline would not be ASAP, but probably given the rate of other changes on this project maybe later this month. Depends on when :jamie and the a11y team can build this. It might be a quick thing or take a week or two to schedule time to do it.

Comment 13

[James Teh [:Jamie]]

(In reply to Joel Maher ( :jmaher ) (UTC -8) from comment #10)

:jamie, can you give a rough idea of what it would take to make this tool

That depends whether you want to automate creation or for me to just do it manually. If I were doing it manually, it'd probably take me an hour or so, maybe less. If we want to automate it so it's repeatable by anyone, that would take longer.

and how we can verify it?

The easiest way to do that would probably be for me to write a browser test using this new framework which uses a new interface. That would fail if the dll isn't registered correctly.

(In reply to Jonathan Moss from comment #11)

What timeline are we looking to implement this?

Honestly, there's no rush on it at all. I'm just getting in with the questions early. We've gone without automated testing for any of the Windows accessibility APIs for a couple of decades, so waiting a few months longer to test a single interface with two methods is hardly worth worrying about in relative terms. I just wanted to know for sure that we had a clear path forward when the time comes for more new interfaces to be added.

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

so I would say best case scenario is to automate it; if you want to hack it together and then validate deployment, etc. that could be done in parallel if you have time to hack on this project.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/867aa56c97d8
part 1: Provide the ability for a11y browser tests to run Python code. r=eeejay,jmaher
https://hg.mozilla.org/integration/autoland/rev/833bd5f06b7b
part 2: Reintroduce the IAccessible2 typelib. r=nlapre
https://hg.mozilla.org/integration/autoland/rev/8c65477b67d8
part 3: Add Windows MSAA/IA2 interfaces, constants and utility functions to the Python environment. r=eeejay,jmaher
https://hg.mozilla.org/integration/autoland/rev/59f4df718928
part 4: Add Windows UIA interfaces, constants and utility functions to the Python environment. r=eeejay

Comment 16

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/867aa56c97d8
https://hg.mozilla.org/mozilla-central/rev/833bd5f06b7b
https://hg.mozilla.org/mozilla-central/rev/8c65477b67d8
https://hg.mozilla.org/mozilla-central/rev/59f4df718928