1957255 - Microsoft UI automation (screen reading) fails to read header HTML elements

Status: UNCONFIRMED

(Core :: Disability Access APIs, defect)

Description

[yo]

Steps to reproduce:

Using windows 11 23h2, trying to read html pages with screen reading software, like microsoft's accessibility insights for windows, fails to read any headers, like h1 or h6, but can read the rest of the page properly. (test w3.org "make the world work" headers).
Used another screen reader that uses the same microsoft UI automation api and still does not work. other headers on other websites don't work.
Tried both screen readers and the same websites on chrome and they work as expected.
Contacted the developer of one of the screen readers and they mention that mozilla's wrapper of the api might be at fault, and I should file a bug report

Actual results:

Most non headers work, headers don't work

Expected results:

I should be able to read any pages' header elements using any screen reading software

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::DOM: Core & HTML' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[James Teh [:Jamie]]

Hi. Thanks for taking the time to report this.

Most screen readers on Windows - e.g. NVDA, JAWS, SuperNova - use MSAA and IAccessible2, not UI Automation, even in Chrome and Edge. Headings should work as expected in those screen readers. (I'm using NVDA as I write this.) Windows Narrator is a notable exception that does use UI Automation. There are other accessibility clients that do use UI Automation, including Microsoft Accessibility Insights, but they aren't screen readers. Can you provide details as to which screen readers you tested and what steps you took?

That said, Firefox doesn't yet support UI Automation natively, so Windows falls back to a proxy which is somewhat buggy at best. We are working on native support for UI Automation, though, which is nearing completion but not quite ready for general usage yet. If you'd like to test it, I can provide instructions.

Comment 3

[yo]

(In reply to James Teh [:Jamie] from comment #2)

Hi. Thanks for taking the time to report this.

Most screen readers on Windows - e.g. NVDA, JAWS, SuperNova - use MSAA and IAccessible2, not UI Automation, even in Chrome and Edge. Headings should work as expected in those screen readers. (I'm using NVDA as I write this.) Windows Narrator is a notable exception that does use UI Automation. There are other accessibility clients that do use UI Automation, including Microsoft Accessibility Insights, but they aren't screen readers. Can you provide details as to which screen readers you tested and what steps you took?

That said, Firefox doesn't yet support UI Automation natively, so Windows falls back to a proxy which is somewhat buggy at best. We are working on native support for UI Automation, though, which is nearing completion but not quite ready for general usage yet. If you'd like to test it, I can provide instructions.

My bad. This is the developer's response verbatim.
Firefox natively implements IAccessible2 which is a step up from Acc / MSAA/ IAccessible. Then there's a wrapper which turns UIA calls to IAccessible2 calls and converts between them. That specific header is not inspectable with any UIAutomation tool like Accessibility Insights or UIAViewer, but it is inspectable with an Acc tool like inspect.exe, so I'm guessing it's a bug in the MSAA-UIAutomation wrapper. This is not a bug in UIA.ahk but instead Firefox, so if you wish to get it fixed I'd recommend filing a bug report with them (or use the slower Acc.ahk instead).
My use case is not strictly related to accessibility access, more towards UI automation and macros using Autohotkey. I am using a library that allows interaction with UIAutomation framework, to read and translate romaji titles (japanese in english literals, in the form of headers).

This is the library's github route (Descolada/UIA-v2), which will need Autohotkey to be installed. I mentioned Microsoft Accessibility Insights as to check if it's the library's bug, and since it is the more available tool.

For the time being, i am using another library that wraps Acc (from the same developer)

Comment 4

[James Teh [:Jamie]]

(In reply to yo from comment #3)

That specific header is not inspectable with any UIAutomation tool like Accessibility Insights or UIAViewer, but it is inspectable with an Acc tool like inspect.exe, so I'm guessing it's a bug in the MSAA-UIAutomation wrapper.

I took a look at the raw UIA tree exposed from the Windows proxy. The heading is definitely present and narrator can see it. I did notice that IsControlElement returns false, but IsContentElement returns true. That is a violation of the UIA documentation which states:

The content view of the UI Automation tree is a subset of the control view.

So Microsoft's proxy violates their own spec. Terrific. For what it's worth, this kind of bugginess is one of the reasons we're currently working on a native UIA implementation.

I'm not sure how this UIA.ahk library works, but you could work around this by having it walk the raw view instead of the control view, filtering out any elements which specify false for both IsControlElement and IsContentElement. But it may just be better to wait until Firefox's native UIA implementation ships, which should hopefully be later this year. Again, I'm happy to provide instructions if you'd like to test with it.

Comment 5

[yo]

(In reply to James Teh [:Jamie] from comment #4)

(In reply to yo from comment #3)

That specific header is not inspectable with any UIAutomation tool like Accessibility Insights or UIAViewer, but it is inspectable with an Acc tool like inspect.exe, so I'm guessing it's a bug in the MSAA-UIAutomation wrapper.

I took a look at the raw UIA tree exposed from the Windows proxy. The heading is definitely present and narrator can see it. I did notice that IsControlElement returns false, but IsContentElement returns true. That is a violation of the UIA documentation which states:

The content view of the UI Automation tree is a subset of the control view.

So Microsoft's proxy violates their own spec. Terrific. For what it's worth, this kind of bugginess is one of the reasons we're currently working on a native UIA implementation.

I'm not sure how this UIA.ahk library works, but you could work around this by having it walk the raw view instead of the control view, filtering out any elements which specify false for both IsControlElement and IsContentElement. But it may just be better to wait until Firefox's native UIA implementation ships, which should hopefully be later this year. Again, I'm happy to provide instructions if you'd like to test with it.

The api i use from this library gets the element directly under the mouse, I am not sure how I would replicate this by crawling a raw tree.
Since I have a fallback the covers my needs, and since you guys are working on a fix, I will wait for it.
Thanks for your response!