1843832 - Firefox reports wrong element when Accesibility tools ask for what element is under the specific point and the element is an anchor

Status: RESOLVED FIXED

(Core :: Disability Access APIs, defect)

Description

[iopanag]

Attachment: UIAFirefox.cpp

Steps to reproduce:

This is an issue when using Firefox on windows and trying to get element details under the mouse cursor via Windows UI Automation which is used for accesibility. If the element is an html anchor and is underlined when hovered, firefox reports that under the cursor is the whole Firefox window instead. Here are some reproductions steps:

• Install https://accessibilityinsights.io/ and launch it.
• Launch Firefox on a page with underlined anchors eg: https://www.google.com/search?q=firefox
• Try to hover over a html element other than anchor. It should work
• Try to hover over a blue anchor. It shows that the whole Firefox window.

It is reprodicable on Firefox 115.0.2 on Windows 11 Preview and Windows 10 Pro 22H2. It is NOT reproducible on Firefox ESR 102.13.0.
I created a simple console program in C++ to make sure it is not an issue with Accessibility Insights. This program reports some element information under the mouse cursor via UIAutomation. I attached the source file. It should compile on VS with no special configuration.

Actual results:

When UIAutomation asks Firefox for the element under specific point and the element is an html anchor that the text underlined, Firefox reports the whole window.

Expected results:

When asked for the element under the specific point and the element is a html anchor, it should report that anchor.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Disability Access APIs' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[James Teh [:Jamie] (away returning 28 Jan)]

Firefox doesn't support UI Automation natively yet. Something weird is going on with the Windows UI Automation -> IAccessible2 proxy here. AccessibleObjectFromPoint returns the right thing, but whenever there is a link, UIA falls back to the window root. I don't yet know why.

Comment 3

[James Teh [:Jamie] (away returning 28 Jan)]

I get the same behaviour in Google Chrome, except it falls back to the document, not the window. Edge is different because it has a native UIA implementation.

Comment 4

[James Teh [:Jamie] (away returning 28 Jan)]

I did a bit more digging:

1. This seems to be specific to links.
2. Both Firefox and Chrome expose text leaf children in links and most other elements. AccessibleObjectAtPoint returns the text leaf, not the link, because the text leaf is the deepest child.
3. Prior to Firefox 113, IAccessible::accHitTest only ever returned a direct child of the element it was called on. After Firefox 113, it returns the deepest child. This change was made for performance and accuracy reasons. AccessibleObjectFromPoint ends up calling accHitTest all the way down the tree anyway, so we may as well save the caller the trouble and do it in Firefox. Chrome has done this for quite some time now (perhaps always).
4. For some reason, when we return the text leaf child of a link from IAccessible::accHitTest, UI Automation falls back to the window. I don't understand why. There is no good reason for this behaviour that I can think of. This impacts Chrome as well, except it returns the document instead of the window, probably because the document lives in its own HWND in Chrome.

I think we need to understand what the UIA -> IA2 proxy is trying to do here. It seems to me that this is a bug in that proxy, but even if we have to implement a workaround in Firefox, it would be helpful to know how broad that workaround needs to be.

Comment 5

[James Teh [:Jamie] (away returning 28 Jan)]

Curiously, returning ROLE_SYSTEM_GROUPING instead of ROLE_SYSTEM_LINK for links doesn't fix this, so it's not just the role that's the issue here. I can't figure out what else it could be, though.

Comment 6

[James Teh [:Jamie] (away returning 28 Jan)]

:iopanag, any thoughts on comment 4? There seems to be a bug in the UIA -> IA2 proxy here and it'd be good to understand what it's trying to do. I'm hoping you might be able to ask within Microsoft.

Comment 7

[iopanag]

Hello, I am sorry for the long time to reply. Indeed this seems to be the issue you describe since chromium based applications have the same issue. I've notified the related team, but it might not be investigated any time soon, because chromium is already developing to move to UIA implementation which will resolve this issue.

Comment 8

[James Teh [:Jamie] (away returning 28 Jan)]

Attachment: Bug 1843832: Never return the text leaf child of a link when calling IAccessible::accHitTest on an ancestor of the link.

This is necessary to work around a bug in the Windows UI Automation -> IAccessible2 proxy.
See the code comments for details.

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

There is an r+ patch which didn't land and no activity in this bug for 2 weeks.
:Jamie, could you have a look please?
If you still have some work to do, you can add an action "Plan Changes" in Phabricator.
For more information, please visit BugBot documentation.

Comment 10

[Nathan LaPré]

We're waiting for the resolution of Bug 1852190 to land this.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c406ccc1889d
Never return the text leaf child of a link when calling IAccessible::accHitTest on an ancestor of the link. r=nlapre

Comment 12

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/c406ccc1889d