741750 - Add support for non-gecko thread accessibility calls to the accessibility library

Status: RESOLVED WONTFIX

(Core :: Disability Access APIs, defect)

Description

[Jim Mathies [:jimm]]

I am just starting to dig into this. From what I understand the lack of an accessible connection to our UI in metro is the reason we can't get the soft keyboard to show up for XUL text input in Metro.

UI automation overview on MSDN - 

http://msdn.microsoft.com/en-us/library/windows/apps/xaml/ee684076.aspx

Overview of the win8 soft keyboard - 

http://msdn.microsoft.com/en-us/library/windows/apps/hh465404.aspx

Accessible COM interfaces supported -

http://msdn.microsoft.com/en-us/library/windows/apps/hh452772.aspx

In metro we do not receive WM_GETOBJECT messages since we don't actually deal with any standard windows. So our accessible code is currently not connected up. On the main ICoreWindow interface we do have, there is a 'request automation provider' event and an 'automation host' property - 

http://msdn.microsoft.com/en-us/library/windows/apps/windows.ui.core.icorewindow.aspx

This looks like what we need to hook up to although I haven't found any documentation yet on the interfaces the Object's these interfaces provide support. I might be able to query for this through the inspectable interface.

Comment 1

[Marco Zehe (:MarcoZ)]

CC'ing the NVDA developers, too, since this may most definitely be of interest to them.

Comment 2

[alexander :surkov (:asurkov)]

(In reply to Jim Mathies [:jimm] from comment #0)
> I am just starting to dig into this.

Cool. We were planned to add UIA implementation via IAccessibleEx in few/several months. But we don't have a real plans for full UIA implementation. If you're going to do the work on this way then I'm happy to help.

Comment 3

[Jim Mathies [:jimm]]

(In reply to alexander :surkov from comment #2)
> (In reply to Jim Mathies [:jimm] from comment #0)
> > I am just starting to dig into this.
> 
> Cool. We were planned to add UIA implementation via IAccessibleEx in
> few/several months. But we don't have a real plans for full UIA
> implementation. If you're going to do the work on this way then I'm happy to
> help.

http://msdn.microsoft.com/en-us/library/windows/apps/hh437315.aspx

There's an auto msaa wrapper, have we looked at that? AFAICT UIA may be what we need here but I can't confirm that at this point.

Comment 4

[Jim Mathies [:jimm]]

Ah - 

http://msdn.microsoft.com/en-us/library/windows/apps/hh437315.aspx

Minimum supported client
Windows 8 Consumer Preview

Comment 5

[alexander :surkov (:asurkov)]

(In reply to Jim Mathies [:jimm] from comment #3)

> http://msdn.microsoft.com/en-us/library/windows/apps/hh437315.aspx
> 
> There's an auto msaa wrapper, have we looked at that? AFAICT UIA may be what
> we need here but I can't confirm that at this point.

If I get right then Uiautomationcore wraps MSAA automatically and that's why we have basic accessibility (but since MSAA is restrictive then it's not amazing).

Comment 6

[David Bolter [:davidb] (NeedInfo me for attention)]

The elephant in the room: performance (see also bug 538530) -- we have to have a good plan for metro. One obvious one is improving our a11y engine performance, but there could be special casing etc.

Comment 7

[Jim Mathies [:jimm]]

(In reply to David Bolter [:davidb] from comment #6)
> The elephant in the room: performance (see also bug 538530) -- we have to
> have a good plan for metro. One obvious one is improving our a11y engine
> performance, but there could be special casing etc.

Reading through this reminded me of tsf which is off by default. Turning that on I found it was also broken for some reason.

Comment 8

[Jim Mathies [:jimm]]

Initial work on this is taking place in bug 762817.

Comment 9

[Jim Mathies [:jimm]]

Requesting triage on this, we need to figure out where this sits in our priorities list. Currently we have no accessibility support.

Comment 10

[Jim Mathies [:jimm]]

Morphing this bug into something more useful.

A couple notes - 

1) for accessibility to work in metrofx, we'll either need a full uia implementation or a completed partial implementation using IAccessbielEx. The former is preferred. (tracking bug uia)
2) We want to migrate metrofx over to a more multi-threaded architecture such that:
- Windows input and accessibility calls occur on a new and separate input thread.
- Input events are handed off async to the gecko thread after being processed by the apz on the input thread.

As such, accessibility will need to be able to handle calls from apps on the input thread. TBD whether accessibility will need to block on cross thread dom calls or maybe cache state that gets updated asynchronously  from dom. I think the latter would be better for performance but I'm not sure how feasible that it.

Currently we are not using accessibility for much in metrofx, although we do make calls to it for focus tracking to get the soft keyboard working. We could implement this threading work now using our light-weight UIA wrapper however this would lock out accessibility support until the library supports the thread separation we ultimately want. That might be a bad idea. If we don't jump on this, we can go ahead and try to get the IAccessbileEx impl. or full UIA impl. working on the gecko thread. The negative effects of this approach are that apz input processing will have to remain on the gecko thread for some time.

Note from my understanding desktop will also be migrating toward this model.

Comment 11

[James Teh [:Jamie]]

Just to clarify, I assume accessibility will use the input HWND (hwndFocus)? Accessibles must be created in the same thread as the HWND that answered WM_GETOBJECT. I suspect you're already very much aware of this, but just wanted to make certain.

Comment 12

[Jim Mathies [:jimm]]

(In reply to James Teh [:Jamie] from comment #11)
> Just to clarify, I assume accessibility will use the input HWND (hwndFocus)?
> Accessibles must be created in the same thread as the HWND that answered
> WM_GETOBJECT. I suspect you're already very much aware of this, but just
> wanted to make certain.

Yes. The separation will need to be between the input thread and the gecko thread that calls into the dom. Another possible fix might be to keep a cache of the dom state in the input thread that the dom thread updates.

Comment 13

[James Teh [:Jamie]]

Two other possible gotchas:
1. Having to call into (and wait on) another thread to answer a11y calls might cause performance issues. Windows ATs tend to inject in-process and slurp the entire a11y tree to render their own representation, so any performance degradation is exponential. Caching might fix this, but that introduces its own problems. I know Chrome used to cache the a11y tree in the UI process, but I'm not sure whether they still do this.
2. In-process win events are synchronous. This is particularly significant for IA2 textInserted events, as the text that was just inserted (and its offsets) is exposed via IAccessibleText::newText. Therefore, it's important that newText exposes the correct data at the time the event is fired. The same is true for oldText, though I don't know how well Gecko currently implements this and am not sure whether ATs use it (NVDA doesn't currently). ATs may rely on the synchronous nature of in-process win events for other reasons as well.

Comment 14

[Jim Mathies [:jimm]]

(In reply to James Teh [:Jamie] from comment #13)
> Two other possible gotchas:
> 1. Having to call into (and wait on) another thread to answer a11y calls
> might cause performance issues. Windows ATs tend to inject in-process and
> slurp the entire a11y tree to render their own representation, so any
> performance degradation is exponential. Caching might fix this, but that
> introduces its own problems. I know Chrome used to cache the a11y tree in
> the UI process, but I'm not sure whether they still do this.
> 2. In-process win events are synchronous. This is particularly significant
> for IA2 textInserted events, as the text that was just inserted (and its
> offsets) is exposed via IAccessibleText::newText. Therefore, it's important
> that newText exposes the correct data at the time the event is fired. The
> same is true for oldText, though I don't know how well Gecko currently
> implements this and am not sure whether ATs use it (NVDA doesn't currently).
> ATs may rely on the synchronous nature of in-process win events for other
> reasons as well.

Currently ATs call through on the gecko thread, so any overhead associated with the ATs is already a problem. I'm not convinced this is a major issue though. For example, if an AT needs to query the dom for state for a sight impaired user, jank in the browser isn't going to be a major issue for that user.

I like the idea of caching on the input thread but I can't really comment on how feasible this approach is with our existing accessibility library. dbolter?

For accessible events that need to be synchronous our ipc mechanism should be able to handle this via synchronous cross thread calls.

Comment 15

[James Teh [:Jamie]]

(In reply to Jim Mathies [:jimm] from comment #14)
> Currently ATs call through on the gecko thread, so any overhead associated
> with the ATs is already a problem. I'm not convinced this is a major issue
> though. For example, if an AT needs to query the dom for state for a sight
> impaired user, jank in the browser isn't going to be a major issue for that
> user.
I mean that unless I'm missing something, the time for any call from an AT will be slightly longer, since it is no longer in the same thread as the DOM. If an AT is slurping the a11y tree, this will be an exponential increase.

> For accessible events that need to be synchronous our ipc mechanism should
> be able to handle this via synchronous cross thread calls.
The problem is that the state in the Gecko thread might have already changed by the time the event is handled in the a11y thread, unless the Gecko thread blocks until the a11y thread has finished handling any event.

Comment 16

[Sylvestre Ledru [:Sylvestre]]

Closing as we never shipped Metro!

Comment 17

[Marco Zehe (:MarcoZ)]

We might still need this.

Comment 18

[James Teh [:Jamie]]

This is likely to require quite a bit of caching (since many of our queries must be serviced by DOM). Even with future Windows developments, I think we'll either do what we do now and handle incoming calls on the MTA and forward them to the main thread, or better still, we might be able to find a way to just handle incoming calls in the (content) main thread. I'm going to close this for now, since I really hope it won't be necessary. We can always reopen if worse comes to the worst.