762817 - (uiatext) implement UIA Text pattern

Status: NEW

(Core :: Disability Access APIs, task)

Comment 2

[Jim Mathies [:jimm]]

Based on the progress being made I think we'll just try to get the new uia accessibility interfaces hooked up directly, which was the long term goal all along. Currently we have a hackish impl. of basic uia for text inputs down in widget that's metro specific, but we don't want to rely on that.

Comment 3

[Trevor Saunders (:tbsaunde)]

I probably won't work on this itself soon, but I may be working on the accessibility stuff this will call into a bit in the near future, and can otherwise help some one  working on this with what to do and how to design it.  It should be reasonably possible to break implementing this up into a set of chunks than can be done one at a time.

Comment 4

[Jim Mathies [:jimm]]

How would the text pattern interface with our existing access code? Would it be built on top of the existing code library and do the two match up fairly well?

I'm curious because I'd like to try and get a basic version of this hooked up to text edits for testing purposes. I still do not know if our accessibility lib and metro will play well together. Initial attempts didn't work out. But now that we have IAccessibleEx implemented, we might have better luck.

For reference this is the very very basic UIA impl we did to get the soft keyboard working - 

http://mxr.mozilla.org/projects-central/source/elm/widget/windows/winrt/UIABridge.cpp

Comment 5

[Trevor Saunders (:tbsaunde)]

(In reply to Jim Mathies [:jimm] from comment #4)
> How would the text pattern interface with our existing access code? Would it
> be built on top of the existing code library and do the two match up fairly
> well?

yeah, it looks like a fair bit of it can be implemented by just calling to the core accessibility code similar to what atk does in accessible/src/atk/nsMaiInterface{,Editable}Text.cpp or ia2 in msaa/ia2Acessible{Editable,}Text.cpp There are some bits that the text provider has that we don't have yet, mostly the visibility stuff, but I thinkmac may want some visibility stuff to so implementing that may need to happen at some point anyway.

However in general it seems like atk / ia2 have a fairly similar model for how text is retrieved to uia so it shouldn't be too bad.

> I'm curious because I'd like to try and get a basic version of this hooked
> up to text edits for testing purposes. I still do not know if our
> accessibility lib and metro will play well together. Initial attempts didn't
> work out. But now that we have IAccessibleEx implemented, we might have
> better luck.
> 
> For reference this is the very very basic UIA impl we did to get the soft
> keyboard working - 
> 
> http://mxr.mozilla.org/projects-central/source/elm/widget/windows/winrt/
> UIABridge.cpp

It seems like it should be prety easy given some time to move that much functionality onto what we have already for uia.  Of course that isn't much, it seems all you really implment is focus tracking, and bounds.

Comment 6

[James Teh [:Jamie]]

It's worth noting that for compound/rich text access (i.e. anything beyond simple text inputs, textareas and similar), UIA generally requires a very different approach to what Gecko currently exposes for IA2 and ATK. Gecko currently exposes embedded object characters for any object and the client then has to walk into the object itself to get the text. This has always been somewhat of a controversial approach and nothing else does this to such a large extent. In contrast, based on how IE exposes the UIA text pattern (which is probably what Narrator requires), all text is exposed on the document object with the ability to retrieve objects within a given range and a range for a given object. Interestingly, this is the same idea i had for an alternative IA2 text implementation.

Comment 7

[Trevor Saunders (:tbsaunde)]

(In reply to James Teh [:Jamie] from comment #6)
> It's worth noting that for compound/rich text access (i.e. anything beyond
> simple text inputs, textareas and similar), UIA generally requires a very
> different approach to what Gecko currently exposes for IA2 and ATK. Gecko
> currently exposes embedded object characters for any object and the client
> then has to walk into the object itself to get the text. This has always
> been somewhat of a controversial approach and nothing else does this to such
> a large extent. In contrast, based on how IE exposes the UIA text pattern
> (which is probably what Narrator requires), all text is exposed on the
> document object with the ability to retrieve objects within a given range
> and a range for a given object. Interestingly, this is the same idea i had
> for an alternative IA2 text implementation.

interesting!  that seems contrary to my read of http://msdn.microsoft.com/en-us/library/windows/desktop/ee671665%28v=vs.85%29.aspx
which I read as being basically what gecko does if you strip out the embed object characters, and just have nothing where the embed text is.

That said what you say IE does seems like a reasonable way to do things, so while it'll take work I'm not really opposed to implementing that.

Comment 8

[James Teh [:Jamie]]

(In reply to Trevor Saunders (:tbsaunde) from comment #7)
> interesting!  that seems contrary to my read of
> http://msdn.microsoft.com/en-us/library/windows/desktop/ee671665%28v=vs.
> 85%29.aspx
> which I read as being basically what gecko does if you strip out the embed
> object characters, and just have nothing where the embed text is.
I don't come to that conclusion reading that article:
> The textual content (or inner text) of a text container and an embedded object, such as a hyperlink or table cell, is exposed as a single, continuous text stream in both the control view and the content view of the UI Automation tree; object boundaries are ignored.
The examples also suggest this; for instance:
> {The URL http://www.microsoft.com is embedded in text}.
> IUIAutomationTextPattern::RangeFromChild, where the UI Automation element was returned by the previous IUIAutomationTextRange::GetChildren method. Returns the range that represents "http://www.microsoft.com".

Comment 9

[Trevor Saunders (:tbsaunde)]

(In reply to James Teh [:Jamie] from comment #8)
> (In reply to Trevor Saunders (:tbsaunde) from comment #7)
> > interesting!  that seems contrary to my read of
> > http://msdn.microsoft.com/en-us/library/windows/desktop/ee671665%28v=vs.
> > 85%29.aspx
> > which I read as being basically what gecko does if you strip out the embed
> > object characters, and just have nothing where the embed text is.
> I don't come to that conclusion reading that article:
> > The textual content (or inner text) of a text container and an embedded object, such as a hyperlink or table cell, is exposed as a single, continuous text stream in both the control view and the content view of the UI Automation tree; object boundaries are ignored.
> The examples also suggest this; for instance:
> > {The URL http://www.microsoft.com is embedded in text}.
> > IUIAutomationTextPattern::RangeFromChild, where the UI Automation element was returned by the previous IUIAutomationTextRange::GetChildren method. Returns the range that represents "http://www.microsoft.com".

hm, maybe I was a bit confused :/

On the other hand the example using an ilustration of a shuttle appears to work as I suggest, which I don't really understand in the context of the bits you quote.

in either case while this certainly matters to screen readers and for fixing this bug it appears just returning E_NOTIMPL for all text methods is enough to make the on screen keyboard work and get us parity with the stuff Jimm mentions with a cleaner archetecture.

Comment 10

[Jim Mathies [:jimm]]

(In reply to Trevor Saunders (:tbsaunde) from comment #9)
> in either case while this certainly matters to screen readers and for fixing
> this bug it appears just returning E_NOTIMPL for all text methods is enough
> to make the on screen keyboard work and get us parity with the stuff Jimm
> mentions with a cleaner archetecture.

First step is to test to see if IAccessibleEx and it's slimmed down version of UIA are compatible with what Metro expects us to support. So I'm starting off trying to get what we have here connected to metro using the UIABridge winrt component as a wrapper.

Comment 11

[Trevor Saunders (:tbsaunde)]

(In reply to Jim Mathies [:jimm] from comment #10)
> (In reply to Trevor Saunders (:tbsaunde) from comment #9)
> > in either case while this certainly matters to screen readers and for fixing
> > this bug it appears just returning E_NOTIMPL for all text methods is enough
> > to make the on screen keyboard work and get us parity with the stuff Jimm
> > mentions with a cleaner archetecture.
> 
> First step is to test to see if IAccessibleEx and it's slimmed down version
> of UIA are compatible with what Metro expects us to support. So I'm starting
> off trying to get what we have here connected to metro using the UIABridge
> winrt component as a wrapper.

ok, it'd be good to know that, since if it doesn't work we may want to move to the normal uia impl setup if it won't work.

Comment 12

[Jim Mathies [:jimm]]

Quick update on this work. I have metro's uia client talking to our accessibility layer. I'm still working through all the props for various roles in uiaRawElmProvider's GetPropertyValue but I do have the few I've been testing with set up and working. I also have basic ITextProvider and IValueProvider patterns set up, and a rudimentary version of ITextRangeProvider.

Currently battling a problem with text inputs where the client queries for both the ITextProvider and IValueProvider interfaces, and queries IValueProvider, but then fails to query ITextProvider for text/selection info. Not sure what's going on there. inspect.exe doesn't seem to have any issues digging down into the control.

The (msaa + IAccessibleEx + UIA) docs are few and far between which is slowing things down. If anyone knows of any good docs or samples on that please post them here.

Comment 13

[Jim Mathies [:jimm]]

I've contacted ms on metro support to confirm this approach is correct. Should hear back next week.

Comment 14

[Anil Kulkarni [MSFT]]

(In reply to Jim Mathies [:jimm] from comment #12)
> Quick update on this work. I have metro's uia client talking to our
> accessibility layer. I'm still working through all the props for various
> roles in uiaRawElmProvider's GetPropertyValue but I do have the few I've
> been testing with set up and working. I also have basic ITextProvider and
> IValueProvider patterns set up, and a rudimentary version of
> ITextRangeProvider.
> 
> Currently battling a problem with text inputs where the client queries for
> both the ITextProvider and IValueProvider interfaces, and queries
> IValueProvider, but then fails to query ITextProvider for text/selection
> info. Not sure what's going on there. inspect.exe doesn't seem to have any
> issues digging down into the control.
> 
> The (msaa + IAccessibleEx + UIA) docs are few and far between which is
> slowing things down. If anyone knows of any good docs or samples on that
> please post them here.

Hi Jim,

There are three UIA patterns which the touch keyboard uses to describe the editability of a field: IUIAutomationValuePattern, IUIAutomationTextPattern, and IUIAutomationTextChildPattern. In order to be considered a text field, the element must implement at least one of the latter two interfaces. Once the element is considered a text field, it is checked to see if the element is read-only. The provider could provide conflicting read-only information for each of the three patterns, so if any of the interfaces reports that the field is not read-only, then that is considered sufficient.

Thanks,
Anil

This posting is provided "AS IS" with no warranties, and confers no rights.