1601537 - Accessibles with text leaves are not navigable with character or word granularity

Status: RESOLVED FIXED

(Core :: Disability Access APIs, defect, P1)

Description

[Marco Zehe (:MarcoZ)]

All text on a page within accessibles that don't have their own name, but have text leaf accessibles, is not navigable by character or word granularity. Even when virtual cursor lands on them with standard navigation, character or word navigation cannot walk into that text. Real-life example: https://www.heise.de/newsticker/meldung/Hirnabbau-bei-Antarktis-Aufenthalt-Problem-auch-bei-Fluegen-ins-All-4603988.html.

STR:

1. Start on the main heading.
2. Switch TalkBack granularity to character.
   • Result: The heading can be navigated. It has a name, and it has inner text.
3. Navigate further.
   • Result: TalkBack says "Paragraph". But the text is not read.
4. Switch to standard navigation and swipe left or right.
   • Result: TalkBack lands on the text.
5. Switch back to character granularity while focused on the paragraph, and try to navigate it.
   • Result: Text is not navigable, it skips to the next item.
6. Navigate further until you hear text again.
7. Switch to standard granularity and swipe right then left to find out what you landed on.
   • Result: Depending on whether an ad is shown or such, it is probably a link or a figure element with a name. Or a sub heading.

Comment 1

[Marco Zehe (:MarcoZ)]

Attachment: Testcase

This simple testcase shows all kinds of issues with paragraph and span and div navigation, as well as showing the differences with links and headings.

Comment 2

[James Teh [:Jamie]]

I've finally managed to figure out some of what's going on here and it's not fun. I even had to dig into Talkback logging and source code to work it out.

The crux of the issue is that even though text navigation is async (you perform an action on the focused accessible and then wait for a text traversal event), firing a traversal event with a different accessible from the focused accessible is not supported by Talkback. This bites us in two ways:

1. For paragraphs, divs, spans, etc., a11y focus goes to text leaf nodes. However, when you navigate text, we fire an a11y focus event then a text traversal event on the HyperTextAccessible parent (e.g. paragraph).
   • Firing a11y focus causes the entire HyperTextAccessible to be reported.
   • If we don't fire a11y focus, Talkback won't update the origin it navigates from. That is, it will just start from the last a11y focus, which will result in incorrect offsets. Unfortunately, Talkback doesn't update the origin on text traversal events.
   • Even if Talkback did update the origin, that breaks focus for the user if the HyperTextAccessible has multiple text nodes. For example, if you have a paragraph containing a link followed by some text and the user is focused on the text, navigating the text would throw focus onto the paragraph, at which point flicking right would take them to the link.
   • Firing a text traversal event on a HyperTextAccessible (e.g. paragraph) without a label seems to cause the text traversal event to be treated as having empty text:

     12-10 16:53:17.755  2556  2556 D ProcessorEventQueue: Processing event: EventType: TYPE_VIEW_TEXT_TRAVERSED_AT_MOVEMENT_GRANULARITY; EventTime: 636765839; PackageName: org.mozilla.geckoview_example; MovementGranularity: 0; Action: 0 [ ClassName: android.view.View; Text: [First Second]; ContentDescription: ; ItemCount: -1; CurrentItemIndex: -1; IsEnabled: true; IsPassword: false; IsChecked: false; IsFullScreen: false; Scrollable: false; BeforeText: ; FromIndex: 0; ToIndex: 1; ScrollX: -1; ScrollY: -1; MaxScrollX: -1; MaxScrollY: -1; AddedCount: -1; RemovedCount: -1; ParcelableData: null ]; recordCount: 0
     12-10 16:53:17.756  2556  2556 I TextEventInterpreter: interpretation: Event=EVENT_TYPE_INPUT_CHANGE_INVALID Reason="Text is empty." 
     

     I still can't figure out why this happens. As you can see from the log, there is indeed text in the event, and the part of the Talkback code that logs this "Text is empty." message should definitely be getting this text. I've tried setting contentDescription and itemCount which parts of that code also check, but no change. And yet for a heading or a link, this works just fine.
     If I give the paragraph a label, this problem seems to go away. While I couldn't find anything scouring the Talkback code, my guess is that it ends up fetching some data like text from the node instead of the event? This is somewhat separate from the other issues here.
2. Normally, when you navigate to the end of the text in a node, Talkback moves into the next node. Unfortunately, as explained above, Talkback doesn't update the origin based on text traversed events, and we don't want to fire a11y focus because it reports the entire node. The way you have to manage this with Talkback is to return false from performAction when there's no more text to navigate, at which point Talkback tries navigating in the next node.
   • This is problematic for us because we can't perform the navigation synchronously in order to know whether there's more text, since we might have to go cross-process to do that. I guess we could use sync IPC, but ug. Or maybe we could cache the length of the text in the Java layer and use that to determine whether further navigation is possible.

Comment 3

[James Teh [:Jamie]]

I think what we're going to need to do here is make sure the result of text navigation gets translated to the originating Accessible where possible. For example, if we requested text navigation on a text leaf and Pivot::NextText returns the paragraph, we need to translate the offsets back to the original text leaf.

We should probably deal with the inability to text nav into the next node in a separate bug.

Comment 4

[James Teh [:Jamie]]

(Removed duplicate comment.)

Comment 5

[James Teh [:Jamie]]

Attachment: Bug 1601537: Fix text navigation in text leaf Accessibles on Android. ?MarcoZ

For paragraphs, divs, spans, etc., a11y focus on Android goes to text leaf Accessibles, rather than to the HyperTextAccessible container.
This does make sense, as these containers frequently embed other content, so the text needs to be reachable as a separate item.
However, previously, performing text navigation on these text leaf Accessibles returned the HyperTextAccessible parent.
This isn't supported by Talkback, and even if it were, it causes other problems; e.g. a11y focus being lost if the user was focused on a child other than the first child of such a container.
Therefore, if text navigation was performed on a text leaf Accessible, we now return a result within the text leaf Accessible if possible, rather than the HyperTextAccessible.

1. Make AccessibleWrap::GetTextContents support text leaf Accessibles (for both local and remote proxied Accessibles).
   This is used when providing text for text traversal events.

2. When navigating text on Android, we use Pivot::Next/PrevText.
   However, this will always return a HyperTextAccessible, even when starting on a text leaf.
   Therefore, if the result from Pivot::Next/prevText resides entirely within the same text leaf, translate the offsets from the HyperTextAccessible so they're relative to the text leaf and return the text leaf.

3. Pivot::Next/PrevText already supported starting from a text leaf Accessible.
   However, they ignored the offsets, which meant that navigating from a text leaf would always navigate to the start/end of the text leaf.
   Now, if a text leaf is passed to Pivot::Next/PrevText, the offsets (if specified) are translated to the HyperTextAccessible parent first.

4. Adjust the existing character/word/line tests so they ensure that navigation returns the node that has a11y focus; i.e. the text leaf.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f7f5cecc40dd
Fix text navigation in text leaf Accessibles on Android. ?MarcoZ r=MarcoZ

Comment 7

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/f7f5cecc40dd