1544973 - Using atk_text_set_selection in more than one element causes selection to fail for all but the most recent element

Status: NEW

(Core :: Disability Access APIs, defect, P3)

Description

[Joanmarie Diggs]

1. Launch Firefox and view: data:text/html,<div>foo</div><div>bar</div><div>baz</div>

2. Launch Accerciser, and for each accessible object corresponding to a div in the content:

   a. Select the accessible object in Accerciser's tree of objects

   b. Type: pyatspi.Atspi.Text.set_selection(acc, 0, 0, -1)

3. Return focus back to Firefox press Ctrl+C to copy the selection.

4. Switch to a plain-text editor and press Ctrl+V to paste the copied selection.

Expected results: Three following text would be pasted:

foo
bar
baz

Actual results: The following text is pasted:

baz

Note: When I perform these steps using Accerciser (as opposed to using some local changes I've made to Orca), I get a visual hint that things are about to fail. Namely, when I return focus to Firefox, the first two elements are highlighted with a grey background whereas the third element is highlighted with the normal color (happens to be blue in my environment). But all three elements do remain highlighted.

Impact: This completely blocks my ability to add support in Orca to select text via AT-SPI2. Adding this support to Orca is desired both to improve presentation of selection changes, aas well as to work around missing text-selection-changed events (which cause Orca to not present anything when the selection changes). Therefore, it would be super if this issue could be prioritized and fixed. Thanks in advance!!

Comment 1

[James Teh [:Jamie]]

Does this behave better if you use add_selection instead of set_selection? This seems to work on Windows.

This one is a bit tricky. I can understand that it's confusing that set_selection wouldn't work here, since we're dealing with different objects. On the other hand, we're really dealing with a single selection here; it just happens to be split up into multiple objects, since that's the way semantics are exposed.

Dealing with multiple object selection is quirky/confusing/edge-casey enough that for IA2, we're looking at a selectionRanges API which allows you to pass an entire range (anchor object, anchor offset, active object, active offset) in one call. Would there be any interest in doing tihs for ATK?

Comment 2

[Joanmarie Diggs]

(In reply to James Teh [:Jamie] from comment #1)

Does this behave better if you use add_selection instead of set_selection? This seems to work on Windows.

Huh. Yes and no. Yes, in that all of the elements are selected. No in that result of the copy/paste test is "foobarbaz" rather than "foo\nbar\n\baz".

This one is a bit tricky. I can understand that it's confusing that set_selection wouldn't work here, since we're dealing with different objects. On the other hand, we're really dealing with a single selection here; it just happens to be split up into multiple objects, since that's the way semantics are exposed.

Originally (in my implementation in Orca), I had used add_selection instead of set_selection because that seemed more logical both in terms of the API (as described in the ATK docs) and functionality. For instance, if the user is selecting by char, word, or line in a single element, all I want to do is extend the selection by the new range. That seemed to call for add_selection rather than set_selection.

But then I discovered that if I use add_selection rather than set_selection, I kept getting new indexed selections. Thus selecting the characters in "foo" one at a time (e.g. user is pressing Shift+Right), add_selection causes me to have three selections, each of length 1, rather than one selection of length 3. And if the user then did Ctrl+Shift+Left, I have to iterate through them all to do remove_selection(). That struck me as silly. set_selection, while inconvenient, gave me the desired result of having one selection rather than three. And it appeared to work, at least visually (looking at the blue highlight being where it belonged).

Furthermore, some apps (e.g. LibreOffice Writer) do support non-contiguous selection. For instance, given the space-separated string "foo bar baz", one can Ctrl + double click on "foo" and then Ctrl + double click on "baz" and select "foo" and "baz" with "bar" left unselected. While I didn't create ATK/AT-SPI2, this struck me as a more plausible use case for add_selection: Selection 0 has the range associated with "foo" and Selection 1 has the range associated with "baz".

Getting back to Gecko: I suppose I can use set_selection within an element and add_selection otherwise.... But that still seems contrary to the API. Do I have to live with that? Or is it possible to get set_selection working across elements?

If I do have to live with it, the goal is of course to replicate the selection behavior one achieves via the keyboard and/or mouse via accessibility API. And "foobarbaz" (sans newline chars) is not the expected result. Can that be fixed?

Dealing with multiple object selection is quirky/confusing/edge-casey enough that for IA2, we're looking at a selectionRanges API which allows you to pass an entire range (anchor object, anchor offset, active object, active offset) in one call. Would there be any interest in doing tihs for ATK?

Perhaps. But I am still seeking a solution that I can apply now that doesn't require bumping dependencies for end users, some of whom stick with long-term-stable releases.

So, given all this.... What should we do?

Comment 3

[James Teh [:Jamie]]

(In reply to Joanmarie Diggs from comment #2)

Huh. Yes and no. Yes, in that all of the elements are selected. No in that result of the copy/paste test is "foobarbaz" rather than "foo\nbar\n\baz".

Ug. So I think what's happening here is that this is creating a non-contiguous selection of the text "foo", "bar" and "baz", without selecting the actual "blocks" themselves. Selecting a full block would require you to select its embedded object in the document; e.g. (0, 1) for "foo\n", (0, 2) for "foo\nbar\n", etc.

Here we start to get into confusingly ambiguous territory. If I say select between the start and end offset of a div, do I mean the start and end of everything in that div or do I mean the div as a whole? The unambiguous way is to do what I noted above (specify the embedded object), but that is kinda tricky sometimes, especially if you want to select the entirety of one div but only part of another; e.g. "foo\nb". In that case, you have to select foo's embedded object in the document, but only the "b" within the bar div.

Furthermore, some apps (e.g. LibreOffice Writer) do support non-contiguous selection. For instance, given the space-separated string "foo bar baz", one can Ctrl + double click on "foo" and then Ctrl + double click on "baz" and select "foo" and "baz" with "bar" left unselected. While I didn't create ATK/AT-SPI2, this struck me as a more plausible use case for add_selection: Selection 0 has the range associated with "foo" and Selection 1 has the range associated with "baz".

As above, that's precisely what's happening with Gecko. The problem for Gecko is that set_selection is supposed to change the bounds of (AKA replace) an existing selection. Are we talking about the entire selection or just the selection within this object? Intuitively, the latter. But then when you want to select some text (disregarding the existing selection), you have to remove all other selections in the document because there might be existing selections in other objects you don't know about.

Interestingly, removing all selections in Gecko is fairly easy because you can just removeSelection for all selections on the document accessible.

Comment 4

[Joanmarie Diggs]

See also this list-item/list-marker selection bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1536145

Comment 5

[Joanmarie Diggs]

I'm still working on seeing what all I can sort out within Orca to get this working, but regarding this comment:

(In reply to James Teh [:Jamie] from comment #3)

Interestingly, removing all selections in Gecko is fairly easy because you can just removeSelection for all selections on the document accessible.

That seems to only work reliably if you use add_selection. If instead you use set_selection on all three elements (as described in the opening report), calling get_n_selections on the document returns 1, and calling remove_selection to remove it only removes the highlight from the most-recently-selected element.

Would it make sense for Gecko to update the selections owned by the document when either set_selection or add_selection is used?

Comment 6

[Joanmarie Diggs]

@jamie: In addition to my question in comment 5, I have a follow-up question: If I want to have at most one selection per element, how do I achieve that? For instance, consider two paragraphs with 50 characters each. The user presses Ctrl+Home and then Shift+Right 100 times. (Yeah, unlikely, but not impossible.)

If I do what you suggest and use add_selection, I will wind up with 50 selections per element, each with length of 1. What I want is 1 selection per element, each with length of 50.

Based on what you stated, I was hoping that I could use add_selection when nothing in the element was selected, and then update/replace that existing selection. But as soon as I replace an existing selection using set_selection, I'm back to the problem stated in the opening report (i.e. as if I never used add_selection in the first place).

Is the behavior I'm seeing in Gecko really the correct behavior?

Comment 7

[James Teh [:Jamie]]

(In reply to Joanmarie Diggs from comment #5)

Interestingly, removing all selections in Gecko is fairly easy because you can just removeSelection for all selections on the document accessible.

That seems to only work reliably if you use add_selection. If instead you use set_selection on all three elements (as described in the opening report), calling get_n_selections on the document returns 1, and calling remove_selection to remove it only removes the highlight from the most-recently-selected element.

I don't really understand the highlight on the previous elements. When you copy, you don't get the text from them, so I don't understand what that highlight means. There's definitely something wrong here, and looking briefly at the code, I don't fully understand the intent.

Would it make sense for Gecko to update the selections owned by the document when either set_selection or add_selection is used?

I'd say so, yes, though there's still the question of whether set_selection in an object should replace all selections or just the one in this object.

(In reply to Joanmarie Diggs from comment #6)

Is the behavior I'm seeing in Gecko really the correct behavior?

To be clear, I don't think it's correct behaviour. Worse, I'm not even sure what the behaviour should be. I don't think either ATK or IA2 really thought about cross-object selections when this was specified.

On a single object, this is really simple: set_selection replaces (or adds if non-existent), add/remove_selection adds/removes. For multiple objects, it's trickier. Should set_selection just replace the selection within the element (ignoring all other elements)? If so, how does a client replace an existing selection entirely (across objects) easily?

How does set_selection work in LO (where there is no document hypertext)? How do you completely replace an existing selection (across objects) there?

Comment 8

[Joanmarie Diggs]

(In reply to James Teh [:Jamie] from comment #7)

[...]

I'd say so, yes, though there's still the question of whether set_selection in an object should replace all selections or just the one in this object.

I would say it should replace the specified selection, where "specified" means "the indexed selection on just that accessible object."

According to https://developer.gnome.org/atk/stable/AtkText.html#atk-text-set-selection:

gboolean atk_text_set_selection (AtkText *text, gint selection_num, gint start_offset, gint end_offset);
Changes the start and end offset of the specified selection.

[...]

How does set_selection work in LO (where there is no document hypertext)? How do you completely replace an existing selection (across objects) there?

Seems that it doesn't. I recreated the scenario from my opening report here, namely creating a Writer document with three paragraphs: One for "foo", one for "bar", and one for "baz". Having done so: Using set_selection or add_selection on any of the paragraphs causes the chosen paragraph to become selected BUT, it causes any other selection to be cleared. :(

And for what it's worth, WebKitGtk doesn't yet implement add_selection. And when you use set_selection, it does what LO Writer does.

Long way of saying: It's all broken. Now's our chance to figure out what the right thing is and make it so. :)

Just out of curiosity, how does NVDA handle selection in Gecko? (Orca currently lets Gecko consume selection-related keystrokes and then does its best to present the result in response to the events emitted.)

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

The priority flag is not set for this bug.
:Jamie, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 10

[James Teh [:Jamie]]

(In reply to Joanmarie Diggs from comment #8)

Just out of curiosity, how does NVDA handle selection in Gecko?

In editable content, NVDA does similarly to Orca: it lets Gecko handle selection and queries the selection in response to events/commands.
In browse mode, NVDA can't manipulate Gecko's selection at all at present. You can copy text, but NVDA manages a virtual selection, within which it can only copy flat text, no semantics. So, fixing this would definitely help NVDA too.