Closed Bug 783488 Opened 11 years ago Closed 5 years ago

Add an optional caret "handle" for touch screens

Categories

(Core :: DOM: Selection, defect, P3)

Product:
Core
Component:
DOM: Selection
Type:
defect

Tracking

()

Status:
RESOLVED WORKSFORME
Project Flags:
blocking-kilimanjaro +
blocking-basecamp -

People

(Reporter: cjones, Unassigned)

References

(Blocks 1 open bug)

Details

(Whiteboard: [UX-P?][tech-p1]) 

+++ This bug was initially created as a clone of Bug #768503 +++

See discussion in bug 768503 about the handle spec.

blocking-basecamp-, but *really* nice to have.
Blocks: b2g-v-next
Whiteboard: [UX-P?]
Needed for competitive parity, and to fix bad UX.
Whiteboard: [UX-P?] → [UX-P?][tech-p1]
Assigning to Josh temporarily for an interaction spec.  Please reassign as you see fit.

With that in hand, we can decide on the best way to implement.  Ideally this would go in gecko, as decoration on top of the existing caret along with a new event listener to detect caret drags.
Assignee: nobody → jcarpenter
I think we need more than just a marker on the caret. We have a large amount of logic up in front end in all our mobile platforms that needs to be moved down into dom so that it can be shared. 

Each front end is doing selection slightly differently but is based on the original xul fennec selection code, which was kind of a hack.

I think we should consider adding a touch friendly selection mode to dom selection with various controls the front end can flip to get the right behavior.
(In reply to Jim Mathies [:jimm] from comment #4)

> http://mxr.mozilla.org/mozilla-central/source/mobile/xul/chrome/content/
> content.js#1390

This is the old xul fennec code. You can ignore anything in the mobile/xul directory (I think it's going away soon anyway).
(In reply to comment #3)
> I think we should consider adding a touch friendly selection mode to dom
> selection with various controls the front end can flip to get the right
> behavior.

What are the differences between the touch and mouse selection behavior?
(In reply to :Ehsan Akhgari from comment #6)
> (In reply to comment #3)
> > I think we should consider adding a touch friendly selection mode to dom
> > selection with various controls the front end can flip to get the right
> > behavior.
> 
> What are the differences between the touch and mouse selection behavior?

- touch centric drag markers on selection bounds
- snap to word or sentence features would be nice to have
- ignore text whitespace option
- built-in magnify (like android's address bar) would be nice
- ignore content, just select text. our current selection has a habit of selecting underlying content elements like divs rather than just text. This throws off drag marker positioning.
- text input view scrolling doesn't work well with touch input / front end drag markers. For example, you currently can't expand from the start node without mucking with focus.
- text input drag to select off caret (see http://msdn.microsoft.com/en-us/library/windows/apps/hh465334.aspx for what we would like to have in metro.)

All of these behaviors are *hard* to do well via front end code, and we end up duplicating across platforms. Some of this currently can't be done without additional apis. 

There are more pie in the sky type features too, like having a rectangular selection mode vs. caret positioning.
(In reply to comment #7)
> (In reply to :Ehsan Akhgari from comment #6)
> > (In reply to comment #3)
> > > I think we should consider adding a touch friendly selection mode to dom
> > > selection with various controls the front end can flip to get the right
> > > behavior.
> > 
> > What are the differences between the touch and mouse selection behavior?
> 
> - touch centric drag markers on selection bounds

This I believe needs a different caret implementation.  We might need to do some surgery to make it possible to have more than one selection caret though.

> - snap to word or sentence features would be nice to have

We already have support for that (that's what you get when pressing Ctrl+Shift+Left/Right on desktop.)

> - ignore text whitespace option

Not sure what that means.

> - built-in magnify (like android's address bar) would be nice

That is outside of the scope of the selection behavior.  It's sort of a UI thing I think.

> - ignore content, just select text. our current selection has a habit of
> selecting underlying content elements like divs rather than just text. This
> throws off drag marker positioning.

Hmm, I'm not entirely sure what that means.  What kind of "content" would you want us to not select?

> - text input view scrolling doesn't work well with touch input / front end drag
> markers. For example, you currently can't expand from the start node without
> mucking with focus.

I think a lot of this is because of the broken way that the current handles are implemented.  I don't think that the selection code needs to be fixed in order to make this possible.

> - text input drag to select off caret (see
> http://msdn.microsoft.com/en-us/library/windows/apps/hh465334.aspx for what we
> would like to have in metro.)

I'm not really sure which part of that document you're referring to here.

> All of these behaviors are *hard* to do well via front end code, and we end up
> duplicating across platforms. Some of this currently can't be done without
> additional apis. 

Yes, absolutely.  I've been advocating this myself ever since we did the first Android implementation.  :-)  I'm just trying to figure out what changes we're going to need to make in the selection backend code.

> There are more pie in the sky type features too, like having a rectangular
> selection mode vs. caret positioning.

Rectangular selection mode is going to be a lot more work, let's discuss that once we have the basics of this worked out.  :-)
(In reply to :Ehsan Akhgari from comment #8)
> (In reply to comment #7)
> > (In reply to :Ehsan Akhgari from comment #6)
> > > (In reply to comment #3)
> > > > I think we should consider adding a touch friendly selection mode to dom
> > > > selection with various controls the front end can flip to get the right
> > > > behavior.
> > > 
> > > What are the differences between the touch and mouse selection behavior?
> > 
> > - touch centric drag markers on selection bounds
> 
> This I believe needs a different caret implementation.  We might need to do
> some surgery to make it possible to have more than one selection caret
> though.

There's no way to bring these up automatically since front end code doesn't get good events to tell it when selection is added/removed. Displaying these in front end is also hard if you want their position to track changes in the view (e.g. scrolling).

> > - snap to word or sentence features would be nice to have
> 
> We already have support for that (that's what you get when pressing
> Ctrl+Shift+Left/Right on desktop.)

Ok, but how does front end code turn these characteristics on/off without using simulated keyboard events? and wouldn't it be simpler to have all this built-in vs. invoking a bunch of front end script to do it for the dom?

> > - ignore text whitespace option
> 
> Not sure what that means.

On windows desktop selection always grabs the last space in a selection drag. Maybe there's a pref that turns this off, not sure. We don't want to treat touch input like mouse input so we would want to flip a switch on this type of functionality.

> > - built-in magnify (like android's address bar) would be nice
> 
> That is outside of the scope of the selection behavior.  It's sort of a UI
> thing I think.

Maybe. Seems like dom could be a lot more touch input friendly imo though. By moving this type of feature into dom, you also get constant behavior across all of our platforms. I won't quibble with you on this one feature though, doing something like this in front end is ok.

> 
> > - ignore content, just select text. our current selection has a habit of
> > selecting underlying content elements like divs rather than just text. This
> > throws off drag marker positioning.
> 
> Hmm, I'm not entirely sure what that means.  What kind of "content" would
> you want us to not select?

I will file a fresh metro bug on what I'm referring to and cc you.

> > - text input view scrolling doesn't work well with touch input / front end drag
> > markers. For example, you currently can't expand from the start node without
> > mucking with focus.
> 
> I think a lot of this is because of the broken way that the current handles
> are implemented.  I don't think that the selection code needs to be fixed in
> order to make this possible.

The apis aren't there to do this right from the front end. We can make anything work if we add the right apis, but why not make the dom more touch friendly and save all our front ends the effort?

> > - text input drag to select off caret (see
> > http://msdn.microsoft.com/en-us/library/windows/apps/hh465334.aspx for what > > would like to have in metro.)
> 
> I'm not really sure which part of that document you're referring to here.

str:

1) tap in a text input
result: one grabber is displayed at the caret position
2) tap down and drag away using the dragger
result: selection expands and two draggers are displayed on each end of the selection rect.

In addition current copy text interaction is based on front end input event hooks. This kind of functionality could be built into the dom.
(In reply to Jim Mathies [:jimm] from comment #9)
> (In reply to :Ehsan Akhgari from comment #8)
> > (In reply to comment #7)
> > > (In reply to :Ehsan Akhgari from comment #6)
> > > > (In reply to comment #3)
> > > > > I think we should consider adding a touch friendly selection mode to dom
> > > > > selection with various controls the front end can flip to get the right
> > > > > behavior.
> > > > 
> > > > What are the differences between the touch and mouse selection behavior?
> > > 
> > > - touch centric drag markers on selection bounds
> > 
> > This I believe needs a different caret implementation.  We might need to do
> > some surgery to make it possible to have more than one selection caret
> > though.
> 
> There's no way to bring these up automatically since front end code doesn't
> get good events to tell it when selection is added/removed. Displaying these
> in front end is also hard if you want their position to track changes in the
> view (e.g. scrolling).

Yes, we don't have any selection change events, unfortunately.

> > > - snap to word or sentence features would be nice to have
> > 
> > We already have support for that (that's what you get when pressing
> > Ctrl+Shift+Left/Right on desktop.)
> 
> Ok, but how does front end code turn these characteristics on/off without
> using simulated keyboard events? and wouldn't it be simpler to have all this
> built-in vs. invoking a bunch of front end script to do it for the dom?

nsISelectionController.wordMove <http://mxr.mozilla.org/mozilla-central/source/content/base/public/nsISelectionController.idl#171>

> > > - ignore text whitespace option
> > 
> > Not sure what that means.
> 
> On windows desktop selection always grabs the last space in a selection
> drag. Maybe there's a pref that turns this off, not sure. We don't want to
> treat touch input like mouse input so we would want to flip a switch on this
> type of functionality.

layout.word_select.eat_space_to_next_word :-)

> > > - built-in magnify (like android's address bar) would be nice
> > 
> > That is outside of the scope of the selection behavior.  It's sort of a UI
> > thing I think.
> 
> Maybe. Seems like dom could be a lot more touch input friendly imo though.
> By moving this type of feature into dom, you also get constant behavior
> across all of our platforms. I won't quibble with you on this one feature
> though, doing something like this in front end is ok.

Yeah I was talking about the zoom specifically.

> > > - text input view scrolling doesn't work well with touch input / front end drag
> > > markers. For example, you currently can't expand from the start node without
> > > mucking with focus.
> > 
> > I think a lot of this is because of the broken way that the current handles
> > are implemented.  I don't think that the selection code needs to be fixed in
> > order to make this possible.
> 
> The apis aren't there to do this right from the front end. We can make
> anything work if we add the right apis, but why not make the dom more touch
> friendly and save all our front ends the effort?

Yes, please!  I have no objection to that.  :-)

> > > - text input drag to select off caret (see
> > > http://msdn.microsoft.com/en-us/library/windows/apps/hh465334.aspx for what > > would like to have in metro.)
> > 
> > I'm not really sure which part of that document you're referring to here.
> 
> str:
> 
> 1) tap in a text input
> result: one grabber is displayed at the caret position
> 2) tap down and drag away using the dragger
> result: selection expands and two draggers are displayed on each end of the
> selection rect.
> 
> In addition current copy text interaction is based on front end input event
> hooks. This kind of functionality could be built into the dom.

Oh yeah, like I said, I agree!  As far as this functionality, I think we just need to be able to correctly differentiate between collapsed and non-collapsed selections, which we can already do.
(In reply to :Ehsan Akhgari from comment #10)
> > Ok, but how does front end code turn these characteristics on/off without
> > using simulated keyboard events? and wouldn't it be simpler to have all this
> > built-in vs. invoking a bunch of front end script to do it for the dom?
> 
> nsISelectionController.wordMove
> <http://mxr.mozilla.org/mozilla-central/source/content/base/public/
> nsISelectionController.idl#171>

These apis only work from the caret position out. We can write apis and manipulate this via front end code. Just seems like commonly needed mobile functionality like this could go in dom.

> 
> layout.word_select.eat_space_to_next_word :-)

ah, thanks for that tip. 

> > The apis aren't there to do this right from the front end. We can make
> > anything work if we add the right apis, but why not make the dom more touch
> > friendly and save all our front ends the effort?
> 
> Yes, please!  I have no objection to that.  :-)

That was the plan actually, which is how I found this bug.
(In reply to comment #11)
> (In reply to :Ehsan Akhgari from comment #10)
> > > Ok, but how does front end code turn these characteristics on/off without
> > > using simulated keyboard events? and wouldn't it be simpler to have all this
> > > built-in vs. invoking a bunch of front end script to do it for the dom?
> > 
> > nsISelectionController.wordMove
> > <http://mxr.mozilla.org/mozilla-central/source/content/base/public/
> > nsISelectionController.idl#171>
> 
> These apis only work from the caret position out. We can write apis and
> manipulate this via front end code. Just seems like commonly needed mobile
> functionality like this could go in dom.

Hmm, OK I think one discrepancy here is that our selection implementation doesn't support direction agnostic (aka Mac style) selections, and the anchor position of the selection is attached to where the caret would have been positioned if the selection would get collapsed.

Now, fixing that problem and supporting directionless selections is much harder as that involves fixing all of the existing assumptions about the direction of a selection...
(In reply to :Ehsan Akhgari from comment #12)
> (In reply to comment #11)
> > (In reply to :Ehsan Akhgari from comment #10)
> > > > Ok, but how does front end code turn these characteristics on/off without
> > > > using simulated keyboard events? and wouldn't it be simpler to have all this
> > > > built-in vs. invoking a bunch of front end script to do it for the dom?
> > > 
> > > nsISelectionController.wordMove
> > > <http://mxr.mozilla.org/mozilla-central/source/content/base/public/
> > > nsISelectionController.idl#171>
> > 
> > These apis only work from the caret position out. We can write apis and
> > manipulate this via front end code. Just seems like commonly needed mobile
> > functionality like this could go in dom.
> 
> Hmm, OK I think one discrepancy here is that our selection implementation
> doesn't support direction agnostic (aka Mac style) selections, and the
> anchor position of the selection is attached to where the caret would have
> been positioned if the selection would get collapsed.
> 
> Now, fixing that problem and supporting directionless selections is much
> harder as that involves fixing all of the existing assumptions about the
> direction of a selection...

I've run into this a few times. (bug 848594 is a good example of one of the problem we have.) One thing I tried to do was flip the start and end nodes and thus move the caret. Although from what I remember that met with limited success.
(In reply to comment #13)
> (In reply to :Ehsan Akhgari from comment #12)
> > (In reply to comment #11)
> > > (In reply to :Ehsan Akhgari from comment #10)
> > > > > Ok, but how does front end code turn these characteristics on/off without
> > > > > using simulated keyboard events? and wouldn't it be simpler to have all this
> > > > > built-in vs. invoking a bunch of front end script to do it for the dom?
> > > > 
> > > > nsISelectionController.wordMove
> > > > <http://mxr.mozilla.org/mozilla-central/source/content/base/public/
> > > > nsISelectionController.idl#171>
> > > 
> > > These apis only work from the caret position out. We can write apis and
> > > manipulate this via front end code. Just seems like commonly needed mobile
> > > functionality like this could go in dom.
> > 
> > Hmm, OK I think one discrepancy here is that our selection implementation
> > doesn't support direction agnostic (aka Mac style) selections, and the
> > anchor position of the selection is attached to where the caret would have
> > been positioned if the selection would get collapsed.
> > 
> > Now, fixing that problem and supporting directionless selections is much
> > harder as that involves fixing all of the existing assumptions about the
> > direction of a selection...
> 
> I've run into this a few times. (bug 848594 is a good example of one of the
> problem we have.) One thing I tried to do was flip the start and end nodes and
> thus move the caret. Although from what I remember that met with limited
> success.

One extremely hacky solution which might be simple to implement is to flip the anchor and target nodes inside the selection object internally, but that's sucky...  And I'm not sure if making it work properly is easy or not...
(In reply to :Ehsan Akhgari from comment #14)
> (In reply to comment #13)
> > (In reply to :Ehsan Akhgari from comment #12)
> > > (In reply to comment #11)
> > > > (In reply to :Ehsan Akhgari from comment #10)
> > > > > > Ok, but how does front end code turn these characteristics on/off without
> > > > > > using simulated keyboard events? and wouldn't it be simpler to have all this
> > > > > > built-in vs. invoking a bunch of front end script to do it for the dom?
> > > > > 
> > > > > nsISelectionController.wordMove
> > > > > <http://mxr.mozilla.org/mozilla-central/source/content/base/public/
> > > > > nsISelectionController.idl#171>
> > > > 
> > > > These apis only work from the caret position out. We can write apis and
> > > > manipulate this via front end code. Just seems like commonly needed mobile
> > > > functionality like this could go in dom.
> > > 
> > > Hmm, OK I think one discrepancy here is that our selection implementation
> > > doesn't support direction agnostic (aka Mac style) selections, and the
> > > anchor position of the selection is attached to where the caret would have
> > > been positioned if the selection would get collapsed.
> > > 
> > > Now, fixing that problem and supporting directionless selections is much
> > > harder as that involves fixing all of the existing assumptions about the
> > > direction of a selection...
> > 
> > I've run into this a few times. (bug 848594 is a good example of one of the
> > problem we have.) One thing I tried to do was flip the start and end nodes and
> > thus move the caret. Although from what I remember that met with limited
> > success.
> 
> One extremely hacky solution which might be simple to implement is to flip
> the anchor and target nodes inside the selection object internally, but
> that's sucky...  And I'm not sure if making it work properly is easy or
> not...

Yes, this is what I tried. Sorry my terminology isn't up to snuff - start and end nodes == anchor and target.
(In reply to comment #15)
> (In reply to :Ehsan Akhgari from comment #14)
> > (In reply to comment #13)
> > > (In reply to :Ehsan Akhgari from comment #12)
> > > > (In reply to comment #11)
> > > > > (In reply to :Ehsan Akhgari from comment #10)
> > > > > > > Ok, but how does front end code turn these characteristics on/off without
> > > > > > > using simulated keyboard events? and wouldn't it be simpler to have all this
> > > > > > > built-in vs. invoking a bunch of front end script to do it for the dom?
> > > > > > 
> > > > > > nsISelectionController.wordMove
> > > > > > <http://mxr.mozilla.org/mozilla-central/source/content/base/public/
> > > > > > nsISelectionController.idl#171>
> > > > > 
> > > > > These apis only work from the caret position out. We can write apis and
> > > > > manipulate this via front end code. Just seems like commonly needed mobile
> > > > > functionality like this could go in dom.
> > > > 
> > > > Hmm, OK I think one discrepancy here is that our selection implementation
> > > > doesn't support direction agnostic (aka Mac style) selections, and the
> > > > anchor position of the selection is attached to where the caret would have
> > > > been positioned if the selection would get collapsed.
> > > > 
> > > > Now, fixing that problem and supporting directionless selections is much
> > > > harder as that involves fixing all of the existing assumptions about the
> > > > direction of a selection...
> > > 
> > > I've run into this a few times. (bug 848594 is a good example of one of the
> > > problem we have.) One thing I tried to do was flip the start and end nodes and
> > > thus move the caret. Although from what I remember that met with limited
> > > success.
> > 
> > One extremely hacky solution which might be simple to implement is to flip
> > the anchor and target nodes inside the selection object internally, but
> > that's sucky...  And I'm not sure if making it work properly is easy or
> > not...
> 
> Yes, this is what I tried. Sorry my terminology isn't up to snuff - start and
> end nodes == anchor and target.

OK but I was talking about doing that in Gecko.
Assignee: jcarpenter → nobody
Moving to p3 because no activity for at least 1 year(s).
See https://github.com/mozilla/bug-handling/blob/master/policy/triage-bugzilla.md#how-do-you-triage for more information
Priority: P2 → P3
Moving to p3 because no activity for at least 1 year(s).
See https://github.com/mozilla/bug-handling/blob/master/policy/triage-bugzilla.md#how-do-you-triage for more information
I believe AccessibleCaret (implemented in Gecko) should have fulfilled the needs. If there's something we should consider doing, please file a bug blocking Bug 1124074.
Blocks: AccessibleCaret
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.