Closed Bug 768503 Opened 12 years ago Closed 12 years ago

Allow moving cursor to overflowed text in selectable content

Categories

(Core :: DOM: Selection, defect, P2)

Product:
Core
Component:
DOM: Selection
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla17
Project Flags:
blocking-kilimanjaro +
blocking-basecamp +

People

(Reporter: cjones, Assigned: ehsan.akhgari)

Details

Attachments

(1 file)

Patch (v1)
976 bytes, patch
roc
: review+
cjones
: feedback+
Details | Diff | Splinter Review 
The problem to be solved here is that on touch devices, the user's finger adjusts the caret position.  But fingers are much fatter than the caret and text around the caret, so the finger can obscure where the caret position is.

To help avoid this problem, UX has proposed implementing a "caret handle".  The proposed behavior of the handle is
 * The caret is always visible
 * Tap anywhere on the text to instantly position the caret on the spot (approximately, I know) and show the handle for a couple of seconds to allow for fine tuning.
 * Start dragging from the handle to move the caret, and be free to move away from the handle while still dragging (so you can prevent your hand/finger for not seeing where you're moving the caret to)
 * Swipe anywhere over the text anytime to scroll.
 * The handle disappears when not used for a couple of seconds or activity elsewhere (kbd, scroll, etc.)

CC'ing jcarpenter for clarification (can't find cyee in bugzilla).  Also see https://github.com/mozilla-b2g/gaia/issues/1685 .
Do we have precise mockups on what the user interaction would look like?  Do we want the implementation to live in core (which is where the current caret's implementation lives) or do we want to implement this in JS?
Component: Editor → Selection
QA Contact: editor → selection
This is related to bug 695173 (selection in web content) and bug 765079 (selection in text boxes) and probably a dupe of bug 652168.

I am fine with closing out bug 652168 since it's old and was filed for XUL Fennec.

Margaret is currently using XBL and some JS to control the text selection / caret placement. We are open to other approaches too.
(In reply to Ehsan Akhgari [:ehsan] from comment #1)
> Do we have precise mockups on what the user interaction would look like?  Do
> we want the implementation to live in core (which is where the current
> caret's implementation lives) or do we want to implement this in JS?

I'd love to see this in platform tied to drawing the blinking cursor. On Android, the system method of doing this is something like this:

http://www.xiaomiapp.com/wp-content/uploads/2011/04/Android-design-trend-analysis003.jpg
(see image on right).

i.e. a small Frame below the cursor that would be draggable via touch events. The handle should follow the caret (not necessarily at the same point as the user's finger), when it is dragged. We don't have access to this image from Android (not sure if that's possible or not), so the ability to style it with CSS or by some other means would be nice.

Not sure if the B2G team wants something different.
I don't think what native fennec does is the correct approach (even for Fennec).  Adding stuff into the page's DOM is messy, and there might be any number of bugs which would cause Gecko's notion of the selection to get out of sync with that of the front-end.

I don't know much about the platform integration side of things, and how the icon can be customizable from content, but I presume that it should be possible with some hackery in the widget code (which is where we get our other platform dependent images from.)
One question is how the caret handle should behave when the text is contained in an element subject to various CSS effects --- e.g., opacity, clipping, transforms, being covered by other content, etc. Currently we allow the regular caret to be subject to those effects. Is that what you want for the caret handle? I suspect not.
blocking-basecamp: --- → +
blocking-kilimanjaro: --- → +
(In reply to Ehsan Akhgari [:ehsan] from comment #1)
> Do we have precise mockups on what the user interaction would look like? 

More details here:
https://wiki.mozilla.org/Gaia/System/CursorManagement

and in the PDF there.
(In reply to Ehsan Akhgari [:ehsan] from comment #4)
> I don't think what native fennec does is the correct approach (even for
> Fennec).  Adding stuff into the page's DOM is messy, and there might be any
> number of bugs which would cause Gecko's notion of the selection to get out
> of sync with that of the front-end.

The intent in Fennec is to create caret and selection handles using anonymous XBL content, like the video controls. We do not want to inject elements into the DOM in a non-anonymous way.

> I don't know much about the platform integration side of things, and how the
> icon can be customizable from content, but I presume that it should be
> possible with some hackery in the widget code (which is where we get our
> other platform dependent images from.)

For Fennec, we might have a need to dynamically style (using CSS or images) the appearance of the handles depending on the version of Android.
(In reply to JP Rosevear [:jpr] from comment #6)
> (In reply to Ehsan Akhgari [:ehsan] from comment #1)
> > Do we have precise mockups on what the user interaction would look like? 
> 
> More details here:
> https://wiki.mozilla.org/Gaia/System/CursorManagement
> 
> and in the PDF there.

How is "moving the finger away while dragging" supposed to work in cases where, for example, the entire document is editable?  Shouldn't the handle follow the finger in that case?
(In reply to Mark Finkle (:mfinkle) from comment #7)
> (In reply to Ehsan Akhgari [:ehsan] from comment #4)
> > I don't think what native fennec does is the correct approach (even for
> > Fennec).  Adding stuff into the page's DOM is messy, and there might be any
> > number of bugs which would cause Gecko's notion of the selection to get out
> > of sync with that of the front-end.
> 
> The intent in Fennec is to create caret and selection handles using
> anonymous XBL content, like the video controls. We do not want to inject
> elements into the DOM in a non-anonymous way.

Yeah, I understand.

> > I don't know much about the platform integration side of things, and how the
> > icon can be customizable from content, but I presume that it should be
> > possible with some hackery in the widget code (which is where we get our
> > other platform dependent images from.)
> 
> For Fennec, we might have a need to dynamically style (using CSS or images)
> the appearance of the handles depending on the version of Android.

I think with the current implementation in Fennec that is possible, but that's outside of the scope of this bug.  This bug is hopefully about doing this appropriately in the core code, as opposed to hack a selection mechanism in the front-end for it.  :-)
(In reply to Ehsan Akhgari [:ehsan] from comment #8)
> How is "moving the finger away while dragging" supposed to work in cases
> where, for example, the entire document is editable?  Shouldn't the handle
> follow the finger in that case?

The cursor can still only be in certain positions (next to images, not in the middle of them, at the end of lines not way out in white space, at the bottom of a line of text, not in the middle of a letter). In those cases, the caret should stay where the cursor can be and not necessarily where the users finger is.
(In reply to Wesley Johnston (:wesj) from comment #10)
> (In reply to Ehsan Akhgari [:ehsan] from comment #8)
> > How is "moving the finger away while dragging" supposed to work in cases
> > where, for example, the entire document is editable?  Shouldn't the handle
> > follow the finger in that case?
> 
> The cursor can still only be in certain positions (next to images, not in
> the middle of them, at the end of lines not way out in white space, at the
> bottom of a line of text, not in the middle of a letter). In those cases,
> the caret should stay where the cursor can be and not necessarily where the
> users finger is.

Yeah but I don't read the wiki page that way.  It talks about making it possible for the user to move their finger away while dragging to see the handle, as if the cursor is locked into either horizontal (or vertical) movement.
(In reply to Ehsan Akhgari [:ehsan] from comment #11)
> (In reply to Wesley Johnston (:wesj) from comment #10)
> > (In reply to Ehsan Akhgari [:ehsan] from comment #8)
> > > How is "moving the finger away while dragging" supposed to work in cases
> > > where, for example, the entire document is editable?  Shouldn't the handle
> > > follow the finger in that case?
> > 
> > The cursor can still only be in certain positions (next to images, not in
> > the middle of them, at the end of lines not way out in white space, at the
> > bottom of a line of text, not in the middle of a letter). In those cases,
> > the caret should stay where the cursor can be and not necessarily where the
> > users finger is.
> 
> Yeah but I don't read the wiki page that way.  It talks about making it
> possible for the user to move their finger away while dragging to see the
> handle, as if the cursor is locked into either horizontal (or vertical)
> movement.

Ehsan described the behavior correctly, though your right, in a multi-line scenario, the cursor wouldn't be locked in a vertical or horizontal movement but should be able to traverse multiple lines.   When you encounter a situation where you at text boundaries (as per Ehan's examples), your finger would be able to move from the handle.   

We (UX) will update the wiki with more details reflective of this conversation.




 we (UX) will update the doc to reflect
OK, would you mind also updating this bug when you clarify this on the wiki, please?  Thanks!
Priority: -- → P2
Renom if you think we can't ship a v1 without this.
blocking-basecamp: + → ---
Per IRC conversations with a few other folks, I think the best course of action if there is disagreement on whether this blocks or not is to do the following:

- Move the blocking-basecamp flag to ? for re-evaluation
- Indicate a rationale for why you disagree
blocking-basecamp: --- → ?
Chris Lee/Josh:  do either of you have thoughts on this from a UX/product standpoint?
Whiteboard: [blocked-on-input Chris Lee and Josh Carpenter]
Casey can better speak to this once he's back from PTO (returns Tuesday 8/13 after a 6 day stretch), but in the interim:

blocking-basecamp? I haven't read the full thread, but I've encountered this problem already in my device usage. I'd entered a long URL into the Browser, IIRC, and needed to edit "HTTP" to "HTTPS". Without caret management I had no way to drag the cursor back to start of the text. Annoying as hell.

This is less pressing if we can enable users to scroll freely within contents of text field. In the example above, I could have scrolled to left, for example, to reveal start of text, and tapped where I wanted to insert cursor. If we had that, it would be easier to defer full caret management to v2.
I'm going to make a call here and minus this. This hasn't been listed as an explicit product requirement and we just won't be able to match all the features of competing OSs in v1.
blocking-basecamp: ? → -
Whiteboard: [blocked-on-input Chris Lee and Josh Carpenter]
We're also discussing this here: https://github.com/mozilla-b2g/gaia/issues/3270

In absence of caret, we at minimum need way for users to traverse the contents of a text field whose cup runneth over.
IOW, shipping a phone without a way to move the caret in text fields is flat-out unacceptable.
We need an owner here.
blocking-basecamp: - → +
This bug doesn't necessarily block, but if it doesn't, then we need to make overflowed text areas pannable.
This is a *must* have or we'll have tons of customer care calls.

Once additional criteria to consider when deciding if a bug is blocking is the number of customer support calls we believe we would receive.  

Already marked blocking, but we should keep this blocking unless we have an alternative solution per comment 22.
Ehsan, last I heard you were investigating this --- will you be able to take it through?
Assignee: nobody → ehsan
I was not really investigating anything closely, I was just looking to find out information about what kind of user experience we must support and whether we need to expose something to the web in order to make this possible to implement in HTML/CSS/JS.

The short answer is that for the text selection, I think what Gecko already does is enough here.  For caret movement, the selection.modify API should provide you with what you need.  For rendering the caret at the right position, we need to know the location of the selection end points, and I don't think we have any web-exposed APIs for retrieving that information.

Also, the solution for <input type=text> and <textarea> needs to be specialized, since both those elements put their text inside a native anonymous subtree which is not exposed to the web.

Please let me know if I can provide more information here.
Assignee: ehsan → nobody
Oh dear, there's been a considerable amount of miscommunication about this work.

STR the badness here
 (1) Load browser app
 (2) Go to google.com
 (3) Click link to "Privacy"
 (4) Click URL bar to try to edit URL text

If you try to move the caret within the <input>, it's clunky but it works.  Note that the URL text overflows the input box --- if you try to move the caret to the overflowed text, it doesn't work at all.

The goal with this specific bug is to kill two birds with one stone, using a (touch-aware?) caret handle to fix the clunkiness and inability to move the caret.  And do this within gecko instead of with fragile frontend hacks as has been the past m.o.

JP or roc, is there someone who can take over this work?

If not, frontend hackery, I guess ... :(
(In reply to comment #26)
> Oh dear, there's been a considerable amount of miscommunication about this
> work.
> 
> STR the badness here
>  (1) Load browser app
>  (2) Go to google.com
>  (3) Click link to "Privacy"
>  (4) Click URL bar to try to edit URL text
> 
> If you try to move the caret within the <input>, it's clunky but it works. 
> Note that the URL text overflows the input box --- if you try to move the caret
> to the overflowed text, it doesn't work at all.

Huh, OK, I had no idea this bug is filed about this STR.  I found a b2g device at the office which I'm charging right now in order to test what's happening here.  It apparently has a ROM which is a couple of weeks old.  Is that good enough to reproduce the bug?

About what happens inside Gecko, we listen for mousedown, mousemove and mouseup.  If the touch events are translated to those events properly, then everything _should_ work the same way it does in desktop Firefox.

Let me test on the device here to figure out what the exact problem is.

> The goal with this specific bug is to kill two birds with one stone, using a
> (touch-aware?) caret handle to fix the clunkiness and inability to move the
> caret.  And do this within gecko instead of with fragile frontend hacks as has
> been the past m.o.

That is also different from what I had heard from b2g folks in the past!  It is definitely possible to do a caret handle inside Gecko, and I think overall it should be easier than doing it in the front-end (and getting it right of course).  But I was under the impression that an implementation living in Gecko is not interesting to b2g.
(In reply to Ehsan Akhgari [:ehsan] from comment #27)
> (In reply to comment #26)
> > Oh dear, there's been a considerable amount of miscommunication about this
> > work.
> > 
> > STR the badness here
> >  (1) Load browser app
> >  (2) Go to google.com
> >  (3) Click link to "Privacy"
> >  (4) Click URL bar to try to edit URL text
> > 
> > If you try to move the caret within the <input>, it's clunky but it works. 
> > Note that the URL text overflows the input box --- if you try to move the caret
> > to the overflowed text, it doesn't work at all.
> 
> Huh, OK, I had no idea this bug is filed about this STR.  I found a b2g
> device at the office which I'm charging right now in order to test what's
> happening here.  It apparently has a ROM which is a couple of weeks old.  Is
> that good enough to reproduce the bug?
> 

Should.  bjacob also has a recent b2g build.
So I tested the device using the latest nightly.  What I see is that it is completely impossible to scroll the caret, and it doesn't matter at all whether you want to scroll to the parts of the text that are visible in the input box or not.  In fact, what happens is that the browser UI itself is scrolled up and down (which is probably its own bug.)

As a quick update, I chatted with cjones about this on IRC.  The behavior added in bug 707734 is not entirely correct for editable content.  I may have a simple patch to fix it -- I'm planning to test it locally first.
Assignee: nobody → ehsan
Attached patch Patch (v1) — — Splinter Review 
This patch restores Gecko's text selection support for editable content.  It doesn't regress what bug 707734 tried to fix as far as I can tell.

Here is the behavior that I get with this patch:
* In the browser's address bar, I can now tap and hold and move my finger to select text, and I can also scroll past the overflow section.  I can use the IME keyboard to delete the selected text or replace it with something, as one would expect.
* In the browser's content area, tapping on an input, textarea or a contenteditable brings up the keyboard, and tapping again to move my finger to select text makes the IME keyboard disappear and therefore the caret goes away.  This seems to be a bug in the IME keyboard.  I think making the IME keyboard stay on the screen would fix that, since in that case those elements would effectively be treated the same way as the browser's address bar.
* In the browser's content area, the keyboard appears with a lag, so if you tap to activate an input, textarea or contenteditable and don't lift your finger, wait for the IME keybaord to show up and then move your finger, you can do selection, but that is not pleasant at all.
* There seems to be rendering bugs where you select and scroll text in a textarea, so we should probably fix those as part of a follow-up bug as well.

Asking feedback from cjones mostly because I want him to test this patch as well.
Attachment #652467 - Flags: review?(roc)
Attachment #652467 - Flags: feedback?(jones.chris.g)
Comment on attachment 652467 [details] [diff] [review]
Patch (v1)

Ah, the comment needs to be adjusted as well.  I'll do that when landing.
Ehsan - Can you test this on Firefox for Android too? I just want to be sure we don't regress text selection there.
(In reply to comment #32)
> Ehsan - Can you test this on Firefox for Android too? I just want to be sure we
> don't regress text selection there.

Hrm, the pref in question is not set on Android, is it?
(In reply to Ehsan Akhgari [:ehsan] from comment #33)
> (In reply to comment #32)
> > Ehsan - Can you test this on Firefox for Android too? I just want to be sure we
> > don't regress text selection there.
> 
> Hrm, the pref in question is not set on Android, is it?

Checked out the patch. Right, not currently used on Fennec.
Comment on attachment 652467 [details] [diff] [review]
Patch (v1)

Works like a charm!  Nice fix :).
Attachment #652467 - Flags: feedback?(jones.chris.g) → feedback+
Morphing to what was fixed here.
Summary: Add an optional caret "handle" for touch screens → Allow moving cursor to overflowed text in selectable content
https://hg.mozilla.org/mozilla-central/rev/fc8dc06d7592
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla17
Thanks for landing the patch!  I fixed the comment around this code as promised: https://hg.mozilla.org/mozilla-central/rev/9315dc9bb36f
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: