501440 - Support selecting text in touch-enabled devices

Status: RESOLVED FIXED

(Core :: Widget: Win32, defect)

Description

[:Felipe Gomes (needinfo for replies!)]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.1pre) Gecko/20090630 Shiretoko/3.5.1pre (.NET CLR 3.5.30729)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.1pre) Gecko/20090630 Shiretoko/3.5.1pre (.NET CLR 3.5.30729)

Currently when using a touch tablet, it's only possible to pan the page using the 1 or 2 finger scrolling. We should be able to select text as well.

The simple case for selecting text (and fortunately the most common case I think) is when starting the selection horizontally in a page that doesn't have horizontal scrollbars.

For other cases, IE currently does some guessing, apparently based on the distance from nearby text nodes.

We should determine how to deal with both scrolling and panning together and implement it.

Reproducible: Always

Steps to Reproduce:
1. Try to select text on a touch tablet
Actual Results:  
You can't. The page always pan. If there are no scrollbars on the page, nothing happens.

Expected Results:  
User should be able to select text, both from input fields as well as regular text nodes.

Comment 1

[Jim Mathies [:jimm]]

This will likely end up a dupe of bug 492791, but let's leave it open in case splitting the selection stuff off makes sense.

Comment 2

[:Felipe Gomes (needinfo for replies!)]

Attachment: Patch rev1

This patch allows to select text and drag elements through horizontal movements, as can be seen for example in http://script.aculo.us/

This behaves the same way as IE currently does.

Comment 3

[:Felipe Gomes (needinfo for replies!)]

well, let me just improve the information here: this is not just how IE behaves, it is the system default setting, per comment 17 of bug 485101. Horizontal single finger triggers default mouse click+hold (which allows text selection and dragging), and vertical triggers touch panning.

Comment 4

[Jim Mathies [:jimm]]

(In reply to comment #3)
> well, let me just improve the information here: this is not just how IE
> behaves, it is the system default setting, per comment 17 of bug 485101.
> Horizontal single finger triggers default mouse click+hold (which allows text
> selection and dragging), and vertical triggers touch panning.

If the page the user is viewing has horizontal scroll bars visible and they try and pan horizontally using a single finger, would we default to panning or text selection with this patch applied?

Comment 5

[:Felipe Gomes (needinfo for replies!)]

(In reply to comment #4)
> If the page the user is viewing has horizontal scroll bars visible and they try
> and pan horizontally using a single finger, would we default to panning or text
> selection with this patch applied?

We would default to text selection. To pan, the user would have to start the movement vertically. Once a vertical movement triggers panning, then it's possible to keep panning in any direction.

Comment 6

[Robert Strong (they/them - no direct email)]

The pages I tested with the patch worked much better than without and appear to emulate the same behavior as IE. Felipe, could you test the behavior of IE when there is only a horizontal scrollbar to see if it is possible to initiate single finger panning?

Comment 7

[:Felipe Gomes (needinfo for replies!)]

(In reply to comment #6)
> Felipe, could you test the behavior of IE when
> there is only a horizontal scrollbar to see if it is possible to initiate
> single finger panning?

Just tested on IE and turns out you cannot scroll horizontally when there's no vertical scrollbar. It always starts a selection, and you can't scroll the site. An example page for testing: http://www.peter-pearson.com/

Even with two fingers it acts a little weird. It looks like the two fingers triggers a zoom even if I keep my fingers firmly aligned, and then after the zoom it is able to pan due to the zoomed page getting a scrollbar.

Comment 8

[Jim Mathies [:jimm]]

I'm wondering if selection should be tied to the presence of horizontal scroll. (Bug 492791 really ties into this as that information is up in the view.) In playing with it I became accustomed to moving things around with a single finger, the selection stuff seemed to "get in the way". The selection though would be nice for well formatted pages that didn't have horizontal scroll distance showing.

I really think we need to move control over this type of stuff out of widget to an area where we have more knowledge about the state of the view. (We could use events to get this information down into widget, but that seems like a hack to me.) What we could do is add an interface to the gesture stuff that you retrieve through nsWindow, then the view could control how it's configured, the feedback, etc.. I think we're going to need this anyway to fix up some of the other problems with menus and scroll bars.

All of this though really can't be done until we convert from mouse event simulation to some form of touch / gesture gui event.

Comment 9

[:Felipe Gomes (needinfo for replies!)]

Yeah I agree that we will need more fine grained control to cover well most of the use cases for webpages and our UI. But while we're not at it I believe that this change makes the situation much better because it makes it possible to select text, drag tabs and even sites like Google Maps works.  Currently none of this is possible if single finger always pan.  Horizontal scroll is not that common and if the user really needs it it can be triggered with two fingers.

Comment 10

[Jim Mathies [:jimm]]

building up some try builds to take a look.

Comment 11

[Jim Mathies [:jimm]]

https://build.mozilla.org/tryserver-builds/[e-mail]-sftc/

Comment 12

[Jim Mathies [:jimm]]

Hmm, on 7100 two finger pan wasn't working with these builds. I'll get this tablet upgraded today to the RC and see if that addresses it..

Comment 13

[:Felipe Gomes (needinfo for replies!)]

Hi Jim, for these to work on trunk you'll have to have the patch from bug 494281 applied as well.

Comment 14

[Jim Mathies [:jimm]]

(In reply to comment #13)
> Hi Jim, for these to work on trunk you'll have to have the patch from bug
> 494281 applied as well.

That shouldn't affect this as long as I don't close a window.

I guess 7100 is the RC, does that jibe with what you guys have? I'm updating my drivers to 2.59 now to be sure I have the latest.

Comment 15

[:Felipe Gomes (needinfo for replies!)]

Hi Jim, from my tests this did affect the touch. It looks like some sub-windows are created and destroyed during startup, so the touch never worked even in the first window.

But yes, I just checked and the windows 7 RC from here is indeed build 7100. The driver version installed here is the 2.59 so we will be with the same settings. Obs.: the pen works here after we upgraded to 2.59.

Comment 16

[Jim Mathies [:jimm]]

Comment on attachment 386634 [details] [diff] [review]
Patch rev1


Ok, this seems fine with me. At first I found myself trying to drag the page around single finger and accidentally triggering selection, but after a while you get used to it.

Comment 17

[Jim Mathies [:jimm]]

(In reply to comment #15)
> Hi Jim, from my tests this did affect the touch. It looks like some sub-windows
> are created and destroyed during startup, so the touch never worked even in the
> first window.
> 
> But yes, I just checked and the windows 7 RC from here is indeed build 7100.
> The driver version installed here is the 2.59 so we will be with the same
> settings. Obs.: the pen works here after we upgraded to 2.59.

Ok, cool. Lets get sr's on both of these, land them on trunk and nom for approval1.9.1.1, hopfully they'll make it in.

Nice work Felipe!

Comment 18

[:Felipe Gomes (needinfo for replies!)]

Comment on attachment 386634 [details] [diff] [review]
Patch rev1

vlad, could you take a look at this patch?

Comment 19

[Mark Finkle (:mfinkle) (use needinfo?)]

Madhava - is text selection in pannable contextes something we want in Fennec too?

Comment 20

[Robert Strong (they/them - no direct email)]

note: this is specific to Win7's WM_GESTURE events and GID_PAN which isn't part of mobile touch.

Comment 21

[:Felipe Gomes (needinfo for replies!)]

Yes, the code for this patch is win7 sdk based, although the interaction discussed here might be valuable for Fennec too.

Should I be using a different term instead of "touch enabled devices" to make the description clearer about that?

Comment 22

[Robert Strong (they/them - no direct email)]

Pushed to mozilla-central
http://hg.mozilla.org/mozilla-central/rev/db7c21404bb4

Comment 23

[Jim Mathies [:jimm]]

Comment on attachment 386634 [details] [diff] [review]
Patch rev1

This is simply a configuration change that enables single finger text selection. We'd like this for 1.9.1.2.

Comment 24

[Mike Beltzner [:beltzner, not reading bugmail]]

Comment on attachment 386634 [details] [diff] [review]
Patch rev1

Jim: does this affect the mozilla-1.9.1 branch? If so, please renominate for 1.9.1.3

Comment 25

[Robert Strong (they/them - no direct email)]

Comment on attachment 386634 [details] [diff] [review]
Patch rev1

This does affect mozilla-1.9.1 and is a usability improvement when using touch

Comment 26

[Daniel Veditz [:dveditz]]

Comment on attachment 386634 [details] [diff] [review]
Patch rev1

Approved for 1.9.1.4, a=dveditz for release-drivers

Comment 27

[Robert Strong (they/them - no direct email)]

Pushed to mozilla-1.9.1
http://hg.mozilla.org/releases/mozilla-1.9.1/rev/03a61233af0f

Comment 28

[Samuel Sidler (old account; do not CC)]

You sir, should use the ".4-fixed" option of the status1.9.1 flag to indicate this has landed.

Comment 29

[Robert Strong (they/them - no direct email)]

bah... old habits die hard

Comment 30

[Marcia Knous [:marcia]]

Verified fixed on the 1.9.2 branch using Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2b1pre) Gecko/20090923 Namoroka/3.6b1pre (.NET CLR 3.5.30729). Adding keyword.