Surface Pen "touches" don't trigger actions until cursor is moved via touchpad

RESOLVED WORKSFORME

Status

()

defect
P3
normal
RESOLVED WORKSFORME
2 years ago
2 months ago

People

(Reporter: whisdol, Unassigned)

Tracking

({regression})

55 Branch
x86
Windows 10
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox-esr52 unaffected, firefox-esr60 wontfix, firefox55 wontfix, firefox59- wontfix, firefox60- wontfix, firefox61- wontfix, firefox62 fixed)

Details

(Whiteboard: tpi:+)

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:55.0) Gecko/20100101 Firefox/55.0
Build ID: 20170317111607

Steps to reproduce:

I'm using a Surface Pro 4 Tablet (with Windows 10) with the smartcover/keyboard in normal "laptop" mode (i.e. not in "Tablet" mode) and Firefox Nightly 55.0a1 (2017-03-17).
When using the Surface Pen to touch and select elements, the elements are "selected" and change color, but no action is triggered until I move the cursor by touching the trackpad. The first touch always works as expected, but no further touches until I move the cursor with the touchpad.
Elements that have exhibited this behavior are:
- Links
- Tabs

The only element I could find that works as expected is the Firefox menu.

In the following description, a "touch" with the Surface Pen means: "touch down on, then lift the pen".

Example 1:
Open two tabs (Tab 1, Tab 2), where Tab 1 is active at first
Using the Surface Pen, touch Tab 2 in the tab bar
Notice how the tab is switched to Tab 1
Using the Surface Pen, touch Tab 1 in the tab bar

----

Example 2:
Open http://google.com in a new tab
Click next to the big search field to remove the focus from it
Using the Surface Pen, touch the input field.
Notice how the input field is now focused
Using the Surface Pen, touch a link (e.g. "About Google")


Actual results:

Example 1:
The color of Tab 1 changes (like it is selected)
Nothing further happens

----

Example 2:
The link turns red and is underlined (like it is currently being clicked)
Nothing further happens


Expected results:

Example 1:
Firefox should switch to Tab 1.
Now, touch the touchpad with your finger to move the cursor. Notice how Firefox switches to Tab 1.

----

Example 2:
Firefox should open the link.
Now, touch the touchpad with your finger to move the cursor. Notice how Firefox opens the link.
Sorry, the steps to reproduce for Example 1 should say:
Example 1:
Open two tabs (Tab 1, Tab 2), where Tab 1 is active at first
Using the Surface Pen, touch Tab 2 in the tab bar
Notice how the tab is switched to Tab 2
Using the Surface Pen, touch Tab 1 in the tab bar
Component: Untriaged → Widget: Win32
Flags: needinfo?(bugmail)
OS: Unspecified → Windows 10
Product: Firefox → Core
Hardware: Unspecified → x86
Interesting, I can reproduce this as well intermittently on my Surface Pro. I got a regression window, although since I couldn't reproduce it 100% reliably it may not be accurate:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=e297bafab&tochange=5ef0e255

This range does have bug 1331551 which seems like a likely culprit.
Blocks: 1331551
Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: needinfo?(bugmail) → needinfo?(sshih)
Assignee: nobody → sshih
Flags: needinfo?(sshih)
Priority: -- → P2
Whiteboard: tpi:+
regression range:

touch pen D-n-D broke with Nightly changeset from 20170301 (works) and 20170302 (broken): 
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=34c6c2f302e7b48e3ad2cec575cbd34d423a9d32&tochange=e91de6fb2b3dce9c932428265b0fdb630ea470d7
This behavior is drastically improved with Nightly 55.0a1 (2017-06-02). I can now switch tabs and open links with the Surface Pen almost without issues.

Playing around with the Pen in this build, I now only noticed weird behavior on the Google My Account page (https://myaccount.google.com/?hl=de) after opening google.de, clicking on "Datenschutzerklärung" ("Privacy Policy") and then "Mein Konto" ("My Account") in the top right. Sometimes, this page completely hangs and I can't select/tap on anything it with the Surface Pen until I move my cursor. But (unlike before) I can't reproduce this reliably, I only saw it three times since the update.

I also managed to get the original behavior described in example 2 of the bug report when trying to open the "Datenschutzerklärung"-link on the Google home page, but only twice: the first time, the link opened and the page turned white, but did not complete loading until I moved the cursor with the touch pad. The second time, the turned red, but did not open and I could not select anything else until I moved the cursor with the touch pad.

I did not manage to trigger unexpected behavior by switching tabs with the Pen.
This may have regressed in Firefox 57. 

I am using a (rather old) Wacom tablet that exhibits the same issue with Firefox 57, and I never saw it with the 55 or 56 releases. Other users also seem to be affected, eg: 
https://www.reddit.com/r/firefox/comments/7d2vy1/getting_proper_support_for_digitizer_pens_in_ff57/
Hi Iulia,
Could you please help us find the regression window for comment 5? The window got in comment 3 shouldn't be the cause for this 57 regression, as bug 1331551 impacted only nightly build, not releases.
Flags: needinfo?(iulia.cristescu)
It could very well be a different bug, but the symptoms are similar. If it is more appropriate to open a new bug report, let me know.
Flags: needinfo?(iulia.cristescu) → needinfo?(alexandru.simonca)
Hello,
I have taken this request since Iulia doesn't have a Surface Pro 2 laptop available.
As far as I can tell, selecting a text on this particular laptop doesn't work as far back as Nightly 53.0a1 (id: 20170114030206. Even though the user on Reddit says that it worked in 56 I can't make it to select text with the stylus. It always scrolls the page. This behavior is not present on the Surface Pro 4 laptop on none of the builds. There must be something wrong with the Surface Pro 2. In all fairness I have been using a Samsung Note stylus but it works for selecting text on Chrome, so I think we are doing something wrong somewhere. 
Leaving the ni? in place for further reference. I'll try a regression range first thing Monday morning.
For what it's worth, I also can't select text on my Surface Pro 4 (Windows 10 version 1709, build 16299.19), neither with Nightly 59.0a1 (2017-11-17) nor with Firefox Stable 56.0.2. However, I feel like I would have noticed not being able to select any text when initially creating this bug report, so maybe it was a change introduced by a Windows Update.
Flags: needinfo?(alexandru.simonca)
I just switched from 52-ESR to 59.0.2 and have the exact same problem. Occasionally, events like link-click and tab-close are not triggered until the mouse cursor is moved.

This is with a Win10-1709, Dell XPS-13 (9333) with a touchscreen and not a pen.
The problem stops if I set:
dom.w3c_pointer_events.enabled = false
dom.w3c_touch_events.enabled = 0

Doing this, however, also breaks coasting scroll actions.

On a possibly related note, when either touch_events or pointer_events are enabled, scrolling via the touchscreen causes the mouse cursor to behave oddly.

For instance in 59: 
1 - Place the cursor over a link, it gets highlighted.
2 - Use the touchscreen to scroll such that a new link is under the expected cursor location.
3 - The new link is highlighted.
4 - Move the mouse/touchpad.
5 - Note that the cursor position has jumped to the last touchscreen contact point.
6 - The new link is no longer highlighted.

In 52-ESR:
1 - Place the cursor over a link, it gets highlighted.
2 - Use the touchscreen to scroll such that a new link is under the expected cursor location.
3 - The new link is highlighted.
4 - Move the mouse/touchpad.
5 - The cursor position remains at the location last assigned by the mouse.
6 - The new link remains highlighted.

To me, this suggests that mouse and touch/pointer events are out of sync and looks like it is related to the event firing problem.
This seems like a pretty bad experience. If we can't fix this we might want to consider disabling pointer events.
Too late to fix this in 59 and it is very late in the beta cycle to take a patch for 60. We could still fix this for 61.
Wasn't able to reproduce neither the issue described in comment 0 or the problem stated in comment 10, using 61.0a1 (2018-04-24), 60.0b15 build1 (20180423171039) and 59.0.2 build1 (20180323154952) on Dell XPS 12 (with a touchscreen and not a pen).
(In reply to Iulia Cristescu, QA [:JuliaC] from comment #13)
> Wasn't able to reproduce neither the issue described in comment 0 or the
> problem stated in comment 10, using 61.0a1 (2018-04-24), 60.0b15 build1
> (20180423171039) and 59.0.2 build1 (20180323154952) on Dell XPS 12 (with a
> touchscreen and not a pen).

I can recreate the behavior with a fresh install of 59.0.2 64bit build 20180323154952 and a new, clean profile.

Additionally, the test found here shows that Firefox is correctly differentiating between mouse and touch events:
https://patrickhlauke.github.io/touch/tests/event-listener.html

Mouse example.
Start with cursor on the test button and mouse off the button:
pointerover
pointerenter (0ms)
pointermove (0ms)
mouseover (0ms)
mouseenter (0ms)
mousemove (0ms)
pointermove (0ms)
mousemove (0ms)
pointermove (0ms)
mousemove (0ms)
pointerout (0ms)
pointerleave (0ms)
mouseout (0ms)
mouseleave (0ms)


Touch example.
Swipe over the button on the touchscreen:
pointerover
pointerenter (0ms)
pointerdown (0ms)
touchstart (0ms)
pointermove (0ms)
touchmove (0ms)
pointerup (0ms)
pointerout (0ms)
pointerleave (0ms)
touchend (0ms)

Combined example.
With mouse over the button, refresh the page.
Swipe the touchscreen elsewhere.
Wait ~5 seconds.
Move the mouse (cursor has jumped to last swipe location):
mouseover
mouseenter (0ms)
mouseout (4800ms)
mouseleave (0ms)
Smaug, can you please take a look at this bug? This sounds like a pretty bad bug.
Rather old one.
Assignee: stone123456 → nobody
Flags: needinfo?(bugs)
Priority: P2 → P3
Flags: needinfo?(bugs)
My original issue from comment 0 does not seem to occur for me in 62.0a1 (2018-05-11) on my Surface Pro 4 with the Surface Pen.
Would you be willing to use mozregression to bisect when it started working for you? You can make use of the --find-fix option to assist with that.
Flags: needinfo?(whisdol)
After testing different builds for some time, I'm not sure what results I come to.
I played around with different builds with the mozregression-tool and found that I was initially unable to reproduce the issue at all in the 55.0a1 (2017-06-02) build (comment 4, although I was able to trigger it in some cases at the time) or in some builds after that.
So I looked for the fix between 2017-03-17 and 2017-06-02. The fix seems to be in 2017-04-06 (the bug exists in 2017-04-05). Mozregression was "unable to find enough data to dissect".

I also noticed that switching, closing or opening a new tab with the Pen does not work intermittently in 2017-06-02 (and other builds, like 2017-08-02). This bug appears to me to be a different bug as moving the cursor does not trigger the intended behavior.

While testing that, I once managed to trigger the bug from comment 4 when opening a link and switching tabs in quick succession (with build 2017-06-03, I think) when only a white page was shown until I moved the cursor. I also managed to trigger the original issue from this bug in 2018-01-22, 2018-03-23, 2018-04-17. The bisection eventually led me to changeset 2e462060cbedc258c3cfbb7aba14e22fd13cf854 / CommitID GDejDQE7sj3, which seems unlikely.

In those three recent nightly builds, a way to reproduce seems to be to open a link (e.g. with the twitter.com-tile on the new tab page) and directly after that, before even the tab title/icon changes, tapping on the "new tab" button, but that's hard to do. If the bug is triggered, switching or closing tabs does not work until the cursor is moved with the touchpad. The issue also does not occur every time, but is actually rather hard to reproduce for me (which might explain that I can't reproduce it after the 2018-04-17 build).

TLDR: Original issue fixed between 2017-04-05/2017-04-06. Only occurring intermittently for me since then, last successful reproduction in 2018-04-17.
Flags: needinfo?(whisdol)
Does that mean current Beta61 builds are also working OK for you, based on that last successful reproduction date? Thanks again for all your diagnostic help!
Flags: needinfo?(whisdol)
The latest Nightly 62 builds (or any Nightly I tested after 2018-04-17) have been working ok for me, I did not encounter the issue. I did not test any Beta builds for 61 (and am currently unable to as my Surface is being exchanged due to flickergate).
Flags: needinfo?(whisdol)

According to the latest comment, we should be able to close it.

Status: NEW → RESOLVED
Closed: 2 months ago
Flags: needinfo?(bugs)
Resolution: --- → FIXED

Bugbug thinks this bug is a regression, but please revert this change in case of error.

Keywords: regression
You need to log in before you can comment on or make changes to this bug.