Open Bug 1716355 Opened 5 years ago Updated 10 months ago

No pointermove events for the initial pen move because of tolerance

Categories

(Core :: Panning and Zooming, defect, P3)

Product:
Core
Component:
Panning and Zooming
Version:
Firefox 91
Type:
defect

Tracking

()

People

(Reporter: davidj, Unassigned)

References

(Blocks 1 open bug)

Details

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36

Steps to reproduce:

  1. Go to https://stackoverflow.com/a/8398189
  2. Draw a curved line quickly with a Wacom or Huion tablet.

Works fine on Chrome. Tested on latest Nightly on Windows 10.

Actual results:

You see a jiggly kinked curve.

Expected results:

Smooth curve.

That example doesn't seem to be using pointer events but mouse events.

Component: DOM: Events → DOM: UI Events & Focus Handling

Yes, you're right. Our app uses Pointer events, but I tried to find a simple demonstration and I see this one does in fact use mouse events. Doing some testing in our code I've confirmed that it only uses Pointer events and the bug still occurs, so it seems to not be specific to Pointer events but also happens during mouse events.

Summary: Pointer events not giving smooth drawing for Huion/Wacom graphics tablets → Pointer/mouse events not giving smooth drawing for Huion/Wacom graphics tablets

Representing Wacom, we recently had a developer reach out for support for this issue that they're only experiencing in Firefox. We like to use the following page for browser event testing, and you can see some very strange behavior in Firefox when compared to other browsers.

https://patrickhlauke.github.io/touch/tracker/multi-touch-tracker-pointer-hud.html

This behavior was not observed in older versions of Firefox, it may be related to a recent change.

This bugs exists for quite some time.
I have made a simple example page here:
https://www.keepvogel.nl/Apps/index_FF.html

  • draw quick loops and the order of connecting the subsequent dots will be wrong
  • it happens only in Firefox Windows, not in Linux
  • I have tried to repair it by taking timestamps into account, but that is not the issue

I have found old mentions of this behaviour, it occurred for the first time somewhere bettween Firefox 60 and 69

Blocks: 1631377
Severity: -- → S3
Status: UNCONFIRMED → NEW
Ever confirmed: true
Priority: -- → P3

Have you also tried Nightly? Does the issue also happen on Nightly? (https://nightly.mozilla.org/)

Flags: needinfo?(sqalab)
Flags: needinfo?(box4)

(In reply to Kagami :saschanaz from comment #6)

Have you also tried Nightly? Does the issue also happen on Nightly? (https://nightly.mozilla.org/)

Yes, It also happens in Nightly.

Flags: needinfo?(box4)

This issue still occurs with Firefox 126.0.

Reviewing behavior with https://patrickhlauke.github.io/touch/tracker/multi-touch-tracker-pointer-hud.html

In Firefox: Hovering the pen, holding still or moving around the tablet, shows that it's considered the primary input device. Touching the pen tip to click, but not move, the pen is still considered the primary input device. Moving the pen while he's clicking changes it to isPrimary: false, which appears to correlate with the jump in cursor movement that disrupts the user's action.

The change in primary input device status does not occur in other browsers.

Flags: needinfo?(sqalab)

The isPrimary issue is similar to what I reported in bug 1892537 and probably a separate issue.

The kinkiness issue seems like an APZ tolerance issue; the pen input does not fire move events until the pointer moves certain amount of distance. That's touch specific and should not apply to pens. (Check the behavior in https://www.keepvogel.nl/Apps/index_FF.html or https://patrickhlauke.github.io/touch/tracker/multi-touch-tracker-pointer-hud.html)

Component: DOM: UI Events & Focus Handling → Panning and Zooming
Flags: needinfo?(tnikkel)
See Also: → 1892537

Kagami, do you have a device that reproduces the problem?

If so, could I ask you to collect some logs please?

  1. In about:logging, in the "log modules" text field, put Widget:5,apz.manager:5,apz.controller:4
  2. Click "Set Log Modules"
  3. Choose "Logging to a file"
  4. Reproduce the issue with the pen tablet
  5. Share the log file

(Or if you prefer to use "Logging to the Firefox Profiler" and take a profile while reproducing the issue and share that, that works as well.)

Flags: needinfo?(krosylight)

Alright, recorded in a fresh profile from mozregression: https://share.firefox.dev/3UXTs6z

Flags: needinfo?(krosylight)

Hmm, I don't see log messages from the log modules listed in comment 10 (Widget:5,apz.manager:5,apz.controller:4) in that profile.

I see a lot of (apz.manager) and (apz.controller) in GPU Process in comment #11's profile, are you not seeing them? This is a new one in case something went wrong. https://share.firefox.dev/3VlJtbB

Flags: needinfo?(botond)

We fire WidgetTouchEvent for pens to achieve Windows scroll-by-pen behavior, which is managed by the pref dom.w3c_pointer_events.scroll_by_pen.enabled. My current assumption is that APZ assumes all WidgetTouchEvent is from touch input and applies TouchBlockState. Maybe we should disable this by checking mInputSource (or add WidgetPenEvent? 🤔)

(In reply to Kagami Rosylight [:saschanaz] (they/them) from comment #13)

I see a lot of (apz.manager) and (apz.controller) in GPU Process in comment #11's profile, are you not seeing them?

My bad, I was looking in the wrong place (parent process).

(In reply to Kagami Rosylight [:saschanaz] (they/them) from comment #14)

We fire WidgetTouchEvent for pens

Thanks! Yeah, this was the main thing I was looking for (what type of widget events the pen produces).

My current assumption is that APZ assumes all WidgetTouchEvent is from touch input and applies TouchBlockState. Maybe we should disable this by checking mInputSource (or add WidgetPenEvent? 🤔)

I think it may be sufficient to condition the tolerance values returned by these functions on the input source, and use smaller values (possibly zero) for MOZ_SOURCE_PEN.

Flags: needinfo?(botond)

Sounds like a good plan! 👍 Do you want to try it yourself or should I?

Hmm actually, conditional tolerance might not be a best plan for scroll-by-pen case.

Do you know whether we apply tolerance for touch-action: none case for touch, and if yes, the background behind it?

Flags: needinfo?(botond)

(In reply to Kagami Rosylight [:saschanaz] (they/them) from comment #18)

Do you know whether we apply tolerance for touch-action: none case for touch, and if yes, the background behind it?

No, the tolerance only applies to the browser's own handling of the touch events (e.g. to scroll natively). The touch events are dispatched to web content immediately without any tolerance threshold, so if e.g. a drawing app uses touch-action: none and responds to touch events by drawing a line, it will start getting touchmove events immediately after the gesture starts. (The web app can always implement its own tolerance threshold, if it wants to.)

Flags: needinfo?(botond)

The touch events are dispatched to web content immediately without any tolerance threshold

Hmm? I don't think that's the case here, if you set touchmove tolerance to a ridiculous amount (e.g. 5) then the page becomes significantly affected with touchscreen input.

(In reply to Kagami Rosylight [:saschanaz] (they/them) from comment #20)

The touch events are dispatched to web content immediately without any tolerance threshold

Hmm? I don't think that's the case here, if you set touchmove tolerance to a ridiculous amount (e.g. 5) then the page becomes significantly affected with touchscreen input.

You're right, I was misremembering. We do drop events (i.e. don't send them to web content) before the tolerance threshold is reached.

This behaviour dates back to bug 1141127. The motivating scenario there was actually a map application which had a different handlers for click (touchstart+touchend) and touchdrag (touchstart+touchmoves+touchend) and it was hard for users to trigger the click handler because the touch hardware would often generate a few touchmoves even for gestures where the user intended it as a tap.

Alright, this is interesting. I saw almost no tolerance in Chrome and wondered why, and it turns out Chrome conditionally applies tolerance based on the event type, not the input type, i.e. touchmove has higher tolerance than pointermove for the same touchscreen input. (Would be interesting to find the relevant line in Blink but I don't immediately have the pointer.)

What do you think with this approach? 👀

Flags: needinfo?(botond)

(In reply to Kagami Rosylight [:saschanaz] (they/them) from comment #22)

What do you think with this approach? 👀

Pointer events are generated entirely in the content process, so if we wanted to dispatch pointer events in cases where we don't dispatch touch events, we'd have to relax the tolerance in the compositor (so the touch events do get sent to the content process, even if they are ultimately not dispatched to the page), and implement the tolerance logic on the content side.

I think it's possible to do in principle, but sounds fairly involved.

In the shorter term, I think reducing the tolerance for pen input devices as described in comment 16 should mitigate the issue effectively.

Flags: needinfo?(botond)

Unfortunately I don't think reducing tolerance will work, reducing it from the default 0.1 to to 0.05 already makes it impossible to click a browser tab without causing scroll. (The current 0.1 is already tricky as pens slip easier than fingers do.)

Oh, huh. I thought pen devices would be more precise in that respect.

So is the idea that in such cases of the pen slipping when you're intending to do a click, it's not a problem if we fire pointermove events to the page, as long as we don't also fire touchmove events (and don't cause native scrolling)?

(In reply to Botond Ballo [:botond] from comment #25)

Oh, huh. I thought pen devices would be more precise in that respect.

"precise" meaning easier to know where the pen pointer is. Meanwhile good pens are more sensitive, meaning the user want to track even a little bit of movement (but without causing false gesture).

So is the idea that in such cases of the pen slipping when you're intending to do a click, it's not a problem if we fire pointermove events to the page, as long as we don't also fire touchmove events (and don't cause native scrolling)?

Yup, I think that way we can fulfill both use cases.

Here are some notes on how a lower tolerance threshold for pointermove events could be implemented:

  • In this block, do not call result.SetStatusAsConsumeNoDefault();, as this is what prevents the event from being dispatched to the content process. (Probably this can be called instead.)
  • In the same block, annotate the event (MultiTouchInput) as being before the tolerance threshold (since the content process will now receive the event, and want to do things differently in that case). This may require adding a new field to MultiTouchInput.
  • Around here propagate the annotation to the WidgetTouchEvent. (This happens in the parent process.)
  • In the content process, check the annotation on the WidgetTouchEvent event, and if it's marked as being before the tolerance threshold, then do not dispatch the event to the page (but still do run the logic that synthesizes pointer events, optionally implementing a lower tolerance threshold for the pointer events as well).

(The last bullet is kind of vague, I think you or Edgar will know more about the details of that.)

clearing ni, Botond seems to have this. If my input is required please don't hesitate to ask again.

Flags: needinfo?(tnikkel)

Hi,

I'm the author of the topic with bug 1892537.

I'm very grateful for your responses and efforts to fix the problem.

I just want to add that in my topic for bug 1892537, Kagami wrote (comment no. 4 in that topic):

For now the workaround is to disable dom.w3c_pointer_events.scroll_by_pen.enabled, but as the name says it'll disable the scroll-by-pen behavior.

I checked this suggestion, and it's helpful, but not 100%. When dom.w3c_pointer_events.scroll_by_pen.enabled is true, the pointer needs to move about 10 pixels to start drawing, but when this option is disabled, the pointer needs to move about 4 pixels to start drawing.

I'll apply this suggestion because I don't use the scroll-by-pen behavior, but it is still not a perfect solution (still, 60% of the problem is gone).

(sorry for confusion, my topic was bug 1893418)

I don't get 4px tolerance but I do get 2px tolerance. Unsure where this is coming from, as I don't see the same behavior for mouse. Might be something with Windows API?

Summary: Pointer/mouse events not giving smooth drawing for Huion/Wacom graphics tablets → No pointermove events for the initial pen move because of tolerance

This is super annoying when writing on pdf.js 😬

Is the work on this problem moving forward?

This certainly is in my wishlist but currently no one is actively working on this.

You need to log in before you can comment on or make changes to this bug.