1493128 - Browser recognizes touch-screen capability incorrectly

Status: VERIFIED FIXED

(Core :: Layout, defect, P2)

Description

[Daniel Bodea [:danibodea]]

[Affected versions]: 
- v64.0a1

[Affected platforms]:
- Windows

[Steps to reproduce]:
1. Open browser on a touch-screen capable device.
2. Open page: https://hiikezoe.github.io/interaction-media-features.html
3. Disconnect any other pointing devices (like mouse/touch-pad) so that the touch-screen is the only available pointing device.
4. Notice that the page's content shows that the browser detects the touch-screen incorrectly.

[Expected result]:
pointer: coarse
any-pointer: coarse
hover: none
any-hover: none

[Actual result]:
pointer: coarse
any-pointer: fine
any-pointer: coarse
hover: none
any-hover: hover

[Additional notes]:
- This issue does not occur in the case of the Chrome browser.

Comment 1

[Hiroyuki Ikezoe (:hiro)]

From a comment [1] for the function used for any-pointer in Chrome.

 // - If the device is a detachable/convertible win8/10 device and the keyboard/ 
 // trackpad is detached/flipped it will always return POINTER_TYPE_COARSE. 

So, I am pretty sure that 'any-pointer:fine' doesn't appear on Chrome if the device is used as tablet mode even if there is a mouse connected.

[1] https://chromium.googlesource.com/chromium/src.git/+/d157b3968431272cf98ef6b030886c938bcd1047/ui/base/touch/touch_device_win.cc#26

Comment 2

[Hiroyuki Ikezoe (:hiro)]

Brian told me last night on his Surface there is a device appearing in the device manager on Surface with slate mode.  The device name is 'Intel Precise Touch Device', I believe the device is the touchscreen of the Surface and GetSystemMetrics(SM_MOUSEPRESENT) that we are using to tell whether there is no mouse (fine device) or not queries the touchscreen device.

So we have to figure out another way to tell whether there is no mouse or not.  After some googling, what I can think of now is that 

1) Count mouse type devices with SetupDiEnumDeviceInterfaces [1]
2) If the count is 1 and if IsTabletDevice() or IsTouchDeviceSupportPresent() returns true, the device should be COARSE

[1] https://msdn.microsoft.com/ja-jp/library/cc429195.aspx

Comment 3

[Hiroyuki Ikezoe (:hiro)]

(In reply to Hiroyuki Ikezoe (:hiro) (PTO on Sep 26) from comment #2)

> [1] https://msdn.microsoft.com/ja-jp/library/cc429195.aspx

Err, wrong link.  Here is the correct link.

https://msdn.microsoft.com/ja-jp/library/cc429192.aspx

Comment 4

[Hiroyuki Ikezoe (:hiro)]

:danibodea, can you please try this binary to see if it works as expected?

Thanks!

https://treeherder.mozilla.org/#/jobs?repo=try&revision=70af8a1754efb1a617711c8d3782c2034f83c9d7

Comment 5

[Daniel Bodea [:danibodea]]

:hiro, I have tried a build from this revision, more exactly, Windows 2012 x64 opt, Bs, artifact uploaded: target.zip
https://queue.taskcluster.net/v1/task/GuezljoUSJe9RZFqJJn2jw/runs/0/artifacts/public/build/target.zip

It appears that the issue in question was fixed, but there is another situation:
When the mouse remains the last pointing device on the system and then it gets removed, the test page does not show "none" to all, in fact, the test page's content does not change at all, as if it does not detect the removal of the mouse. 
(Probably relevant: The Chrome browser does not detect the connection of the mouse at all, let alone the disconnection.)

All the other situations appear to be working as intended IF the following situation is expected as correct:
When touch-screen and mouse are connected/enabled the following results are seen:
pointer: coarse
any-pointer: fine
any-pointer: coarse
hover: none
any-hover: hover

Thanks.

Comment 6

[Hiroyuki Ikezoe (:hiro)]

Thanks!

(In reply to Bodea Daniel [:danibodea] from comment #5)

> It appears that the issue in question was fixed, but there is another
> situation:
> When the mouse remains the last pointing device on the system and then it
> gets removed, the test page does not show "none" to all, in fact, the test
> page's content does not change at all, as if it does not detect the removal
> of the mouse. 

It's the case where the touchscreen is mis-recognized as 'mouse'.  I believe this issue happens in the first place.

I think I found another approach to use MouseCapabilities [1] in UWP.
:danibodea, could you please this try?  If it didn't work, there is no other way I can think of.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=99b1f311381094318df63ca725543da460da2019

> (Probably relevant: The Chrome browser does not detect the connection of the
> mouse at all, let alone the disconnection.)
> 
> All the other situations appear to be working as intended IF the following
> situation is expected as correct:
> When touch-screen and mouse are connected/enabled the following results are
> seen:
> pointer: coarse
> any-pointer: fine
> any-pointer: coarse
> hover: none
> any-hover: hover

Yes, this is the result what I am expecting.

[1] https://docs.microsoft.com/en-us/uwp/api/windows.devices.input.mousecapabilities

Comment 7

[Hiroyuki Ikezoe (:hiro)]

CCing masayuki, I will probably ask him to review.

Comment 8

[Daniel Bodea [:danibodea]]

Hello Hiro!

This new build did not fix the previous issue where the only pointing device left connected is a mouse, and the parameters don't change to "none" after its removal. I have to mention again that this is also how Chrome behaves.

Furthermore, this new build introduces a new issue: when touch-screen is activated and a touch-pad connected, then, if the touch-pad gets disconnected, the results are incorrect, basically re-introducing the original issue of this bug, as follows:

Actual: 
pointer: coarse
any-pointer: fine
any-pointer: coarse
hover: none
any-hover: hover

Expected: 
pointer: coarse
any-pointer: coarse
hover: none
any-hover: none

My opinion is that the behavior of the previous build (the one from comment 4) is the better one (if there is no other way). It seems that the Chrome developers used the same or a similar method since our only remaining issue is also occurring on their browser. Furthermore, Chrome has other bad behaviors besides this one. A full comparison report would take some time, but I can do it if needed. 

Thanks!

Comment 9

[Hiroyuki Ikezoe (:hiro)]

Thanks!  That's a bad news though. :/
So, the MouseCapabilities seems to contain the touchscreen on Surface for some reasons. :/

Ok, here is another attempt that output pointer device types to stdout.
:danibodea, can you please try this binary?  You have to run firefox.exe in a shell something like this (you probably need to install MozillaBuild to run a shell);

 firefox.exe 2>&1 | tee

When open https://hiikezoe.github.io/interaction-media-features.html, you will see some log like this;

 Found 1 devices: Mouse

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c7a4a6a0540fe96d958df56a90d96680cbc4e49d

In a sample code on MSDN, PointerDeviceType looks like independent value, but I suspect it's bitwise enum something like Touch|Mouse. 

[1] https://code.msdn.microsoft.com/windowsapps/Input-device-capabilities-31b67745/sourcecode?fileId=43661&pathId=786035579

Comment 10

[Daniel Bodea [:danibodea]]

Hi,

I have tried what you asked and I might be missing some info on the process because these were my steps and results:

1. I have installed MozillaBuild on Surface.
2. I have downloaded the try build from the treeherder link provided.
3. Opened Windows PowerShell and navigated to the build's location (in PowerShell).
4. I have used the command ".\firefox.exe 2>&1 | tee output.txt".
5. The browser opened as intended, I opened the test page (https://hiikezoe.github.io/interaction-media-features.html).
6. Using Windows Explorer, went to the location of the browser build (where the shell created the output.txt file).
7. Opened output.txt and saw no string similar to "Found 1 devices: Mouse". 

This is what the output.txt file shown:
"
.\firefox.exe : 
At line:1 char:1
+ .\firefox.exe 2>&1 | tee output.txt
+ ~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
###!!! [Child][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost


[GPU 7844, Chrome_ChildThread] WARNING: pi
pe 
err
or: 109: f
ile
 
z:
/
build
/bu
ild
/
src/
i
p
c/c
h
romi
u
m/sr
c/c
h
rome
/common
/i
pc
_
chann
el_
w
in.cc
,
 l
i
ne 
346



###!!! [Child][MessageChannel::SendAndWait] Error: Channel error: cannot send/recv
"

Am I missing something? I couldn't check the last binary, given the fact that I could not see the connected pointing devices for a more informed report, as you would like me to. I am guessing that the last error is the culprit. What do you suggest?

Comment 11

[Hiroyuki Ikezoe (:hiro)]

Hmm the tee in Windows' PowerShell is somewhat different from what I know of... 

You must run the command in the shell spawned by start-shell in the MozillaBuild.

Comment 12

[Hiroyuki Ikezoe (:hiro)]

Brian tested the build on his Surface. And the output log on slate mode is something like this;

  Found 4 devices: Pen Touch Touch Mouse

This means that GetPointerDevices[1] used in the build also returns a mouse even if no mouse is connected to Surface.
So, I think there is no way to tell whether there is no mouse or not at this moment.

I will take the initial approach in comment 2, it's not perfect as :danibodea mentioned in comment 5 (it mis-recognizes as if there is a mouse device even if there is no mouse connected in the case where the touchscreen is disabled on Surface), but I think the situation rarely happens.

[1] https://docs.microsoft.com/en-us/uwp/api/windows.devices.input.pointerdevice.getpointerdevices

Comment 13

[Hiroyuki Ikezoe (:hiro)]

(In reply to Hiroyuki Ikezoe (:hiro) from comment #12)
> I will take the initial approach in comment 2, it's not perfect as
> :danibodea mentioned in comment 5 (it mis-recognizes as if there is a mouse
> device even if there is no mouse connected in the case where the touchscreen
> is disabled on Surface), but I think the situation rarely happens.

Filed bug 1495938 for the mis-recognition issue.

Comment 14

[Hiroyuki Ikezoe (:hiro)]

Attachment: Bug 1493128 - Consider a single mouse type device queried by SetupDiEnumDeviceInterfaces on tablet or on system with a digitizer as COARSE type pointer device. r=masayuki

Unfortunately there is no realiable way to tell whether there is no mouse
connected or not.  So we are going to take a heuristic way here.  The way is
that;

1) Count mouse type devices with SetupDiEnumDeviceInterfaces [1]
2) If the count is 1 and if the system is used as tablet or has a touch device,
   the device should be COARSE

This way causes a misrecognition as if there is a mouse device in the case
where the touchscreen is disabled on the tablet even if there is no mouse
connected (i.e. the case where CountMouseDevices() returns 1 and the system
is on table mode).  We will try to fix the misrecognition in a later bug
(bug 1495938).

[1] https://docs.microsoft.com/en-us/windows/desktop/api/setupapi/nf-setupapi-setupdienumdeviceinterfaces

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c50a3b343ac8
Consider a single mouse type device queried by SetupDiEnumDeviceInterfaces on tablet or on system with a digitizer as COARSE type pointer device. r=masayuki

Comment 16

[Cristian Brindusan [:cbrindusan]]

Backed out changeset c50a3b343ac8 (Bug 1493128) for build bustages on InputDeviceUtils.cpp.

Backout: https://hg.mozilla.org/integration/autoland/rev/ee830971f05015788d8087fb2beb2eb8ba8223bf

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=pending,running,success,testfailed,busted,exception&revision=c50a3b343ac847e4e112bbb8b4c2892167813b80&selectedJob=203313641

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=203313641&repo=autoland&lineNumber=26124

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bf098432003a
Consider a single mouse type device queried by SetupDiEnumDeviceInterfaces on tablet or on system with a digitizer as COARSE type pointer device. r=masayuki

Comment 18

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/bf098432003a

Comment 19

[Daniel Bodea [:danibodea]]

This issue is now verified in Nightly v64.0a1 (2018-10-07); the touchscreen capability of a touch-capable device is correctly recognized.

Furthermore, it appears that the other issue mentioned in comment 5 is also fixed. This means that the bug 1495938 mentioned in comment 13 should probably be marked as "WORKSFORME".

Thank you!

Comment 20

[Louis Semprini]

Hi Hiroyuki! This change broke the CSS Media "pointer" and "any-pointer" Query on Windows 10 laptops with a trackpad and without a mouse (they incorrectly return pointer: coarse instead of the pointer: fine that the spec requires for trackpads/touchpads as opposed to tablet touchscreens).

Please see the details at: https://bugzilla.mozilla.org/show_bug.cgi?id=1638556

Thanks!