Browser recognizes touch-screen capability incorrectly

VERIFIED FIXED in Firefox 64

Status

()

defect
P2
normal
VERIFIED FIXED
8 months ago
7 months ago

People

(Reporter: danibodea, Assigned: hiro)

Tracking

(Depends on 1 bug)

Trunk
mozilla64
Other
Windows
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox64 verified)

Details

Attachments

(1 attachment)

Reporter

Description

8 months ago
[Affected versions]: 
- v64.0a1

[Affected platforms]:
- Windows

[Steps to reproduce]:
1. Open browser on a touch-screen capable device.
2. Open page: https://hiikezoe.github.io/interaction-media-features.html
3. Disconnect any other pointing devices (like mouse/touch-pad) so that the touch-screen is the only available pointing device.
4. Notice that the page's content shows that the browser detects the touch-screen incorrectly.

[Expected result]:
pointer: coarse
any-pointer: coarse
hover: none
any-hover: none

[Actual result]:
pointer: coarse
any-pointer: fine
any-pointer: coarse
hover: none
any-hover: hover

[Additional notes]:
- This issue does not occur in the case of the Chrome browser.
Reporter

Updated

8 months ago
Blocks: 1035774
Assignee: nobody → hikezoe
Priority: -- → P2
Assignee

Comment 1

8 months ago
From a comment [1] for the function used for any-pointer in Chrome.

 // - If the device is a detachable/convertible win8/10 device and the keyboard/ 
 // trackpad is detached/flipped it will always return POINTER_TYPE_COARSE. 

So, I am pretty sure that 'any-pointer:fine' doesn't appear on Chrome if the device is used as tablet mode even if there is a mouse connected.

[1] https://chromium.googlesource.com/chromium/src.git/+/d157b3968431272cf98ef6b030886c938bcd1047/ui/base/touch/touch_device_win.cc#26
Assignee

Comment 2

8 months ago
Brian told me last night on his Surface there is a device appearing in the device manager on Surface with slate mode.  The device name is 'Intel Precise Touch Device', I believe the device is the touchscreen of the Surface and GetSystemMetrics(SM_MOUSEPRESENT) that we are using to tell whether there is no mouse (fine device) or not queries the touchscreen device.

So we have to figure out another way to tell whether there is no mouse or not.  After some googling, what I can think of now is that 

1) Count mouse type devices with SetupDiEnumDeviceInterfaces [1]
2) If the count is 1 and if IsTabletDevice() or IsTouchDeviceSupportPresent() returns true, the device should be COARSE

[1] https://msdn.microsoft.com/ja-jp/library/cc429195.aspx
Assignee

Comment 3

8 months ago
(In reply to Hiroyuki Ikezoe (:hiro) (PTO on Sep 26) from comment #2)

> [1] https://msdn.microsoft.com/ja-jp/library/cc429195.aspx

Err, wrong link.  Here is the correct link.

https://msdn.microsoft.com/ja-jp/library/cc429192.aspx
Assignee

Updated

8 months ago
Status: NEW → ASSIGNED
Assignee

Comment 4

8 months ago
:danibodea, can you please try this binary to see if it works as expected?

Thanks!

https://treeherder.mozilla.org/#/jobs?repo=try&revision=70af8a1754efb1a617711c8d3782c2034f83c9d7
Flags: needinfo?(daniel.bodea)
Reporter

Comment 5

8 months ago
:hiro, I have tried a build from this revision, more exactly, Windows 2012 x64 opt, Bs, artifact uploaded: target.zip
https://queue.taskcluster.net/v1/task/GuezljoUSJe9RZFqJJn2jw/runs/0/artifacts/public/build/target.zip

It appears that the issue in question was fixed, but there is another situation:
When the mouse remains the last pointing device on the system and then it gets removed, the test page does not show "none" to all, in fact, the test page's content does not change at all, as if it does not detect the removal of the mouse. 
(Probably relevant: The Chrome browser does not detect the connection of the mouse at all, let alone the disconnection.)

All the other situations appear to be working as intended IF the following situation is expected as correct:
When touch-screen and mouse are connected/enabled the following results are seen:
pointer: coarse
any-pointer: fine
any-pointer: coarse
hover: none
any-hover: hover

Thanks.
Flags: needinfo?(daniel.bodea)
Assignee

Comment 6

8 months ago
Thanks!

(In reply to Bodea Daniel [:danibodea] from comment #5)

> It appears that the issue in question was fixed, but there is another
> situation:
> When the mouse remains the last pointing device on the system and then it
> gets removed, the test page does not show "none" to all, in fact, the test
> page's content does not change at all, as if it does not detect the removal
> of the mouse. 

It's the case where the touchscreen is mis-recognized as 'mouse'.  I believe this issue happens in the first place.

I think I found another approach to use MouseCapabilities [1] in UWP.
:danibodea, could you please this try?  If it didn't work, there is no other way I can think of.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=99b1f311381094318df63ca725543da460da2019

> (Probably relevant: The Chrome browser does not detect the connection of the
> mouse at all, let alone the disconnection.)
> 
> All the other situations appear to be working as intended IF the following
> situation is expected as correct:
> When touch-screen and mouse are connected/enabled the following results are
> seen:
> pointer: coarse
> any-pointer: fine
> any-pointer: coarse
> hover: none
> any-hover: hover

Yes, this is the result what I am expecting.

[1] https://docs.microsoft.com/en-us/uwp/api/windows.devices.input.mousecapabilities
Flags: needinfo?(daniel.bodea)
Assignee

Comment 7

8 months ago
CCing masayuki, I will probably ask him to review.
Reporter

Comment 8

8 months ago
Hello Hiro!

This new build did not fix the previous issue where the only pointing device left connected is a mouse, and the parameters don't change to "none" after its removal. I have to mention again that this is also how Chrome behaves.

Furthermore, this new build introduces a new issue: when touch-screen is activated and a touch-pad connected, then, if the touch-pad gets disconnected, the results are incorrect, basically re-introducing the original issue of this bug, as follows:

Actual: 
pointer: coarse
any-pointer: fine
any-pointer: coarse
hover: none
any-hover: hover

Expected: 
pointer: coarse
any-pointer: coarse
hover: none
any-hover: none

My opinion is that the behavior of the previous build (the one from comment 4) is the better one (if there is no other way). It seems that the Chrome developers used the same or a similar method since our only remaining issue is also occurring on their browser. Furthermore, Chrome has other bad behaviors besides this one. A full comparison report would take some time, but I can do it if needed. 

Thanks!
Flags: needinfo?(daniel.bodea) → needinfo?(hikezoe)
Assignee

Comment 9

8 months ago
Thanks!  That's a bad news though. :/
So, the MouseCapabilities seems to contain the touchscreen on Surface for some reasons. :/

Ok, here is another attempt that output pointer device types to stdout.
:danibodea, can you please try this binary?  You have to run firefox.exe in a shell something like this (you probably need to install MozillaBuild to run a shell);

 firefox.exe 2>&1 | tee

When open https://hiikezoe.github.io/interaction-media-features.html, you will see some log like this;

 Found 1 devices: Mouse

https://treeherder.mozilla.org/#/jobs?repo=try&revision=c7a4a6a0540fe96d958df56a90d96680cbc4e49d

In a sample code on MSDN, PointerDeviceType looks like independent value, but I suspect it's bitwise enum something like Touch|Mouse. 

[1] https://code.msdn.microsoft.com/windowsapps/Input-device-capabilities-31b67745/sourcecode?fileId=43661&pathId=786035579
Flags: needinfo?(hikezoe) → needinfo?(daniel.bodea)
Reporter

Comment 10

8 months ago
Hi,

I have tried what you asked and I might be missing some info on the process because these were my steps and results:

1. I have installed MozillaBuild on Surface.
2. I have downloaded the try build from the treeherder link provided.
3. Opened Windows PowerShell and navigated to the build's location (in PowerShell).
4. I have used the command ".\firefox.exe 2>&1 | tee output.txt".
5. The browser opened as intended, I opened the test page (https://hiikezoe.github.io/interaction-media-features.html).
6. Using Windows Explorer, went to the location of the browser build (where the shell created the output.txt file).
7. Opened output.txt and saw no string similar to "Found 1 devices: Mouse". 

This is what the output.txt file shown:
"
.\firefox.exe : 
At line:1 char:1
+ .\firefox.exe 2>&1 | tee output.txt
+ ~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
###!!! [Child][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost


[GPU 7844, Chrome_ChildThread] WARNING: pi
pe 
err
or: 109: f
ile
 
z:
/
build
/bu
ild
/
src/
i
p
c/c
h
romi
u
m/sr
c/c
h
rome
/common
/i
pc
_
chann
el_
w
in.cc
,
 l
i
ne 
346



###!!! [Child][MessageChannel::SendAndWait] Error: Channel error: cannot send/recv
"

Am I missing something? I couldn't check the last binary, given the fact that I could not see the connected pointing devices for a more informed report, as you would like me to. I am guessing that the last error is the culprit. What do you suggest?
Flags: needinfo?(daniel.bodea) → needinfo?(hikezoe)
Assignee

Comment 11

8 months ago
Hmm the tee in Windows' PowerShell is somewhat different from what I know of... 

You must run the command in the shell spawned by start-shell in the MozillaBuild.
Flags: needinfo?(hikezoe)
Assignee

Comment 12

8 months ago
Brian tested the build on his Surface. And the output log on slate mode is something like this;

  Found 4 devices: Pen Touch Touch Mouse

This means that GetPointerDevices[1] used in the build also returns a mouse even if no mouse is connected to Surface.
So, I think there is no way to tell whether there is no mouse or not at this moment.

I will take the initial approach in comment 2, it's not perfect as :danibodea mentioned in comment 5 (it mis-recognizes as if there is a mouse device even if there is no mouse connected in the case where the touchscreen is disabled on Surface), but I think the situation rarely happens.

[1] https://docs.microsoft.com/en-us/uwp/api/windows.devices.input.pointerdevice.getpointerdevices
Assignee

Comment 13

8 months ago
(In reply to Hiroyuki Ikezoe (:hiro) from comment #12)
> I will take the initial approach in comment 2, it's not perfect as
> :danibodea mentioned in comment 5 (it mis-recognizes as if there is a mouse
> device even if there is no mouse connected in the case where the touchscreen
> is disabled on Surface), but I think the situation rarely happens.

Filed bug 1495938 for the mis-recognition issue.
Assignee

Comment 14

8 months ago
Unfortunately there is no realiable way to tell whether there is no mouse
connected or not.  So we are going to take a heuristic way here.  The way is
that;

1) Count mouse type devices with SetupDiEnumDeviceInterfaces [1]
2) If the count is 1 and if the system is used as tablet or has a touch device,
   the device should be COARSE

This way causes a misrecognition as if there is a mouse device in the case
where the touchscreen is disabled on the tablet even if there is no mouse
connected (i.e. the case where CountMouseDevices() returns 1 and the system
is on table mode).  We will try to fix the misrecognition in a later bug
(bug 1495938).

[1] https://docs.microsoft.com/en-us/windows/desktop/api/setupapi/nf-setupapi-setupdienumdeviceinterfaces

Comment 15

8 months ago
Pushed by hikezoe@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/c50a3b343ac8
Consider a single mouse type device queried by SetupDiEnumDeviceInterfaces on tablet or on system with a digitizer as COARSE type pointer device. r=masayuki

Comment 17

8 months ago
Pushed by hikezoe@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/bf098432003a
Consider a single mouse type device queried by SetupDiEnumDeviceInterfaces on tablet or on system with a digitizer as COARSE type pointer device. r=masayuki
Assignee

Updated

8 months ago
Flags: needinfo?(hikezoe)

Comment 18

8 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/bf098432003a
Status: ASSIGNED → RESOLVED
Last Resolved: 8 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla64
Reporter

Comment 19

8 months ago
This issue is now verified in Nightly v64.0a1 (2018-10-07); the touchscreen capability of a touch-capable device is correctly recognized.

Furthermore, it appears that the other issue mentioned in comment 5 is also fixed. This means that the bug 1495938 mentioned in comment 13 should probably be marked as "WORKSFORME".

Thank you!
Status: RESOLVED → VERIFIED
Assignee

Updated

7 months ago
Depends on: 1498286
Reporter

Updated

7 months ago
No longer depends on: 1498286
Reporter

Updated

7 months ago
Depends on: 1498286
You need to log in before you can comment on or make changes to this bug.