Implement Interaction Media Features including pointer:coarse that replaces non-standard -moz-touch-enabled

NEW
Unassigned

Status

()

Core
Layout
--
enhancement
4 years ago
20 days ago

People

(Reporter: sebo, Unassigned)

Tracking

(Blocks: 3 bugs, 4 keywords)

Trunk
dev-doc-needed, parity-chrome, parity-edge, parity-safari
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [fingerprinting][fp:m4][fp-triaged], URL)

Attachments

(1 attachment)

(Reporter)

Description

4 years ago
@media should have support for interaction features as described at http://dev.w3.org/csswg/mediaqueries4/#mf-interaction.

According to http://status.modern.ie/mediaquerieslevel4interactionmediafeaturespointerandhover these are already supported by Chrome and Opera.
Those features now have been implemented in all other browsers.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Whiteboard: [parity-edge][parity-blink][parity-webkit]
Duplicate of this bug: 1171900
FWIW, Blink's implementation can be find here: https://chromium.googlesource.com/chromium/src.git/+/master/ui/base/touch/ in the touch_device_*.cc files.

Comment 4

2 years ago
notechnicalvalue
Given how much crippled, bloated, "My first UI" design infuriates me when viewing formerly useful sites like PayPal and My eBay where I'm trying to get work done, I'll be using these to support mobile devices.

As much as I love Firefox (I use it exclusively on the desktop), everyone else implements these and my need for a practical, usable desktop experience trumps brand loyalty.

Until Firefox Mobile supports these, I'll be responding to any reports of touch-unfriendly Firefox Mobile rendering on my sites with RESO WONTFIX.
(Reporter)

Updated

2 years ago
Keywords: dev-doc-needed

Comment 6

2 years ago
This is annoying from a web development perspective because it seriously limits the ability to use :hover effectively across platforms. On most mobile browsers, :hover makes no sense, and is actually the exact opposite behavior that you'd want in many cases.

There are a lot of weird hacks to avoid the :hover state from being activated forever after touching that item on the screen, but they mostly involve not using hover unless the hover media query is active. Since Firefox does not respond to these, there ends up being no :hover functionality available for Firefox desktop users.
Comment hidden (advocacy)

Comment 8

2 years ago
In case this does get worked on at some point, note that the CSS4 MQ spec changed recently to remove the "on-demand" value for "hover/any-hover".

See related bug on Chromium/Blink https://bugs.chromium.org/p/chromium/issues/detail?id=654861 and Edge https://developer.microsoft.com/en-us/microsoft-edge/platform/issues/9308676/

As for the comments here about the usefulness (or lack thereof) of the spec itself, see for instance https://dev.opera.com/articles/media-features/ (tl;dr: it has the potential to be misused, unless developers use it defensively)
Just FYI: Firefox supports the non-standard -moz-touch-enabled media feature. I'm using it in my app.

https://developer.mozilla.org/en-US/docs/Web/CSS/Media_Queries/Using_media_queries#-moz-touch-enabled

So, not yet tested but this should work:

> @media (-moz-touch-enabled: 1), (pointer: coarse) {
>   // Rules for touch screens
> }

Comment 10

11 months ago
(In reply to Kohei Yoshino [:kohei] from comment #9)
> Just FYI: Firefox supports the non-standard -moz-touch-enabled media
> feature. I'm using it in my app.
> 
> https://developer.mozilla.org/en-US/docs/Web/CSS/Media_Queries/
> Using_media_queries#-moz-touch-enabled
> 
> So, not yet tested but this should work:
> 
> > @media (-moz-touch-enabled: 1), (pointer: coarse) {
> >   // Rules for touch screens
> > }

Thank you!
Note: Gmail's touch enabled display density requires this.

Updated

9 months ago
Blocks: 541386

Updated

9 months ago
Summary: Implement Interaction Media Features → Implement Interaction Media Features including pointer:coarse that replaces non-standard -moz-touch-enabled

Comment 12

9 months ago
Created attachment 8904713 [details] [diff] [review]
1035774-add_pointer_hover_any-pointer,_and_any-hover_media_queries.diff

Here's a patch which implements the pointer, hover, any-pointer, and any-hover features, based on what Blink is doing for Window, Linux, Android, and OSX (save for some improvements and avoiding a bug where any-hover:none triggers on Linux when the spec says it should not).

WebKit has a pretty rudimentary implementation, and neither it nor Blink have any fancy logic for determining what the "primary" device is, so I just did what they do in this code, reasoning that it will be sufficient for interop/webcompat.

Note that while I've tested the patch on all the devices I have (OSX, Linux, Windows and Android), I don't have a touchscreen monitor, Android-compatible mouse, a Windows tablet/phone, or other pen/stylus type input devices to test with, so this could use some broader QA.

Worse, I have no clue how I would add automated tests for this (I see no real test-coverage for similar features like MaxTouchPoints either). Kohei, Felip, you wouldn't happen to know who I should ping for info on that, would you?
Flags: needinfo?(kohei.yoshino)
Flags: needinfo?(felipc)
Sorry I'm not sure.
Flags: needinfo?(kohei.yoshino)

Comment 14

9 months ago
Jim, do you know who might be able to help verify what should be done re: testing for this patch?
Flags: needinfo?(jmathies)
There's a new-ish set of tests called "web-platform-tests" that are meant to be interop tests that can be run on any browser. It would be nice if you can figure out how to write and use this type of test here, although I don't know how dynamic is that test suite to know if you can force the touch mode for some tests..

You should send an Intent to Implement e-mail to mozilla.dev.platform talking about your goal of implementing this feature, and there you'll probably be able to get more useful suggestions on how to approach testing for this.
Flags: needinfo?(felipc)

Comment 16

9 months ago
Thanks Felipe, I'll definitely send an intent... but I don't see any web platform tests for these features (nor do I see how they could test these abilities with decent coverage).
No way to automate such tests in wpt currently.

Comment 18

8 months ago
(In reply to Thomas Wisniewski from comment #14)
> Jim, do you know who might be able to help verify what should be done re:
> testing for this patch?

We have apis in nsIDOMWindowUtils [1] that simulate touch events for testing, those call down into widget and trigger actual input events that tests then use. If need be you could add new api if we need new event simulations (assuming platforms provide a way to fire them).

[1] http://searchfox.org/mozilla-central/source/dom/interfaces/base/nsIDOMWindowUtils.idl#348
Flags: needinfo?(jmathies)

Comment 19

8 months ago
Note that testing here wouldn't involve simulating touch events (or any events), but simulating the actual "environment" (however the browser determines, by interfacing with the OS, that a touchscreen is present, or whether or not any mouse/stylus is present, etc)

Comment 20

8 months ago
(In reply to Patrick H. Lauke from comment #19)
> Note that testing here wouldn't involve simulating touch events (or any
> events), but simulating the actual "environment" (however the browser
> determines, by interfacing with the OS, that a touchscreen is present, or
> whether or not any mouse/stylus is present, etc)


Yeah, therein lies the rub. I'm guessing the best route to go would be to go through my code (and its dependencies), and replace each OS-specific call with a call to a wrapper which decides (during runtime) whether to make that OS call or to mock it (so each test-case can mock a different device config). I'm just not sure how to implement that.

My guess would be, roughly:
- add a new method nsIWidget::MockSystemPointers, called by a similar new method nsIDOMWindowUtils::MockSystemPointers, which would set a static/class variable nsWidget::SystemPointerProfile denoting which configuration to mock (or to not mock at all).
- have my code in this patch (and its dependencies) not directly call OS-specific hooks, but rather call wrapper methods (MockableSystemCallsGTK::gdk_display_get_default, MockableSystemCallsWindows::GetSystemMetrics, etc).
- have those methods read nsWidget::SystemPointerProfile, and if a mock was set, not call the OS-specific routines, but rather just return mocked values.

But I'm probably missing something there, plus I'm not sure if there are better/OS-specific ways we should mock these things instead.

Comment 21

8 months ago
Jim, do you think my approach in comment 20 would be fine?
Flags: needinfo?(jmathies)
Comment hidden (advocacy)

Comment 23

8 months ago
The only relevant testing we do of -moz-touch-enabled is in test_media_queries.html[1] to ensure it is parseable. (It's included in a torbug fingerprinting case as well, but I'm not sure that's relevant here)

Adding appropriate cases there seems like it could satisfy the testing requirement.

[1]: http://searchfox.org/mozilla-central/source/layout/style/test/test_media_queries.html

Comment 24

8 months ago
Thanks, chutten. I've just sent an intent to implement with an explanation that the only tests I plan to add are CSS parsing ones. I'll refresh my patch with those tests as soon as I can.

Comment 25

7 months ago
Tagging this for fingerprinting. As mentioned in the email, we'd like this to check nsContentUtils::ShouldResistFingerprinting() and if that returns true, behave like you described:

> Bear in mind that I'm operating on the presumption that we would want anti-fingerprinting to always return values for Windows/OSX/Linux that convey there is only a fine, hover-capable pointer (a mouse), and on Android to convey that there is only a coarse, non-hover-capable pointer (a touchscreen).
Whiteboard: [parity-edge][parity-blink][parity-webkit] → [parity-edge][parity-blink][parity-webkit][fingerprinting][fp:m4]

Comment 26

7 months ago
(In reply to Thomas Wisniewski from comment #20)
> (In reply to Patrick H. Lauke from comment #19)
> > Note that testing here wouldn't involve simulating touch events (or any
> > events), but simulating the actual "environment" (however the browser
> > determines, by interfacing with the OS, that a touchscreen is present, or
> > whether or not any mouse/stylus is present, etc)
> 
> 
> Yeah, therein lies the rub. I'm guessing the best route to go would be to go
> through my code (and its dependencies), and replace each OS-specific call
> with a call to a wrapper which decides (during runtime) whether to make that
> OS call or to mock it (so each test-case can mock a different device
> config). I'm just not sure how to implement that.
> 
> My guess would be, roughly:
> - add a new method nsIWidget::MockSystemPointers, called by a similar new
> method nsIDOMWindowUtils::MockSystemPointers, which would set a static/class
> variable nsWidget::SystemPointerProfile denoting which configuration to mock
> (or to not mock at all).
> - have my code in this patch (and its dependencies) not directly call
> OS-specific hooks, but rather call wrapper methods
> (MockableSystemCallsGTK::gdk_display_get_default,
> MockableSystemCallsWindows::GetSystemMetrics, etc).
> - have those methods read nsWidget::SystemPointerProfile, and if a mock was
> set, not call the OS-specific routines, but rather just return mocked values.
> 
> But I'm probably missing something there, plus I'm not sure if there are
> better/OS-specific ways we should mock these things instead.

Something along these lines I guess. What do we need to simulate? Looks like 

1) touch screen css detection
2) some sort of pointer type (none, coarse, fine)

Note the touch info comes from look and feel info vs. widget. You might chat with someone who know dom better, there may be a way to inject this higher up in the stack.
Flags: needinfo?(jmathies)

Updated

6 months ago
Whiteboard: [parity-edge][parity-blink][parity-webkit][fingerprinting][fp:m4] → [parity-edge][parity-blink][parity-webkit][fingerprinting][fp:m4][fp-triaged]
Duplicate of this bug: 1439209

Updated

3 months ago
Duplicate of this bug: 1443047
Hi Thomas, do you plan to finish this out? Any way I can help? Thanks!
Flags: needinfo?(wisniewskit)

Comment 30

3 months ago
I was hoping to finish it sometime by the end of Q2. But if you have time before that then, the patch needs tests and a special-case for resist-fingerprinting mode where it only mimics a single standard mouse pointer (as per comment 25). If you'd like to finish it sooner, please feel free! I can at least try to rebase it soon, if that would help.
Flags: needinfo?(wisniewskit)
Oh, that sounds awesome actually. One thing I have noticed is that we don't react to dynamic changes to it.

I don't know how easy it is, I guess it's another bunch of platform dependent code, and detecting moves across screens. Or maybe it's not too bad, looking at nsDeviceContext::CheckDPIChange and friends.

Comment 32

3 months ago
Are other browsers dynamic in that way? (If they aren't, then we can potentially defer making it dynamic to a follow-up patch).

Comment 33

3 months ago
I've not had a chance to test, but (based on this bug https://bugs.chromium.org/p/chromium/issues/detail?id=442418) it should be dynamic for Chrome (at least on Android).
Yeah... In any case I agree it's probably worth doing it as a followup, if only to keep this patch under control in terms of complexity.
Mass bug change to replace various 'parity' whiteboard flags with the new canonical keywords. (See bug 1443764 comment 13.)
Keywords: parity-chrome, parity-edge, parity-safari
Whiteboard: [parity-edge][parity-blink][parity-webkit][fingerprinting][fp:m4][fp-triaged] → [fingerprinting][fp:m4][fp-triaged]
You need to log in before you can comment on or make changes to this bug.