Send async "mouse" events at the widget level instead of synthesized DOM events
Categories
(Remote Protocol :: Marionette, task, P2)
Tracking
(firefox147 fixed)
| Tracking | Status | |
|---|---|---|
| firefox147 | --- | fixed |
People
(Reporter: whimboo, Assigned: whimboo)
References
(Blocks 12 open bugs)
Details
(Whiteboard: [webdriver:m18], [wptsync upstream])
Attachments
(4 files)
See more details on bug 1773393. We need widget level events to send mouse events closer to what an actual user would do.
|
Assignee | |
Updated•2 years ago
|
|
|
Assignee | |
Comment 1•2 years ago
|
||
Please note that with bug 1852243 we have a temporary workaround available that will work fine for layout folks to work towards the interop 2023 goals. As such we do not have to work on this feature in M8 anymore.
I realized that InputData which is used by APZ instead of Widget*Event does not have a flag whether the input is caused by a native event or synthesized for tests. This is currently a problem for synthesized mouse events. PresShell::HandleEvent needs to ignore unexpected events coming from OS during tests. So, we need to fix this like this experimental patch. It does not have automated tests, so, if you want, feel free to reuse the patch.
|
|
Assignee | |
Comment 3•1 year ago
|
||
Thanks for the information Masayuki! I haven't started all the work yet but it's planned for next week. Nevertheless it will take a bit until I've everything in place to get started with this bug. So when you filed a bug for your fix please mark it blocking this bug and bug 1885073.
|
|
Assignee | |
Updated•1 year ago
|
No, I don't work on it because of not enough time for this. Feel free to use my patch.
|
|
Assignee | |
Updated•1 year ago
|
|
|
Assignee | |
Updated•1 year ago
|
|
|
Assignee | |
Updated•6 months ago
|
|
|
Assignee | |
Comment 5•6 months ago
|
||
While implementing this new feature I noticed that there is a subtle race between when the callback is called and when the event is actually emitted on the page in the content process. It's just some milliseconds off but this can cause tests like the following to fail:
def test_move_to_fractional_position2(session, inline, mouse_chain):
chain_temp = session.actions.sequence(
"pointer",
"pointer_id2",
{"pointerType": "mouse"})
chain_temp \
.pointer_move(100, 100) \
.perform()
session.url = inline("""
<script>
var allEvents = { events: [] };
window.addEventListener("pointermove", ev => {
dump(`\n*** ${Date.now()} event: ${ev.type}\n`);
allEvents.events.push({
"type": event.type,
"pageX": event.pageX,
"pageY": event.pageY,
});
});
</script>
""")
target_point = {
"x": 5.75,
"y": 10.25,
}
mouse_chain \
.pointer_move(target_point["x"], target_point["y"]) \
.perform()
events = get_events(session)
assert len(events) == 1
# For now we are allowing any of floor, ceil, or precise values, because
# it's unclear what the actual spec requirements really are
assert events[3]["type"] == "pointermove"
assert events[3]["pageX"] == pytest.approx(target_point["x"], abs=1.0)
assert events[3]["pageY"] == pytest.approx(target_point["y"], abs=1.0)
Here the mousemove from the mouse_chain doesn't produce mousemove event in time, so that no event is seen. When I move the chain_temp action after the page load it works all fine. Also when it is removed completely.
Here is the output:
0:07.47 pid:38892 1758121608042 RemoteAgent TRACE PointerMoveAction.performPointerMoveStep [5.75,95.25]
0:07.48 pid:38892 1758121608043 Marionette DEBUG 0 <- [1,9,null,{"value":null}]
0:07.48 pid:38892 1758121608043 webdriver::server DEBUG <- 200 OK {"value":null}
0:07.48 pid:38892 1758121608043 webdriver::server DEBUG -> POST /session/0fa3350a-2c86-4aae-85ef-372410357a76/execute/sync {"script": "return allEvents.events;", "args": []}
0:07.48 pid:38892 1758121608044 Marionette DEBUG 0 -> [0,10,"WebDriver:ExecuteScript",{"args":[],"script":"return allEvents.events;"}]
0:07.48 pid:38892 1758121608044 Marionette DEBUG 0 <- [1,10,null,{"value":[]}]
0:07.48 pid:38892 1758121608045 webdriver::server DEBUG <- 200 OK {"value":[]}
0:07.48 pid:38892 *** 1758121608046 event: pointermove
Once the patches are ready I'll upload to Phabricator to make it reproducible for platform folks for further investigation.
|
|
Assignee | |
Comment 6•6 months ago
|
||
|
|
Assignee | |
Comment 7•6 months ago
|
||
|
|
Assignee | |
Comment 8•6 months ago
|
||
|
|
Assignee | |
Comment 9•6 months ago
|
||
|
|
Assignee | |
Comment 10•6 months ago
|
||
I've attached the current version of my local patches to allow platform folks to investigate the delay of mouse events in the web page. Further I've had a chance to simplify the test and gain more details. The following Marionette test can help. Just run it with:
# coding=UTF-8
from urllib.parse import quote
from marionette_harness import (
MarionetteTestCase,
WindowManagerMixin,
)
BOILERPLATES = {
"html": "<!doctype html>\n<meta charset={charset}>\n{src}",
"xhtml": """<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<title>XHTML might be the future</title>
</head>
<body>
{src}
</body>
</html>""",
"xml": """<?xml version="1.0" encoding="{charset}"?>\n{src}""",
}
MIME_TYPES = {
"html": "text/html",
"xhtml": "application/xhtml+xml",
"xml": "text/xml",
}
def inline(doc, doctype="html", charset="UTF-8"):
content = BOILERPLATES[doctype].format(src=doc, charset=charset)
return f"data:{MIME_TYPES[doctype]},{quote(content)}"
class TestMinimizedTestCase(WindowManagerMixin, MarionetteTestCase):
def tearDown(self):
if (hasattr(self, "close_all_tabs")):
self.close_all_tabs()
super(TestMinimizedTestCase, self).tearDown()
def test_parent_process_events(self):
self.marionette.set_pref("remote.events.async.mouse.enabled", True)
new_tab = self.marionette.open("tab")
self.marionette.switch_to_window(new_tab["handle"], focus=True)
self.marionette.navigate(inline("""
<script>
var allEvents = { events: [] };
window.addEventListener("mousemove", ev => {
dump(`\n\n${Date.now()} *** event: ${ev.type}\n\n`);
allEvents.events.push({
"type": event.type,
"pageX": event.pageX,
"pageY": event.pageY,
});
});
</script>
"""))
mouse_chain = self.marionette.actions.sequence("pointer", "pointer_id")
mouse_chain.pointer_move(8, 8)
mouse_chain.perform()
events = self.marionette.execute_script("return allEvents.events", new_sandbox=False, sandbox=None)
assert len(events) == 1
mach marionette-test --gecko-log=- -vv _a/test_minimized.py --run-until-failure
Things I've noticed:
- The problem is only present when running the test in a new tab. When I change the test to open a new window the problem is gone.
- It's not a timing issue for when we synthesize the events given that it is also present when I add 5s of delay after opening the tab.
Edgar, could you please check why we have that delay?
|
|
Assignee | |
Comment 11•6 months ago
|
||
I'm going to add more wdspec tests once that bug is landed.
|
|
Assignee | |
Comment 12•6 months ago
|
||
Here a try build with the async Wdspec jobs, which shows a crash for MOZ_ASSERT(false) (MOZ_ASSERT_UNREACHABLE: We should always find a saved callback):
Edgar is taking a look into this and can also reproduce locally.
|
||
Comment 13•6 months ago
|
||
(In reply to Henrik Skupin [:whimboo][⌚️UTC+2] from comment #10)
Edgar, could you please check why we have that delay?
I filed bug 1989416.
|
|
Assignee | |
Comment 14•6 months ago
|
||
There is also a crash in the DragSessionProxy which I filed as bug 1989436. Given that this only happens for async jobs in CI it's not a hard-blocker and I should mark those tests as currently crashing.
|
||
Updated•6 months ago
|
|
|
Assignee | |
Comment 15•5 months ago
|
||
With bug 1989416 fixed I pushed a new try build:
https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=92a07cac3c23783e06198224718297ef0c0b6c2e
|
|
Assignee | |
Comment 16•4 months ago
•
|
||
It looks like that I was able to figure out the remaining issues and so far the most recent try build looks fine:
https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=ef54f4e61ef7344b45046423d1e6cadb62a4f0cb
If it stays that way I'll upload an updated series of patches tomorrow.
|
||
Updated•4 months ago
|
|
|
||
Updated•4 months ago
|
|
|
||
Updated•4 months ago
|
|
|
||
Updated•4 months ago
|
|
|
||
Updated•4 months ago
|
|
||
Comment 17•4 months ago
|
||
|
|
||
Comment 19•4 months ago
|
||
|
||
Comment 20•4 months ago
|
||
Backed out for causing failures at pointer_dblclick.py.
Backout link: https://hg-edge.mozilla.org/integration/autoland/rev/9e063737eb82dbbba534d2f041567f427f02b5bd
|
|
||
Comment 21•4 months ago
|
||
|
||
Comment 23•4 months ago
|
||
| bugherder | ||
https://hg.mozilla.org/mozilla-central/rev/41dbafc765a4
https://hg.mozilla.org/mozilla-central/rev/852c5c1290bb
https://hg.mozilla.org/mozilla-central/rev/9cc5104f1ac8
https://hg.mozilla.org/mozilla-central/rev/0ffad14f5d03
|
|
||
Updated•4 months ago
|
|
|
||
Comment 24•4 months ago
|
||
|
|
Assignee | |
Comment 25•4 months ago
|
||
There is an issue with the meta data for pointer_dblclick.py tests, which I've updated before landing but which assumes the given tests to fail everywhere even for test jobs without async events. I need to fix that.
But there is as well an issue on Android when dispatching the event. There is the following JavaScript error thrown due to different timing causing a silent failure and the command to hang:
11-12 10:18:46.240 5829 5851 E GeckoConsole: [JavaScript Error: "ReferenceError: can't access lexical declaration 'preventDefault' before initialization" {file: "chrome://remote/content/shared/webdriver/Event.sys.mjs" line: 93}]
The linter doesn't show this as error or even not as warning. I'm going to fix that.
|
|
Assignee | |
Comment 26•4 months ago
|
||
A new try build is at: https://treeherder.mozilla.org/jobs?repo=try&landoCommitID=163355
|
|
||
Comment 27•4 months ago
|
||
|
||
Comment 29•4 months ago
|
||
| bugherder | ||
https://hg.mozilla.org/mozilla-central/rev/e26b4b7e0032
https://hg.mozilla.org/mozilla-central/rev/cdfa56af3094
https://hg.mozilla.org/mozilla-central/rev/68b77d556f52
https://hg.mozilla.org/mozilla-central/rev/203a0d2dad2a
Description
•