1805146 - Firefox 103 changed behavior for "WebDriver:PerformActions" when using a not visible element (eg. "<option>") as origin

Status: NEW

(Remote Protocol :: Agent, defect, P3)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08)]

Filed based on the comment from bug 1773264 comment 8. It's a regression that started in Firefox 103 and needs to be taken care of.

(In reply to Andrew Nicols from bug 1773264 comment #8)

FYI this change introduced some new error messages and was not noted in the 103 release notes (https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Releases/103).

Specifically the amount of error checking around Origin co-ordinates and the fromJSON function has significantly increased and whilst these error conditions are largely correct, some of them were working previously and are arguably not defined as errors in the specification.

In our case we were using a click action to select an Option element in a Select element, which lead to an "Origin element is not displayed" error. This behaviour was previously working, and still works in other browsers (Chrome).

There is a comment stating that this behaviour is currently not defined in the spec (https://github.com/w3c/webdriver/issues/1642 was opened to discuss it), so I'm surprised that the decision was made to turn it into an Error in Firefox.

In our case it's an easy fix, but I wonder if there are other possible cases where the origin is legitimately not visible.

And James replied:

(In reply to James Graham [:jgraham] from bug 1773264 comment #10)

I think from a spec lawyering point of view "these are arguably not defined as errors in the specification" is missing the point: the specification just fails to consider that there might be an element without bounding client rects, so everything in that case is undefined behaviour.

In general the idea that you can have a coordinate origin that doesn't correspond to a visible point on the screen doesn't seem reasonable.

Presumably in the case of <option> you want something like "the position it would be displayed, if it were being displayed". But even that doesn't quite work: if e.g. the select has a scrollable UI then there are multiple possible postions the element could be in.

So I guess the question is: what's the expected behaviour here? Is it expected that the user does some work to ensure that the element is actually in view, or is the idea that we have some way to dispatch events to specific option elements without actually dealing with the browser-specific select UI? What exactly is Chrome doing in these cases?

I agree this is a regression and we should prioritize a fix, but I don't think we should do so without a clear understanding of what we're trying to achieve.

If you happen to have a testcase we can use to investigate this behaviour, that would be helpful.

I had a look how a minimized test case could look like and here one for Marionette directly with a simple select/option scenario.

    def test_element(self):
        self.marionette.navigate(inline("""
            <select>
                <option value="foo">Foo</option>
                <option value="bar">Bar</option>
            </select>
        """))
        options = self.marionette.find_elements(By.TAG_NAME, "option")
        ActionSequence(self.marionette, "pointer", "12") \
            .click(options[1]) \
            .perform()

Until Firefox 102 this test returned the following failure:

ERROR _a/test_minimized.py TestMinimizedTestCase.test_element - marionette_driver.errors.InvalidArgumentException: Expected obj to have pointerType, got [object Object] {"type":"pointer","id":"12","actions":[{"type":"pointerMove","x":0,"y":0,"origin":{}},{"type":"pointerDown","button":0},{"type":"pointerUp","button":0}]}
stacktrace:
	WebDriverError@chrome://remote/content/shared/webdriver/Errors.jsm:186:5
	InvalidArgumentError@chrome://remote/content/shared/webdriver/Errors.jsm:315:5
	fromJSON@chrome://remote/content/marionette/action.js:195:15
	fromJSON@chrome://remote/content/marionette/action.js:566:39
	fromJSON@chrome://remote/content/marionette/action.js:529:48
	performActions@chrome://remote/content/marionette/actors/MarionetteCommandsChild.jsm:460:20
	receiveMessage@chrome://remote/content/marionette/actors/MarionetteCommandsChild.jsm:144:31

Since Firefox 103 the failure as thrown is:

ERROR _a/test_minimized.py TestMinimizedTestCase.test_element - marionette_driver.errors.MoveTargetOutOfBoundsException: Origin element is not displayed
stacktrace:
	RemoteError@chrome://remote/content/shared/RemoteError.sys.mjs:8:8
	WebDriverError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:180:5
	MoveTargetOutOfBoundsError@chrome://remote/content/shared/webdriver/Errors.sys.mjs:370:5
	getOriginCoordinates@chrome://remote/content/marionette/action.sys.mjs:486:15
	getTargetCoordinates@chrome://remote/content/marionette/action.sys.mjs:430:25
	dispatch@chrome://remote/content/marionette/action.sys.mjs:1004:32
	dispatch/pendingEvents<@chrome://remote/content/marionette/action.sys.mjs:1827:14
	dispatch@chrome://remote/content/marionette/action.sys.mjs:1826:39
	dispatch/chainEvents<@chrome://remote/content/marionette/action.sys.mjs:1753:27
	dispatch@chrome://remote/content/marionette/action.sys.mjs:1755:7
	performActions@chrome://remote/content/marionette/actors/MarionetteCommandsChild.sys.mjs:472:23
	receiveMessage@chrome://remote/content/marionette/actors/MarionetteCommandsChild.sys.mjs:139:31

James, can you please take a look?

Comment 1

[Julian Descottes [:jdescottes]]

For the Firefox 102 error, is it a relevant error or is it just because we had incomplete support for "pointer" in that version?

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08)]

We support pointer since forever. It's really that we return a different error now.

Comment 3

[James Graham [:jgraham]]

It looks to me like the actions aren't well formed in the Firefox 102 version, and that's where the error comes from. My understanding of the original report was that we ended up dispatching a click to the option in that case.

That said, I still don't really know how to fix this. It would require a bunch of tests to work out where the pointer should end up, what happens if the popup requires scrolling, what happens on mobile, etc.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08)]

Andrew, as we discussed yesterday in our meeting it is still unclear to us in which scenario you actually noticed this problem. And the testcase in comment 0 might not reflect that properly. Could you maybe help by telling us the exact scenario? Thanks a lot.

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1773264

Comment 6

[Andrew Nicols]

Hi folks,
Sorry for the delay - a few personal things have popped up and I've not had a chance to comment until now.

From memory the previous behaviour allowed us to open the select, and then click on the option. This is the behaviour which changed.

I believe something like this should show the change in behaviour, but I need to work out how to build an older version of Marionette to confirm this:

diff --git a/testing/marionette/harness/marionette_harness/tests/unit/test_click.py b/testing/marionette/harness/marionette_harness/tests/unit/test_click.py
--- a/testing/marionette/harness/marionette_harness/tests/unit/test_click.py
+++ b/testing/marionette/harness/marionette_harness/tests/unit/test_click.py
@@ -279,6 +279,20 @@ class TestClick(ClickBaseTestCase):
         with self.assertRaises(errors.ElementClickInterceptedException):
             obscured.click()

+    def test_click_on_opened_select_option(self):
+        self.marionette.navigate(inline("""
+            <select>
+                <option value="foo">Foo</option>
+                <option value="bar">Bar</option>
+            </select>
+        """))
+
+        self.marionette.find_element(By.TAG_NAME, "select").click()
+        options = self.marionette.find_elements(By.TAG_NAME, "option")
+        ActionSequence(self.marionette, "pointer", "12") \
+            .click(options[1]) \
+            .perform()
+
     def test_centre_outside_viewport_vertically(self):
         self.marionette.navigate(
             inline(

Comment 7

[Julian Descottes [:jdescottes]]

I tried to investigate this, but I might be missing a proper test case, because I could not get any browser to handle option clicks successfully.

I ported the suggested test from comment 6 to a wdspec test:

def test_click_select(session, url, inline, mouse_chain):
    session.url = inline("""
            <select>
                <option value="foo">Foo</option>
                <option value="bar">Bar</option>
            </select>
        """)

    select = session.find.css("select", all=False)
    mouse_chain.click(element=select).perform()

    option = session.find.css("option[value='bar']", all=False)
    mouse_chain.click(element=option).perform()

    value = session.execute_script("return document.querySelector('select').value;")
    assert value == "bar"

(added to https://github.com/web-platform-tests/wpt/blob/master/webdriver/tests/perform_actions/pointer_mouse.py)

None of the browsers I tried succeed for this test:

• Firefox 114 fails with webdriver.error.MoveTargetOutOfBoundsException: move target out of bounds (500): Origin element is not displayed
• Firefox ESR 102 fails with webdriver.error.UnknownErrorException: unknown error (500): TypeError: rect is undefined
• Firefox ESR 91 manages to complete the test but fails on the assert, meaning that the click did not target the correct option
• Chrome fails with webdriver.error.WebDriverException: element not interactable (400): element not interactable: [object HTMLOptionElement] has no size and location
• Safari is the same as Firefox ESR 91, completes the test but fails the assert

So with this test case, it's very unclear what should be changed. Updating the spec to clearly handle this use case would be great, but I can't see a clear trend where other vendors or older Firefox versions had a useful behavior here.

Andrew: If there is a better test case to consider where there is a clear regression and we can see this works in other browsers or in older Firefox versions, could you share it?

If not, then I think this mostly needs spec work at this point. Failing early seems better than the previous behavior which was to silently fail to click on the option.

Comment 8

[Julian Descottes [:jdescottes]]

We should file a spec issue

Comment 9

[Julian Descottes [:jdescottes]]

Actually the issue at https://github.com/w3c/webdriver/issues/1642 already covers this, so I just reported my findings there.

Andrew, can you take a look at comment 7? As you can see, I tried to implement a test case similar to the suggestion here, but I didn't find any browser able to successfully click on the option.

Do you have a another test case which is/was working with any browser?

Comment 10

[Julian Descottes [:jdescottes]]

Will check if Selenium is having any custom logic in order to click on options.

(clearing ni for James)

Comment 11

[Julian Descottes [:jdescottes]]

(In reply to Julian Descottes [:jdescottes] from comment #10)

Will check if Selenium is having any custom logic in order to click on options.

(clearing ni for James)

Selenium's click is using WebDriver ElementClick which is not relying on PerformActions at all, so there's no connection between the two topics.

Comment 12

[BugBot [:suhaib / :marco/ :calixte]]

Clear a needinfo that is pending on an inactive user.

Inactive users most likely will not respond; if the missing information is essential and cannot be collected another way, the bug maybe should be closed as INCOMPLETE.

For more information, please visit BugBot documentation.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08)]

This is as well happening for elements that have display: contents set like the following:

        <div style="background:green;height:2000px">filler</div>
        <div><button style="display: contents">click me</button></div>

The attempt to click the element fails with Origin element is not displayed. Note that our implementation of the WebDriver classic Element Click command fails as well.

Maybe we need widget event support that goes through APZ to actually get this fixed - means when we no longer have to check for the element's bounds.