1762131 - Intermittent Android webdriver/tests/new_session/platform_name.py | test_corresponds_to_local_system - AssertionError: assert 'android' == 'linux'

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect, P5)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

With the upcoming support for wdspec tests on Android the following test fails because the wrong platform name is assumed:

https://firefoxci.taskcluster-artifacts.net/ZudT0eAHQSKdfO1IywbDqg/1/public/logs/live_backing.log

[task 2022-03-29T12:46:02.701Z] 12:46:02     INFO - STDOUT:     @pytest.mark.skipif(platform_name is None, reason="Unsupported platform {}".format(platform_name))
[task 2022-03-29T12:46:02.701Z] 12:46:02     INFO - STDOUT:     def test_corresponds_to_local_system(new_session, add_browser_capabilities):
[task 2022-03-29T12:46:02.701Z] 12:46:02     INFO - STDOUT:         response, _ = new_session({"capabilities": {"alwaysMatch": add_browser_capabilities({})}})
[task 2022-03-29T12:46:02.702Z] 12:46:02     INFO - STDOUT:         value = assert_success(response)
[task 2022-03-29T12:46:02.702Z] 12:46:02     INFO - STDOUT: >       assert value["capabilities"]["platformName"] == platform_name
[task 2022-03-29T12:46:02.702Z] 12:46:02     INFO - STDOUT: E       AssertionError: assert 'android' == 'linux'
[task 2022-03-29T12:46:02.702Z] 12:46:02     INFO - STDOUT: E         - linux
[task 2022-03-29T12:46:02.702Z] 12:46:02     INFO - STDOUT: E         + android

We should not assume that the platformName capability will be equal to the host system. I'm going to disable this test for now via bug 1749444.

Comment 1

[BMO Automation]

Moving bug to Remote Protocol::Marionette component per bug 1815831.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

We probably should pass the expected platform name via a wdspec config setting to the tests. As it looks like it can / should be done from within the wptrunners executors given that they know on which platform the tests are run.

James, do you have any suggestion?

Comment 3

[James Graham [:jgraham]]

You can pass it in via the session_config: https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/executors/base.py#656

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Yes, I understand, but I may not have been precise enough in my question. We need to determine the platform (name) that we intend to set. I have been trying to find where in wptrunner the target platform (where the tests are actually running) is specified, but I couldn't find any information related to the executors. It seems that the run_info_data isn't passed through intentionally. Perhaps we need to pass this information through the various browser classes?

Comment 5

[James Graham [:jgraham]]

Indeed we don't pass it through unconditionally, but you could add the platform name in https://searchfox.org/mozilla-central/source/testing/web-platform/tests/tools/wptrunner/wptrunner/executors/base.py#38-42 which is where we're adding the other properties we want to pass through to pytest.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

With bug 1903706 the test is no longer disabled and passes now.