1908788 - mach test doesn't work for WebDriver tests on Android

Status: RESOLVED FIXED

(Testing :: geckodriver, defect, P3)

Description

[Hiroyuki Ikezoe (:hiro)]

When I run "./mach test --webdriver-binary=obj-x86_64-unknown-linux-android/dist/bin/geckodriver testing/web-platform/tests/webdriver/tests/classic/element_click/scroll_into_view.py" locally, it fail with;

  File "/home/hiro/android/testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/base.py", line 364, in _run_server
    self._proc.run()
  File "/home/hiro/android/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 850, in run
    self.proc = self.Process([self.cmd] + self.args, **args)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hiro/android/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 146, in __init__
    subprocess.Popen.__init__(self, args, **kwargs)
  File "/home/hiro/android/third_party/python/sentry_sdk/sentry_sdk/integrations/stdlib.py", line 190, in sentry_patched_popen_init
    rv = old_popen_init(self, *a, **kw)  # type: ignore
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.12/subprocess.py", line 1955, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/hiro/android/obj-x86_64-unknown-linux-android/dist/bin/geckodriver'

The geckodriver binary exists there. (you might need to specify ac_add_options --enable-geckodriver in .mozconfig)

Comment 1

[Hiroyuki Ikezoe (:hiro)]

It looks like the geckodriver binary is for Android, it links to /system/bin/linker64. I've heard from Henrik that the geckodriver runs in the host, so that it needs to be a Linux binary?

Comment 2

[Hiroyuki Ikezoe (:hiro)]

With;

diff --git a/testing/geckodriver/moz.build b/testing/geckodriver/moz.build
--- a/testing/geckodriver/moz.build
+++ b/testing/geckodriver/moz.build
@@ -2,7 +2,7 @@
 # License, v. 2.0. If a copy of the MPL was not distributed with this
 # file, You can obtain one at http://mozilla.org/MPL/2.0/.
 
-RUST_PROGRAMS += ["geckodriver"]
+HOST_RUST_PROGRAMS += ["geckodriver"]
 # Some Rust build scripts compile C/C++ sources, don't error on warnings for them.
 AllowCompilerWarnings()

and --webdriver-binary=obj-x86_64-unknown-linux-android/dist/host/bin/geckodriver, the test runs properly.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

That's interesting. Given that we all use artifact builds we never noticed that problem. Also in CI mozharness is used so that's really an issue when running tests locally.

Also with the proposed solution above, is it still required then to specify the --webdriver-binary argument or does mach test find the geckodriver binary automatically? If not, what about mach wpt?

Comment 4

[Hiroyuki Ikezoe (:hiro)]

Though I will try it later.

(In reply to Henrik Skupin [:whimboo][⌚️UTC+1] from comment #3)

That's interesting. Given that we all use artifact builds we never noticed that problem. Also in CI mozharness is used so that's really an issue when running tests locally.

Also with the proposed solution above, is it still required then to specify the --webdriver-binary argument or does mach test find the geckodriver binary automatically? If not, what about mach wpt?

You meant that running the wdspec test with ./mach wpt? It causes the same error. Interestingly for normal web-platform-tests, ./mach wpt has been working fine. That's odd to me, though.

Comment 5

[Hiroyuki Ikezoe (:hiro)]

Without specifying --webdriver-binary the geckobinary wasn't built. (though now I am targetting --arm so that things might be different)

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Normal wpt tests do not need geckodriver but purely run by using the Marionette protocol. So that's why it's not an issue with this kind of tests.

So there are two things to fix:

1. The moz.build flag as you mentioned above.

2. Fixing wpt to accurately find the geckodriver binary when running tests for Android. I'm sure here we just ignore the fact that there is another subfolder called host and just look at obj-x86_64-unknown-linux-android/dist/bin/geckodriver. A fix for that should probably be around these lines.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Hiroyuki Ikezoe (:hiro) from comment #5)

Without specifying --webdriver-binary the geckobinary wasn't built. (though now I am targetting --arm so that things might be different)

That actually shouldn't have any impact if the binary is build or not. It should only be related to the ac_add_options --enable-geckodriver flag.

Comment 8

[Hiroyuki Ikezoe (:hiro)]

(In reply to Henrik Skupin [:whimboo][⌚️UTC+1] from comment #7)

(In reply to Hiroyuki Ikezoe (:hiro) from comment #5)

Without specifying --webdriver-binary the geckobinary wasn't built. (though now I am targetting --arm so that things might be different)

That actually shouldn't have any impact if the binary is build or not. It should only be related to the ac_add_options --enable-geckodriver flag.

Bah, I meant --enable-geckodriver. :/

Comment 9

[Hiroyuki Ikezoe (:hiro)]

Without --webdriver-binary option, either ./mach test or ./mach wpt doesn't work.

Comment 10

[James Graham [:jgraham]]

Attachment: Bug 1908788 - Ensure we build geckodriver for host platform,

Comment 11

[James Graham [:jgraham]]

I put a patch up, but I don't know if it works. Hiro can you test?

Comment 12

[James Graham [:jgraham]]

Attachment: Bug 1908788 - Correctly locate geckodriver binary for mach wpt on Android,

The functional change here is searching the host binary directory as
well as the dist directory.

The code is also refactored a bit to split the firefox_android
processing into its own function, like for desktop Firefox, and to
share the geckodriver search code through a helper.

Comment 13

[Hiroyuki Ikezoe (:hiro)]

With D217953 and without specifying --webdriver-binary option, I got "ValueError: WebDriver server binary must be given to --webdriver-binary argument" error message.

With D217978 (and with D217953) and without specifying --webdriver-binary option, I got "
OSError: WebDriver executable not found: /home/hiro/android/obj-x86_64-unknown-linux-android/dist/bin/geckodriver".

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Hiro, can you please test again with the updated code? Does it work correctly now?

Comment 15

[Hiroyuki Ikezoe (:hiro)]

I applied both of D217953 and D217978, but it doesn't yet work without specifying --webdriver-binary.

Note that I realized without ac_add_options --enable-geckodriver in .mozconfig, the geckodriver binary isn't built.

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Thanks Hiro. Could you maybe have a look at D217978? Especially around the find_webdriver_binary method? Does it not find any binary or does it just produce a wrong path?

Comment 17

[Hiroyuki Ikezoe (:hiro)]

Now it works without specifying --webdriver-binary option. Thank you jgraham!

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e1b438ca7630
Ensure we build geckodriver for host platform, r=webdriver-reviewers,whimboo
https://hg.mozilla.org/integration/autoland/rev/d706a6272644
Correctly locate geckodriver binary for mach wpt on Android, r=webdriver-reviewers,whimboo

Comment 19

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/e1b438ca7630
https://hg.mozilla.org/mozilla-central/rev/d706a6272644