1699044 - [wdspec] Add wdspec tests for Fenix / Firefox on Android

Status: RESOLVED FIXED

(Testing :: geckodriver, task, P2)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

As of now all the wdspec tests that we are running in CI are done on desktop platforms, without any support for Android:

https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=wdspec

That means that the Android support that we are shipping for geckodriver is basically untested. The only pro side is that we have browsertime jobs actually utilizing the code path.

Nevertheless browsertime jobs seem to set some special Firefox preferences like devtools.debugger.remote-enabled=true, which make testing possible in Fenix. But that pref is not set by default in geckodriver, and as such normal users of Selenium/geckodriver will hit bug 1698968. For them it's not possible to test Fenix at all unless the pref is set via the session capabilities.

Joel, would there be enough bandwidth for e.g one Android platform to run wdspec tests? I think that we would also be happy in just running it on mozilla-central for opt.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

I think we can do this specifically on emulators. As for hardware that is even more restricted than Apple, the more we can fine tune what we run, the better the chance we can squeeze it in. Emulators can run whatever you want to run!

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I assume that all emulators and also the hardware are working as rooted devices? If that is the case is there a way to have unrooted support somewhere? That's actually missed most right now, and causes the main problem in geckodriver for Android.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

I know the phones are rooted, I don't know about the emulators, let me redirect to :aerickson to comment if the emulators are rooted.

Comment 4

[Andrew Erickson [:aerickson]]

Emulators are rooted. Most of hardware devices are also rooted. We have 4 S7 Galaxy phones that are unrooted (rooted, but the rooting is disabled).

https://firefox-ci-tc.services.mozilla.com/provisioners/proj-autophone

worker-types are:
gecko-t-bitbar-gw-unit-s7 (2 devices)
gecko-t-bitbar-gw-perf-s7 (2 devices)

Andy

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Andrew Erickson [:aerickson] from comment #4)

Emulators are rooted. Most of hardware devices are also rooted. We have 4 S7 Galaxy phones that are unrooted (rooted, but the rooting is disabled).

Thanks Andy! I think going forward by using emulators might be the way to go then. I will have a look at it once dependencies are fixed.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Maybe we can also get bug 1708105 easily fixed to not have to disable that many wdspec tests for Android because user prompts cannot be handled.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

James, could you maybe give us some hints in where to get started with wptrunner to get Android support added? Basically what we would have to do when creating a new session is to set the androidPackage capability. So it has to be passed-through from wptrunner. Could you foresee anything else that would be necessary?

Comment 8

[James Graham [:jgraham]]

Attachment: Bug 1699044 - Add wdspec support to firefox_android,

We can reuse most of the implementation from Firefox, we just need to
ensure that the corect android options end up in the capabilities.

Comment 9

[James Graham [:jgraham]]

Attachment: Bug 1699044 - Dont' try to set the wdspec window size when it's not supported.

Not all platforms, notably mobile, support this, so we shouldn't
depend on it working in order for tests to run.

Comment 10

[James Graham [:jgraham]]

I put up a couple of patches that are enough to get things to run at least a little bit locally. I am still seeing some errors e.g. openTab not supported in the test browser. Maybe we want to default to a different package for these tests? That's a little tricky at the moment since we are setting the default value in the frontend (mach_commands.py or tools/wpt/run.py). I think we'd need to let it be passed in as None by default, and have the firefox_android.py take care of setting it to the right value depending on the test type, where ever it's used.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Thanks a lot James! That's pretty helpful when we would be able to run wdspec tests locally at least. We are aware that a lot is still not working yet including the WebDriver:NewWindow command, which is used for openTab. But that shouldn't stop us from landing changes to wptrunner. Once we can run locally it will be easier to fix.

I think that we should be fine in just using the default, which I assume is GeckoView_example? If we would like to run for Fenix we would have to create the jobs somewhere else anyway.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Bug 1708105 shouldn't block us from getting the attached patches on this bug landed. It's more the other way around given that they are helping us to better identify broken areas in wdspec that we have to focus on.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c9dafde8b9ee
Add wdspec support to firefox_android, r=webdriver-reviewers,whimboo
https://hg.mozilla.org/integration/autoland/rev/d594570f12d1
Dont' try to set the wdspec window size when it's not supported. r=webdriver-reviewers,whimboo

Comment 14

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/29949 for changes under testing/web-platform/tests

Comment 15

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/c9dafde8b9ee
https://hg.mozilla.org/mozilla-central/rev/d594570f12d1

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Web Platform Test Sync Bot (Matrix: #interop:mozilla.org) from comment #14)

Created web-platform-tests PR
https://github.com/web-platform-tests/wpt/pull/29949 for changes under
testing/web-platform/tests

James, can you please check why this PR doesn't get merged?

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Turns out this was easier as expected. Lets count it for our M1 fixed bugs.

Comment 18

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Upstream PR merged by jgraham

Comment 19

[Julian Descottes [:jdescottes]]

Will need to file follow up bugs to file to start running wdspec tests on Android on central.
As I understand the bugs that landed here get us closer to that goal, but we're still not running the tests.

Comment 20

[Henrik Skupin [:whimboo][⌚️UTC+1]]

We cannot run these tests on central yet due to various reasons, which are all marked as blocking bug 1723923. Nothing more needs to be done here.

Comment 21

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+1] from comment #20)

We cannot run these tests on central yet due to various reasons, which are all marked as blocking bug 1723923. Nothing more needs to be done here.

Actually this bug is BiDi only. I filed bug 1729409 for getting wdspec support added for Android.