1534461 - Build geckodriver for aarch64 on Windows

Status: RESOLVED FIXED

(Testing :: geckodriver, enhancement, P5)

Description

[Treeherder Bug Filer]

#[markdown(off)]
Filed by: egao [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=233233998&repo=try

https://queue.taskcluster.net/v1/task/dyvVY6vJQJm6l8ChoONYBA/runs/0/artifacts/public/logs/live_backing.log

Test: wdspec-1, wdspec-2

Context:
22:29:01 INFO - Calling ['c:\\mozilla-build\\python\\python.exe', '-u', 'C:\\tasks\\task_1552102824\\mozharness\\external_tools\\tooltool.py', '--url', 'https://tooltool.mozilla-releng.net/', 'fetch', '-m', 'C:\\tasks\\task_1552102824\\build\\tests\\config/tooltool-manifests/win32/releng.manifest', '-o', '-c', 'C:\\tooltool-cache'] with output_timeout 600
22:29:01 INFO - INFO - File win32-minidump_stackwalk.exe retrieved from local cache C:\tooltool-cache
22:29:01 INFO - Return code: 0
22:29:01 INFO - Chmoding C:\tasks\task_1552102824\build\win32-minidump_stackwalk.exe to 0755
22:29:01 FATAL - Unable to find geckodriver binary in common test package: C:\tasks\task_1552102824\build\tests\bin\geckodriver.exe
22:29:01 FATAL - Running post_fatal callback...
22:29:01 FATAL - Exiting -1
22:29:01 INFO - Running post-action listener: _package_coverage_data
22:29:01 INFO - Running post-action listener: _resource_record_post_action
22:29:01 INFO - Running post-action listener: process_java_coverage_data
22:29:01 INFO - Running post-action listener: stop_device

Comment 1

[Edwin Takahashi (:egao | infrequent contributor)]

:ato - I'm greening up the tests for windows10-aarch64, but there isn't an abundance of geckodriver.exe issues, though in 1442340 I saw some comments made by you. Are you the correct person to contact regarding this issue, or is there another person better suited to help?

Comment 2

[Andreas Tolfsen ❲:ato❳]

This is explained by the fact that we don’t build geckodriver on aarch64.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Why is that a wontfix? As it looks like we want to run all the tests on that platform and as such we would need a geckodriver build. For the time being we might have to disable the webdriver tests for aarch64.

Comment 4

[(Away)]

(I mid-aired with whimboo...)

(In reply to Andreas Tolfsen ⦗:ato⦘ from comment #2)

This is explained by the fact that we don’t build geckodriver on aarch64.

Then the action item for this bug should be to disable the tests, preferably with a comment explaining why we don't currently build geckodriver, so that we know how to re-evaluate whether we can enable it in the future.

Comment 5

[Geoff Brown [:gbrown]]

I'm not familiar with this issue, but this looks relevant:

https://searchfox.org/mozilla-central/rev/89414a1df52d06cfc35529afb9a5a8542a6e4270/toolkit/moz.configure#971

Comment 6

[Andreas Tolfsen ❲:ato❳]

Thanks for digging that up. The underlying tracking issue seems
to be https://github.com/retep998/winapi-rs/issues/727.

Comment 7

[Intermittent Failures Robot]

2 failures in 2771 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 2

Platform breakdown:

• windows10-aarch64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1534461&startday=2019-03-11&endday=2019-03-17&tree=all

Comment 8

[Tarek Ziadé (:tarek)]

Anything I can do to make this happen ? looks like the winapi issue was resolved

Comment 9

[David Burns :automatedtester]

This is a low priority for my team. I am happy for you to take this and see if you can get it building. We can support you if you have questions

Comment 10

[Nick Alexander :nalexander [he/him]]

This is probably similar to Bug 1580689.

Comment 11

[Nick Alexander :nalexander [he/him]]

Sounds like bumping winapi might address this.

Comment 12

[Tarek Ziadé (:tarek)]

note to my self: Bug 1437571 bumped 0.2.8

Comment 13

[Nathan Froyd [:froydnj]]

The problem is not that winapi needs a bump. The problem is that winapi 0.2.8 is deeply embedded in the dependency graph, thanks to mio and related crates, and the mio maintainer has shown zero interest in releasing a new version of mio that works with modern versions of winapi. See e.g. https://github.com/tokio-rs/mio/issues/870. Given that said issue is a year+ old and there hasn't been a new version of mio...or that other crates by the same author have fielded requests to release new versions with modern winapi dependencies with zero movement (e.g. https://github.com/carllerche/iovec/issues/16), I wouldn't hold your breath about things getting updated anytime soon.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I assume that all the deps for winapi should have been fixed those days. Most recent versionof winapi in tree is 0.3.7:
https://searchfox.org/mozilla-central/source/third_party/rust/winapi

Comment 15

[Stephen Donner [:stephend] Not actively reading bugmail]

Mass-removing myself from cc; search for 12b9dfe4-ece3-40dc-8d23-60e179f64ac1 or any reasonable part thereof, to mass-delete these notifications (and sorry!)

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+2]]

James, when we want to support that platform, would we be fine even that the aarch64-pc-windows-msvc toolchain is still Tier-2 (with the meaning of "guaranteed to build")?

Comment 17

[James Graham [:jgraham]]

I think we could release as experimental or something on that basis, as long as it's clear that the builds might go away if we stop producing Firefox on that platform.

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Ok, sounds fine to me.

Nathan, would we need a separate Rust toolchain job for aarch64 on Windows? I can only find cross-compile toolchains:
https://searchfox.org/mozilla-central/source/taskcluster/ci/toolchain/rust.yml#51-80

Comment 19

[Nathan Froyd [:froydnj]]

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+2] from comment #18)

Ok, sounds fine to me.

Nathan, would we need a separate Rust toolchain job for aarch64 on Windows? I can only find cross-compile toolchains:
https://searchfox.org/mozilla-central/source/taskcluster/ci/toolchain/rust.yml#51-80

It depends whether geckodriver is amenable to being cross-compiled from Windows to Linux. If it is, then those toolchains should suffice. If it's not, then we have native Windows toolchains you can use instead: https://searchfox.org/mozilla-central/source/taskcluster/ci/toolchain/rust.yml#153-176

Comment 20

[u668780]

(In reply to James Graham [:jgraham] from comment #17)

the builds might go away if we stop producing Firefox on that platform.

Could you please elaborate? Are you considering dropping support for Windows Arm64 at the moment?

Comment 21

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Andrew, to prioritize the work I would like to know if testing with Browsertime is a requirement for the performance team. Thanks.

Comment 22

[Andrew Creskey [:acreskey]]

Hi Henrik,

Local testing with Browsertime is definitely a priority for the performance team.
There are many features that have to be tested locally (generally because there is no solution in CI yet).
e.g. Fission, most new network features, experiments, etc.

However I haven't heard of aarch64 Windows being a priority in a long time.

Comment 23

[Henrik Skupin [:whimboo][⌚️UTC+2]]

This is now being worked on bug 1709282.

Comment 24

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Bug 1709282 actually only made the build available, but we don't run any wdspec tests. So lets take care of that here.

Joel, do we have any capacity to run some or all Wdspec tests for Windows aarch64 at least on Nightly builds? Or what is our strategy for this platform?

Comment 25

[Joel Maher ( :jmaher ) (UTC -8)]

windows/aarch64 is still shipped but we have cut our tests to the bare minimum and reduced our pool of machines to match that. The hardware and OS we run on are outdated and do not match reality for the few users- our goal here is to ensure Firefox runs at a minimal level and if in the future this platform becomes more popular then we will increase our investment into it with upgrading machines and increasing them- then looking at more complete unit tests and turning on some performance tests.

Comment 26

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Reverting summary because bug 1709282 didn't take care of CI builds.

(In reply to Joel Maher ( :jmaher ) (UTC -0800) from comment #25)

windows/aarch64 is still shipped but we have cut our tests to the bare minimum and reduced our pool of machines to match that.

Thanks for the feedback. I wonder if it would be useful to have at least builds for that platform, and signed via our CI. Similar to what we do for MacOS AArch64. Do you think that our build machines could easily handle that extra toolchain task build, and that it is worth the efforts? I'm happy to even further delay these kind of build until users of geckodriver really request for it.

Comment 27

[Joel Maher ( :jmaher ) (UTC -8)]

the builds of win/aarch64 are on linux (via cross compile), so adding a build done via linux seems a-ok with me.

Comment 28

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Sounds good. I'll have a look when I find some spare time to do it. Might not bee soon.

Comment 29

[Mike Hommey [:glandium]]

Attachment: Bug 1534461 - Build geckodriver for aarch64 on Windows.

And avoid adding yet another vs-setup script by merging the existing ones.

Comment 30

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3c402727e2e4
Build geckodriver for aarch64 on Windows. r=whimboo,firefox-build-system-reviewers,ahochheiden

Comment 31

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/3c402727e2e4

Comment 32

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Joel, are there plans to have Windows 10 aarch64 test machines in CI as well (similar to Linux)?

Comment 33

[Joel Maher ( :jmaher ) (UTC -8)]

we have 1 machine in CI and plans this year to replace that (although not much has changed, so we are getting 2 or 3 machines). Currently I think we only run mochitest-media on the machines, there would be room for a little more to run. Keep in mind right now our machine goes offline very frequently.