1520867 - Investigate running tests on Windows / arm64

Status: RESOLVED FIXED

(Testing :: General, enhancement, P1)

Description

[Geoff Brown [:gbrown]]

We have firefox builds for Windows aarch64. Can we run tests against them? Let's try running the tests we normally run against Windows 10 builds in continuous integration on a Lenovo Yoga and see what happens...

Comment 1

[Geoff Brown [:gbrown]]

I thought it would be convenient to run in a MozillaBuild environment: https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Windows_Prerequisites#Getting_the_source

Recent releases of MozillaBuild won't install on the Yoga, because they are x86_64 only. But MozillaBuild 2.2.0 does install -- I'm using that. https://ftp.mozilla.org/pub/mozilla/libraries/win32/MozillaBuildSetup-2.2.0.exe

Comment 2

[Geoff Brown [:gbrown]]

It sounds like most (all) folks build for aarch64 on x86 machines -- not on the Yoga. So running in a mach build context is tricky. Instead, let's concentrate on running from test zips....

Comment 3

[Geoff Brown [:gbrown]]

Attachment: initial script for running mozharness on Yoga

A quick and dirty first attempt. This runs in the MozillaBuild environment, kicking off the mozharness script correctly. On my first try, desktop_unittest failed because the screen resolution could not be set.

Comment 4

[Geoff Brown [:gbrown]]

The screen resolution issue is related to scaling. By default, the Yoga display settings have "Change the size of text, apps, and other items" as 150%. With this setting, our code for checking screen resolution reports incorrect (scaled) values, causing a fatal error. Change the setting to 100% to workaround.

Comment 5

[Geoff Brown [:gbrown]]

Next problem I encountered: powershell is called from mozharness, but powershell is not in the PATH of the MozillaBuild environment.

Comment 6

[Geoff Brown [:gbrown]]

Worth noting that there is a considerable delay in/after fetch_url_into_memory that might feel like a hang - just give it a few minutes. I think this is just the effect of limited memory or a slow file system during archive extraction; warrants more investigation, eventually.

Comment 7

[Geoff Brown [:gbrown]]

With powershell in PATH, on my next run the mochitest test harness started firefox; there was a crash and a system alert that the firewall had blocked some features of python.exe; I clicked to allow; later manifests ran mochitests - progress!

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

excellent, I believe we turn those prompts off in automation with the automated setup; I do this manually while running locally all too often.

Comment 9

[Geoff Brown [:gbrown]]

Attachment: initial script for running mozharness on Yoga

...now a little more general, but doesn't handle all suites yet.

Comment 10

[Geoff Brown [:gbrown]]

Attachment: initial script for running mozharness on Yoga

Comment 11

[Geoff Brown [:gbrown]]

I have completed an initial run of cppunit, gtest, and mochitest-plain (all chunks) so far. Most logs have some failures, but most tests pass. There are some crashes, so I'm not sure we are completing all runs, but it looks like run-times are reasonable: maybe 20% slower than existing Windows 10 test tasks seen on treeherder.

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

thanks :gbrown. I would be eager to see if mochitest-media and reftests will run. So far this is not sounding too scary once we get core tooling setup.

Comment 13

[Geoff Brown [:gbrown]]

Attachment: script for running mozharness on Yoga

Comment 14

[Geoff Brown [:gbrown]]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #12)

thanks :gbrown. I would be eager to see if mochitest-media and reftests will run. So far this is not sounding too scary once we get core tooling setup.

Those - mochitest-media and reftest - look okay, more or less. I see about 30 unexpected failures in reftest chunk 1, for example.

Comment 15

[Edwin Takahashi (:egao | infrequent contributor)]

I tried the following:

mochitest-chrome
Tests are started, with the screen resolution being altered to 1280x1024. However, each test under widget/tests/ have status of TEST-SKIP.

There is a failure caused by the step Running manifest: browser/base/content/test/chrome/chrome.ini.

web-platform
Could not get tests to run - dependencies are installed but tests encounter a critical failure at metadata path.

Comment 16

[Joel Maher ( :jmaher ) (UTC -8)]

:egao, can you run the tests that gbrown ran to ensure that you are have a working environment and scripts are setup correctly.

Comment 17

[Geoff Brown [:gbrown]]

(In reply to Edwin Gao (:egao) from comment #15)

I tried the following:

mochitest-chrome
Tests are started, with the screen resolution being altered to 1280x1024. However, each test under widget/tests/ have status of TEST-SKIP.

There are a lot of skipped tests in widget/tests:

https://searchfox.org/mozilla-central/source/widget/tests/chrome.ini

but there should be some that run. For instance, widget/tests/test_bug1151186.html runs and passes for me.

There is a failure caused by the step Running manifest: browser/base/content/test/chrome/chrome.ini.

I don't see that failure in my mochitest-chrome run: I have 4 passing tests on that path.

There might be variation depending on the build. I used task id MEP5IaDDRvagYX5ZN9sMzw.

web-platform
Could not get tests to run - dependencies are installed but tests encounter a critical failure at metadata path.

I haven't successfully run wpt yet either.

Comment 18

[Geoff Brown [:gbrown]]

Attachment: script for running mozharness on Yoga

This has updates for wpt, but I have not run wpt successfully yet.

Comment 19

[Geoff Brown [:gbrown]]

I found an issue with wpt: If I use my script to run any web-platform suite once, then run any web-platform suite again, the second run fails because it cannot delete build/tests/web-platform/tests/fonts/Ahem.ttf. I assume the file is locked because the font is installed? I imagine we would not see this in CI because we reboot between runs...or maybe the Windows worker makes special allowance for this? My workaround: reboot between runs.

Comment 20

[Joel Maher ( :jmaher ) (UTC -8)]

I have confirmed that we terminate an instance in aws when we run wpt on windows10- so this theory of a reboot holds true.

Comment 21

[Edwin Takahashi (:egao | infrequent contributor)]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #16)

:egao, can you run the tests that gbrown ran to ensure that you are have a working environment and scripts are setup correctly.

Definitely. I have gone ahead and created a matrix using Google Sheets that aims to corroborate results that :gbrown is seeing. This way I can also check if the issues are due to my environment, setup, or the task ID chosen.

(In reply to Geoff Brown [:gbrown] from comment #17)

There are a lot of skipped tests in widget/tests:

https://searchfox.org/mozilla-central/source/widget/tests/chrome.ini

but there should be some that run. For instance, widget/tests/test_bug1151186.html runs and passes for me.

Perhaps my difficulties are due to choosing a bad build. I ran mochitest-chrome with both your task ID and my selected task ID. Guess which one worked and which did not. I apparently chose a win64/pgo build which might explain my failures.

The Google Sheet is here.

Comment 22

[Geoff Brown [:gbrown]]

(In reply to Edwin Gao (:egao) from comment #21)

The Google Sheet is here.

Thanks Edwin. Now that we have builds synced up, we seem to get basically the same results (mochitest-plain seems to have an oddly variable number of passes, but I see that too from one run to another).

web-platform tests need some work: there may still be environment or harness problems or wpt.

Otherwise, tests can be run, but we seem to see a lot of crashes. I'd like to verify that we still crash with a recent build (we've been testing with a build from Jan 16).

Comment 23

[(Away)]

Bug 1512822 just landed on central so you may want to test with the next nightly after this post, just in case any of the previous test failures were compiler-related.

Comment 24

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] (pto Jan 28-30) from comment #19)

I found an issue with wpt: If I use my script to run any web-platform suite once, then run any web-platform suite again, the second run fails because it cannot delete build/tests/web-platform/tests/fonts/Ahem.ttf. I assume the file is locked because the font is installed? I imagine we would not see this in CI because we reboot between runs...or maybe the Windows worker makes special allowance for this? My workaround: reboot between runs.

My font-locking problem goes away with the patch from bug 1522696 - that makes sense! - so reboots are no longer required.

Comment 25

[Geoff Brown [:gbrown]]

Attachment: script for running mozharness on Yoga

...now updated to run raptor-tp6-1

Comment 26

[Tom Prince [:tomprince]]

Attachment: Bug 1520867: Add temporary scope for testing bitbar win64-aarch64 worker; r?dustin

If this works, we should change the worker-type to something more permanent.
This is using a worker-group intened for testing generic-worker itself.

Comment 27

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: script for running mozharness on Yoga

Comment 28

[Edwin Takahashi (:egao | infrequent contributor)]

Added jittest to the list.

Comment 29

[Tom Prince [:tomprince]]

Attachment: Bug 1520867: Remove scopes for temporary bitbar worker; r?dustin

Comment 30

[Geoff Brown [:gbrown]]

Attachment: script for running mozharness on Yoga

Comment 31

[Geoff Brown [:gbrown]]

Attachment: script for running mozharness on Yoga

Now updated for all the raptor suites and all the talos suites run on Windows 10 x64. I have not tested all of these -- let me know if you find problems!

Comment 32

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Geoff Brown [:gbrown] from comment #31)

Created attachment 9046793 [details]
script for running mozharness on Yoga

Now updated for all the raptor suites and all the talos suites run on Windows 10 x64. I have not tested all of these -- let me know if you find problems!

This has been working tremendously well, Geoff, and has saved me a ton of effort, mistakes, and frustration -- thanks again for it!

Nothing official yet in this tests-on-arm64 readiness tracking spreadsheet, but it's going well (apart from seemingly spurious Firefox + Windows 10 crashes (they're not often, but they are frequent). I've also taken/been pushed a few Windows 10 updates (the latest of which is their "October Update"), so hopefully that helps.

Next steps (given priorities/time) are probably to start pushing CI-config (yaml?) changes to Try, and start gathering data from larger-scale test runs?

Comment 33

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Stephen Donner [:stephend] from comment #32)

Next steps (given priorities/time) are probably to start pushing CI-config (yaml?) changes to Try, and start gathering data from larger-scale test runs?

We should file bugs for all failures, and either fix or disable them. If disabled, the bugs should remain open until the root cause is identified and resolved.

Comment 34

[Geoff Brown [:gbrown]]

The script attachment here does all that I wanted in this bug: allows simple local runs of virtually all test suites. If anyone needs additional capabilities, please needinfo me.

But now it's generally easier to use try (disabled by default currently, until we get more capacity), and egao is using that to identify and report test failures.

Comment 35

[Geoff Brown [:gbrown]]

Attachment: script for running mozharness on Yoga

The old one bitrotted -- updated!

Comment 36

[Stephen Donner [:stephend] Not actively reading bugmail]

Geoff, mind taking a look at adding (at least a subset of?) awsy-test to this convenience wrapper script? For context, I've been running and working on awsy-tp6 (sy-tp6 in Treeherder), in support of bug 1567138.

A sample of a successful run, today, without invoking via your script, looks like (run from c:/mozilla-build):

python.exe mozharness/scripts/awsy_script.py --cfg mozharness/configs/awsy/taskcluster_windows_config.py --test-packages-url https://q ueue.taskcluster.net/v1/task/VcvAL16PQxChDJ9BK8g8qQ/runs/0/artifacts/public/build/target.test_packages.json --installer-url https://queu e.taskcluster.net/v1/task/VcvAL16PQxChDJ9BK8g8qQ/runs/0/artifacts/public/build/target.zip --download-symbols ondemand --tp6

I'm not well-versed on the various arguments one can pass, etc., but the relevant first entrypoint is https://wiki.mozilla.org/Project_Fission/Memory#AWSY_.28tp6.29.

Comment 37

[Geoff Brown [:gbrown]]

Attachment: script for running mozharness on Yoga

Added support for awsy, awsy-base, awsy-tp6.

Comment 38

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Geoff Brown [:gbrown] from comment #37)

Created attachment 9091960 [details]
script for running mozharness on Yoga

Added support for awsy, awsy-base, awsy-tp6.

Thanks; these additions (particularly awsy-tp6) have been serving me well. Really appreciate the quick turnaround!