1865886 - Use aarch64 builds for PGO profile generation on Android

Status: RESOLVED FIXED

(Core :: Performance Engineering, task)

Description

[Jamie Nicol [:jnicol] out of office until 6th Jan]

Currently we use x86 instrumented builds to generate profile data for arm32 shippable builds, and x86_64 for aarch64.

I see a 3 to 4% improvement in speedometer3 score when doing an aarch64 build using profile data I generated locally from an aarch64 instrumented build compared to an x86_64 one. (By hacking the android_emulator_pgo.py script to run locally and performing the profile generation in an emulator on my M2 macbook).

In terms of performing these builds in CI:

• Recent android arm64 system images cannot be emulated on an x86_64 host. However, we may be able to use an older android version system image. Prior to bug 1718341 we used an arm32 image, but we switched away for a reason.
• Recent x86_64 system images can obviously be run on an x86_64 host, and are able to run aarch64 binaries. However, we need to use the google_apis variant, and I'm not sure whether that's okay.
• Perhaps we can run an aarch64 system image on an aarch64 Linux host

Comment 1

[Jamie Nicol [:jnicol] out of office until 6th Jan]

As well as bug 1677963 blocking the "run on an arm64 linux host" option, there are also not yet officially-supported arm64 linux emulator builds: https://issuetracker.google.com/issues/242699119

Comment 2

[Jamie Nicol [:jnicol] out of office until 6th Jan]

Recent android arm64 system images cannot be emulated on an x86_64 host. However, we may be able to use an older android version system image. Prior to bug 1718341 we used an arm32 image, but we switched away for a reason.

This also doesn't appear to work. While I can get an arm32 SDK 24 emulator image to launch, I cannot get any arm64 one to. Even on a local machine.

On SDK level 24 and 25 it gets stuck in a bootloop. Nothing hugely jumps out in the logs as the main issue, but googling most of the warnings shows other people also failing to launch emulators.

And on SDK 26 onwards it exits immediately with this error:

qemu-system-aarch64-headless: PCI bus not available for hda
VERBOSE | Done with QEMU main loop

which, again, googling results in others failing to launch arm64 emulators.

And on SDK 28 onwards it's a lot more explicit:

PANIC: Avd's CPU Architecture 'arm64' is not supported by the QEMU2 emulator on x86_64 host.

Comment 3

[Jamie Nicol [:jnicol] out of office until 6th Jan]

(In reply to Jamie Nicol [:jnicol] from comment #0)

• Recent x86_64 system images can obviously be run on an x86_64 host, and are able to run aarch64 binaries. However, we need to use the google_apis variant, and I'm not sure whether that's okay.

This approach works, and gives a 4.35% improvement on try: https://treeherder.mozilla.org/perfherder/compare?originalProject=try&originalRevision=6dcce504a500bb2c227f42066d0635c63fd5531d&newProject=try&newRevision=10b59290544284b699b973c9b92e60c66c53650c&framework=13&page=1

However, the profile generation needs to be run on a device with KVM access, which is blocked on bug 1545497. Without KVM, recent android emulator images (especially google_apis variants) are unusable: they take upwards of 30 minutes to boot, frequently crash, and time out when attempting to install an APK.

Comment 4

[Jamie Nicol [:jnicol] out of office until 6th Jan]

I should also note that I needed to use an android-31 google_apis x86_64 emulator image for this to work, which only supports running aarch64 binaries (and x86_64 of course), not arm32. From SDK 31 onwards the emulators are 64-bit only. So we can only get this win for aarch64 builds, not arm32. That's the vast majority of our users though, so that's fine.

Comment 5

[Jamie Nicol [:jnicol] out of office until 6th Jan]

See https://mozilla-hub.atlassian.net/browse/RELOPS-789 for adding required workers to CI

Comment 6

[Jeff Muizelaar [:jrmuizel]]

Attachment: Bug 1865886. Add an android-31 emulator package and add google_apis.

Comment 7

[Jeff Muizelaar [:jrmuizel]]

Attachment: Bug 1865886. Use kvm to repack avd.

Comment 8

[Jeff Muizelaar [:jrmuizel]]

Attachment: Bug 1865886. Add aarch64 instrumented jobs.

Comment 9

[Jeff Muizelaar [:jrmuizel]]

Attachment: Bug 1865886. Add aarch64 profile generation jobs on x86_64 emulator.

Comment 10

[Jeff Muizelaar [:jrmuizel]]

Attachment: Bug 1865886. Use aarch64 profile generate jobs for shippable builds.

Comment 11

[Jeff Muizelaar [:jrmuizel]]

It looks like this breaks the x86-32 PGO run. I get:
INFO - Failed to install /builds/worker/fetches/geckoview-test_runner.apk on None: ADBProcessError args: /builds/worker/fetches/android-sdk-linux/platform-tools/adb wait-for-device install /builds/worker/fetches/geckoview-test_runner.apk, exitcode: 1, stdout: adb: failed to install /builds/worker/fetches/geckoview-test_runner.apk: Failure [INSTALL_FAILED_NO_MATCHING_ABIS: Failed to extract native libraries, res=-113]

Comment 12

[Jeff Muizelaar [:jrmuizel]]

Ah and the reason is in comment 4 "From SDK 31 onwards the emulators are 64-bit only", so we'll need to use separate versions of the emulator.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7bf33179c4b8
Add an android-31 emulator package and add google_apis. r=geckoview-reviewers,glandium,m_kato
https://hg.mozilla.org/integration/autoland/rev/df1b87110c9d
Add aarch64 instrumented jobs. r=geckoview-reviewers,glandium,m_kato
https://hg.mozilla.org/integration/autoland/rev/e2529b2ccc91
Add aarch64 profile generation jobs on x86_64 emulator. r=glandium
https://hg.mozilla.org/integration/autoland/rev/7cc03b5ce4a9
Use aarch64 profile generate jobs for shippable builds. r=glandium

Comment 14

[Noemi Erli[:noemi_erli]]

Backed out 4 changesets (Bug 1865886) for causing python failures CLOSED TREE

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&collapsedPushes=842382&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&searchStr=linux%2C18.04%2Cx64%2Copt%2Cpython%2C3%2Cunit%2Ctests%2Csource-test-python-firefox-ci-py3%2Cci&revision=7cc03b5ce4a92298cc1ed57ebebb7f02c8c71494&selectedTaskRun=Ha6NPorDQ-ykxYDan4D_Fw.0

Log: https://treeherder.mozilla.org/logviewer?job_id=444815773&repo=autoland&lineNumber=698

Backout: https://hg.mozilla.org/integration/autoland/rev/73cc5e578b5aa252f226698a9353564bccb34f54

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e2c2461d3c62
Add an android-31 emulator package and add google_apis. r=geckoview-reviewers,glandium,m_kato
https://hg.mozilla.org/integration/autoland/rev/6f85a2b71370
Add aarch64 instrumented jobs. r=geckoview-reviewers,glandium,m_kato
https://hg.mozilla.org/integration/autoland/rev/b1155fcccab5
Add aarch64 profile generation jobs on x86_64 emulator. r=glandium
https://hg.mozilla.org/integration/autoland/rev/252218f93e3a
Use aarch64 profile generate jobs for shippable builds. r=glandium

Comment 16

[Jeff Muizelaar [:jrmuizel]]

Attachment: Bug 1865886. Fix merge with changes from bug 1874234.

This fixes the ModuleNotFoundError: No module named 'mozinfo' error.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d2a5bbf7755f
Fix merge with changes from bug 1874234. CLOSED TREE

Comment 18

[Jeff Muizelaar [:jrmuizel]]

Comment on attachment 9375833 [details]
Bug 1865886. Use aarch64 profile generate jobs for shippable builds.

Beta/Release Uplift Approval Request

• User impact if declined: 5% on Speedometer3
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: Bug 1877194, Bug 1875490, Bug 1876337
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This changes the set of functions for which PGO works. The risk for change in behaviour should be very low.

The biggest risk is around the needed infrastructure changes. It's possible this could break the build when uplifted to beta or cause some other weirdness. Any problems should be easy to fix though.

• String changes made/needed: n/a
• Is Android affected?: Yes

Comment 19

[Jeff Muizelaar [:jrmuizel]]

I should note that this appears to increase the APK size by about 400k

Comment 20

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/e2c2461d3c62
https://hg.mozilla.org/mozilla-central/rev/6f85a2b71370
https://hg.mozilla.org/mozilla-central/rev/b1155fcccab5
https://hg.mozilla.org/mozilla-central/rev/252218f93e3a
https://hg.mozilla.org/mozilla-central/rev/d2a5bbf7755f

Comment 21

[Pascal Chevrel:pascalc]

Comment on attachment 9375833 [details]
Bug 1865886. Use aarch64 profile generate jobs for shippable builds.

Approved for 123 beta 6, thanks.

(Bug 1875490 landed during the 123 nightly cycle and this won't need an uplift)

Comment 22

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/e47a4b7cf8df
https://hg.mozilla.org/releases/mozilla-beta/rev/cf0204dbf3fa
https://hg.mozilla.org/releases/mozilla-beta/rev/c468ee129f96
https://hg.mozilla.org/releases/mozilla-beta/rev/03c972962a02
https://hg.mozilla.org/releases/mozilla-beta/rev/4c0f3596addd

Comment 23

[Pascal Chevrel:pascalc]

https://hg.mozilla.org/releases/mozilla-beta/rev/e47a4b7cf8df
https://hg.mozilla.org/releases/mozilla-beta/rev/cf0204dbf3fa
https://hg.mozilla.org/releases/mozilla-beta/rev/c468ee129f96
https://hg.mozilla.org/releases/mozilla-beta/rev/03c972962a02
https://hg.mozilla.org/releases/mozilla-beta/rev/4c0f3596addd

Comment 24

[Alex Finder]

(In reply to Pulsebot from comment #17)

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d2a5bbf7755f
Fix merge with changes from bug 1874234. CLOSED TREE

== Change summary for alert #41365 (as of Tue, 06 Feb 2024 17:33:29 GMT) ==

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)	Performance Profiles
15%	cnn-ampstories largestContentfulPaint	android-hw-a51-11-0-aarch64-shippable-qr	cold webrender	1,448.58 -> 1,226.73	Before/After
4%	instagram ContentfulSpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	1,117.23 -> 1,070.02	Before/After
4%	instagram ContentfulSpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	1,122.40 -> 1,078.40	Before/After
3%	instagram SpeedIndex	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	1,406.50 -> 1,367.73	Before/After
2%	espn loadtime	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	918.10 -> 895.78	Before/After
...	...	...	...	...	...
2%	booking LastVisualChange	android-hw-a51-11-0-aarch64-shippable-qr	warm webrender	2,047.40 -> 2,001.91	Before/After

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=41365