1807350 - Use ART AOT baseline profiles to improve startup performance

Status: RESOLVED FIXED

(Fenix :: Performance, enhancement, P1)

Description

[Chris Peterson [:cpeterson]]

From github: https://github.com/mozilla-mobile/fenix/issues/25259.

(Spoiler: other apps saw even 30% - 40% startup improvements)

Baseline profiles

During installation, ART performs Ahead-of-time (AOT) compilation of methods in the profile, resulting in those methods executing faster. If the profile contains methods used in app launch or during frame rendering, the user experiences faster launch times and/or reduced jank.

 

From the cloud.

Android 9 (API level 28) introduced ART optimizing profiles in Play Cloud to improve app startup time and will usually be available the next day after a release.

Disadvantages:

• works only for Android 9+
• app updates clear the Cloud profiles data so with our fast release cycles users might not see big improvements

 

Manually created and shipped with every new release

Advantages:

• will work on Android 7+
• allows to manually tweak the profiles to best suit our needs
• data will continue to be aggregated in the cloud to improve this

Disadvantages:

• creating, tweaking and testing the improvements is a manual process for which Macrobenchmark should be used.

How baseline profiles work and how to create them locally - https://developer.android.com/topic/performance/baselineprofiles#creating-profile-rules

 

Success stories:

https://android-developers.googleblog.com/2022/01/improving-app-performance-with-baseline.html

cc @mcomella

┆Issue is synchronized with this Jira Task

Change performed by the Move to Bugzilla add-on.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:cpeterson, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 2

[Rahul Sainani [:rsainani]]

Been investigating this over the last week or so and have written down my findings in a document here.

Comment 3

[Rahul Sainani [:rsainani]]

Been working on it, have a couple of gradle tasks that can be added in the nightly release pipeline so we can test it out in nightly. Aiming to create a PR next week.

Comment 4

[BMO Github Automation]

Attachment: [mozilla-mobile/firefox-android] Bug 1807350 - Generate baseline profiles for nightly (#1578)

Comment 5

[Chris Peterson [:cpeterson]]

Rahul, your PR mentions arm64-v8a. Are the Baseline Profiles applicable for all Fenix architectures (ARMv7 32, ARM64, x86, and x86-64) or only ARM64?

Also, the Android blog post says: "Baseline Profiles are a new mechanism to provide profiles which can be used on Android 7 (API level 24) and higher." Does that mean the Baseline Profiles are simply unused on Android versions < 7? Or that they are incompatible and we will need separate builds for Android versions < 7? Fenix and Focus still support Android 5 (API level 21).

Comment 6

[Rahul Sainani [:rsainani]]

Hey Chris,

Baseline Profiles are only for Java/Kotlin code so the benefit will be there on all architectures. On the PR, we are just creating an emulator that can be any architecture – arm64 (since our macs are arm64) or x86 (since the CI machines are x86), to generate baseline profiles by running the test on the emulator. Once that's generated, the profiles are packaged with the app during the rest of the build process.

So, on Android 21-23 (Pre 7), android already does Ahead of time compilation for the whole app so baseline profiles are not designed for those as there is no benefit. Baseline Profiles pre compiles the hot methods from the generated profile so android 24 and above can take advantage of mixed - AOT (based on generated profiles) and JIT (usual behaviour).

Why Android moved away from AOT that was there in 21 -23?
You might remember the dreaded "Optimising apps 34 of 246" dialog in those android versions after a system update. That was android compiling the app again, that made installing and updating apps really slow. So this is a solution from android to have the best of both worlds. Hope it answered your questions. Feel free to ask more :)

Also, the resources section in the linked doc has a great podcast which covers the intricacies of different compilations that happen, found it very interesting!

Comment 7

[Herbert Higgins]

(In reply to Rahul Sainani [:rsainani] from comment #2)

Been investigating this over the last week or so and have written down my findings in a document here.

Were you able to gain any improvements in terms performance? The first results from mcomella didn't sound very promising. "However, ignoring the slow outliers on Nightly builds, we don't see a significant performance difference between any of our builds. This implies that, for main first frame start up, baseline profiles aren't significant."

(In reply to Rahul Sainani [:rsainani] from comment #6)

Also, the resources section in the linked doc has a great podcast which covers the intricacies of different compilations that happen, found it very interesting!

Could you please make the most significant parts of the document public? Baseline profiles are an interesting topic for the entire Android app ecosystem and so far there are only Google's numbers published, which seem almost too good to be true.

Comment 8

[Rahul Sainani [:rsainani]]

Thanks for your interest in baseline profiles.

Were you able to gain any improvements in terms performance? The first results from mcomella didn't sound very promising. "However, ignoring the slow outliers on Nightly builds, we don't see a significant performance difference between any of our builds. This implies that, for main first frame start up, baseline profiles aren't significant."

Yes, we did see gains in performance in our testing.

Could you please make the most significant parts of the document public? Baseline profiles are an interesting topic for the entire Android app ecosystem and so far there are only Google's numbers published, which seem almost too good to be true.

Unfortunately, we're not ready to share the document as of now. However, if you're interesting in figuring out the difference in performance with/out baseline profiles, the patch is open on github.

Comment 9

[Rahul Sainani [:rsainani]]

Sharing an update as we've had the chance to work on it for the last few days again.

It has been challenging to generate the profiles on CI. The main blocker has been running the emulator/gradle managed device on the CI. Since the task runs on a docker container, it’s not possible to have hardware accelerated VMs (emulators) run on VMs (docker container).

When the task is run, this error is encountered. Logs here.

Failed 'chattr' for /builds/worker/.android/avd/gradle-managed/dev31_default_x86_64_Pixel_3.avd:
[task 2023-04-21T15:49:50.368Z] -- chattr: Inappropriate ioctl for device while reading flags on /builds/worker/.android/avd/gradle-managed/dev31_default_x86_64_Pixel_3.avd

A couple of strategies that we’ve tried:

1. Breaking the :benchmark:pixel3Api31BenchmarkAndroidTest task to download the gradle managed device, launch it with hardware acceleration mode disabled using adb and then run the baseline profile generator, similar to running test on a connected device. Unfortunately, the same error occurred.

2. Using the flag
   -Pandroid.testoptions.manageddevices.emulator.gpu=swiftshader_indirect
   to disable acceleration as mentioned in the gradle managed devices docs and this google issue. This led to the same issue as earlier, here’s the run log.

So another idea would be to set up and launch the emulator with the -no-snapshot flag. This is what approach 1 is doing but the error occurs before that step is reached. This led us to digging a bit deeper into what the gradle managed device setup task is doing, it actually tries to create a zero snapshot state after initialization. That’s why in both the strategies, we get the same error as the common element is ./gradlew:pixel3Api31Setup

Execution failed for task ':benchmark:pixel3Api31Setup'.
A failure occurred while executing com.android.build.gradle.internal.tasks.ManagedDeviceSetupTask$ManagedDeviceSetupRunnable
java.lang.IllegalStateException: Gradle was not able to complete device setup for: dev31_default_x86_64_Pixel_3
The emulator failed to open the managed device to generate the snapshot. This is because the emulator closed unexpectedly, try updating the emulator and ensure a device can be run from Android Studio.

There are a few possible approaches to move forward, will update as we learn more.

Comment 10

[Marc Leclair [:mleclair]]

The Performance Impact Calculator has determined this bug's performance impact to be medium. If you'd like to request re-triage, you can reset the Performance Impact flag to "?" or needinfo the triage sheriff.

Platforms: Android
Impact on browser: Causes noticeable startup delay
[x] Able to reproduce locally

Comment 11

[BMO Github Automation]

Attachment: [mozilla-mobile/firefox-android] Bug 1807350 - Baseline profiles for nightly (#5089)

Comment 12

[BMO Github Automation]

Authored by https://github.com/rahulsainani
https://github.com/mozilla-mobile/firefox-android/commit/51f88bd6180d80fee4cc69d56607269455cf831c
[main] Bug 1807350 - Baseline profiles for nightly