1969143 - upgrade android emulator to newer version as Mozilla will not support android 7.0

Status: RESOLVED FIXED

(Testing :: General, task)

Description

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

there is a desire in Q3 to upgrade the minimum android supported version. This might be simple from a build perspective, but the large majority of our automation runs on android emulators which is android_version 7.0.

There are a lot of unknowns here, and while this could be simple, we don't know the complexity of the project until we get started.

:aerickson and I have roughly estimated it would take 1 month of engineering effort to upgrade this (we both have no experience working with the android emulators). Factors to consider:

• base OS revision (currently ubuntu 18.04) - this has libraries, kernel versions, etc.; we are working to get Ubuntu 24.04 available for unittests.
• emulation software (I assume there are newer versions, maybe other flags/config options)
• Android version. We need a decision here, do we need the minimum version, or something more modern like 15
• other ancillary tools; python version, scripts, etc. (minor concern, but we shouldn't assume there are no issues)
• tests working on new base os/emulator/tools/android_version. It often takes 1 month just to get tests annotated and odd stuff worked out. Tests might run slower on a new system, have focus issues, file paths might have changed, permissions need adjustment, etc.;

We should outline in this bug what specific versions of tools we want and timelines/engineers working on it.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

:aerickson, :nalexander - do either of you have history in the emulator software, requirements for the host container or machine, or requirements you would like to see in the new upgraded emulator version.

Comment 2

[Andrew Erickson [:aerickson]]

When I started at Mozilla, I took over the manual process of creating Android emulator packages for gbrown. The directions he provided are at https://mozilla-hub.atlassian.net/wiki/spaces/ROPS/pages/26477052/Android+Build+and+Test+Infrastructure#AndroidBuildandTestInfrastructure-AVDEmulators/ImagesToolToolPackages.

A few years ago emulator and SDK packaging was fully automated and moved in tree via https://bugzilla.mozilla.org/show_bug.cgi?id=1624649 and https://bugzilla.mozilla.org/show_bug.cgi?id=1718341.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

I am going to upgrade the base OS to 24.04 in June, this is part of upgrading our overall linux story.

another question- :amoya, will we need to change this on all our ESR branches (140 will be around for another year, 115 is around to keep support for windows7 and is scheduled for EOL September, I suspect we will extend it another 6+ months)?

Comment 4

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

scratch that, :jcristau pointed out that we don't do android tests on esr

Comment 5

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

it appears we have android 12 (api 31) as an existing avd: https://searchfox.org/mozilla-central/source/taskcluster/kinds/toolchain/android.yml (thanks to bug 1865886)

and a patch :jcristau did last summer to try and use that: https://hg-edge.mozilla.org/try/rev/75b8cfa798c8e4134a55488c25310fbc087e7075

:amoya, are there conditions around what version we should be running? minimum supported (soon to be 8?), or modern (like Android 15?)

Comment 6

[Nick Alexander :nalexander [he/him]]

(In reply to Joel Maher ( :jmaher ) (UTC -8) from comment #1)

:aerickson, :nalexander - do either of you have history in the emulator software, requirements for the host container or machine, or requirements you would like to see in the new upgraded emulator version.

I'm going to redirect to Ted, who spent quite a bit of time thinking about this and hopefully can point you at his thoughts.

Comment 7

[Amedyne Moya [:amoya]]

(In reply to Joel Maher ( :jmaher ) (UTC -8) from comment #5)

it appears we have android 12 (api 31) as an existing avd: https://searchfox.org/mozilla-central/source/taskcluster/kinds/toolchain/android.yml (thanks to bug 1865886)

and a patch :jcristau did last summer to try and use that: https://hg-edge.mozilla.org/try/rev/75b8cfa798c8e4134a55488c25310fbc087e7075

:amoya, are there conditions around what version we should be running? minimum supported (soon to be 8?), or modern (like Android 15?)

Joel, We'll be moving to a minimum supported version of Android 8 by October, so at the very least, we need to ensure compatibility of our tests for users on that version and above. Since we're in a position to upgrade our automation setup, I won't be prescriptive about which newer Android versions to use beyond that baseline—especially if running on later versions helps us prepare for upcoming targetSdkVersion changes or platform behaviors.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

browser usage by mobile version:
https://mozilla.cloud.looker.com/explore/combined_browser_metrics/active_users_aggregates?qid=eDq2fanPmUbPy198gHJeVd&origin_space=418&toggle=vis

not much older stuff, but given we run Android 7, upgrading to 12 is 5 years newer, it seems that 13 or 14 would be ideal. Theoretically we could be more proactive and plan to upgrade yearly (or every other year).

We get perf and some unittest coverage on Android 15 (hardware), so picking something on the lower end seems realistic for the emulator. I am not sure if there were features that we need to support on a lower version which needs to be considered.

Comment 9

[Polly [:polly]]

hello!
unless there are specific problems with doing so that would slow us down in achieving this goal - for CI emulators, it makes sense to me to upgrade to as recent a version of Android as we can.
So I agree that 13 or 14 would be great to use!
I can see a few advantages to this:

• more users are on recent builds, so we are more likely to spot issues impacting a large percentage of users
• tests will run faster so builds will be quicker
• we will have more time before we need to do the next upgrade.
  If there are currently older android version specific issues being covered by automated tests, that probably needs more thinking about - if they are still giving us value, maybe those tests could be separated out into a different suite that is run separately from the CI build.

Comment 10

[Ted Campbell [:tcampbell]]

Current versions:

• Physical Phones:
  • A55 and S24: Android 14 (API 34)
  • Pixel6: Android 13 (API 33)
• Default ./mach android-emulator
  • x86_64: Android 7 (API 24)
  • Apple Silicon: Android 11 (API 30)
• UI Tests (Google Firebase Test Lab)
  • android-components: Android 9 (API 28)
  • legacy api tests: API 26, 28, 30, 32
  • robo test (chaos testing): API 30, 31, 32, 34
  • baseline profile: Android 13 (API 33)
  • general: Android 11 (API 30) || Android 14 (API 34)
• GeckoView PGO
  • ARM32: Android 11 (API 30)
  • ARM64: Android 12.0 (API 31)
• Platform Tests
  • Android 7 (API 24)

Note: Current minimum version is Android 5.0 (API 21) which we have zero CI coverage for.

At the very least it should be Android 10.0 (API 29) since we see noticeable changes with Scope Storage and some other security changes. Having tests, scripts, tools work at that version makes local work more feasible.

I am leaning towards Android 11 (API 30) to replace the platform default since we have local Mac devs defaulting to API 30 already, and a bunch of UI tests as well, it seem tractable to get working. I think the jump from API 30 to 34 will be smaller in the future if we desire that.

Related to this I want to have multiple AVD configs in tree so that it is easier to local devs to switch among verions. For local devs, I also have a preference to use google-api instead of aosp images by default. They behave more like a real phone, and default to being unrooted (with an easy toggle if needed) so we avoid building bad habits that make it harder to test things on real phones.

Having the option to target try and central with minimum version test jobs might also be nice, but I don't think we want minVer as the first line of testing.

Comment 11

[Ted Campbell [:tcampbell]]

Joel, something else I'd like to do as part of this is fix up the task names and treeherder names. Having Android 5.0 displayed in places still is a bit silly. My thoughts would be to just have Android for whatever default emulator version we settle on, and then can later consider having Android MinSdk and Android TargetSdk if we want. Having the version number in name and changing it seems more problematic to treeherder/perfherder data than just having baseline expectation changes when we do version updates (similar to other infra changes).

Comment 12

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

so I think the actionable items to move forward:

• create android 10.0 (api 29) AVD in-tree
• create android 11.0 (api 30) AVD in-tree

I suspect given support for those we would need to hack up scripts for local testing to add commands for specifying a specific AVD. One concern with this is that for automation we will only run 1 version of the OS at a time on the emulators (i.e. Android 11). So any tests annotations that exist in mozilla-central will be specific to Android 11.

The request for minsdk/targetsdk- I might not understand fully. For what we run in CI, this is something we need to setup, get tests running, and live with for a year or two. As tests change frequently and OS features change, to keep CI the most effective it is best to have a specific OS version and upgrade at specific points in time.

Ideally we can upgrade 1x/year going forward.

Comment 13

[:gerard-majax]

(In reply to Ted Campbell [:tcampbell] from comment #10)

[...]

Related to this I want to have multiple AVD configs in tree so that it is easier to local devs to switch among verions. For local devs, I also have a preference to use google-api instead of aosp images by default. They behave more like a real phone, and default to being unrooted (with an easy toggle if needed) so we avoid building bad habits that make it harder to test things on real phones.

While working on something related to Remote Settings, I had to do exactly those steps because sideloading an XPI (remote settings devtools here) would require something more recent than Android 7, I think at least 10 or 11. Hence bug 1924749 and bug 1924484

Comment 14

[Paul Adenot (:padenot)]

In the interest of (In reply to Joel Maher ( :jmaher ) (UTC -8) from comment #8)

browser usage by mobile version:
https://mozilla.cloud.looker.com/explore/combined_browser_metrics/active_users_aggregates?qid=eDq2fanPmUbPy198gHJeVd&origin_space=418&toggle=vis

https://sql.telemetry.mozilla.org/dashboard/fenix-app-and-os-version paints a more useful picture of the situation, that allows understanding trend lines and such. https://apilevels.com/ paints the other side of the picture (fenix users vs. android users). I wanted a cumulative android version plot, but redash died on me, sorry.

For example, high end Samsung device users are disproportionally represented in our users (vs. all android users), and samsung started offering Android 15 a few weeks back, this is the kind of dynamics that can inform our decisions here.

Comment 15

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

Assuming we keep the emulator version for 2 years (hopefully only 1 year), I would like to ensure that the emulator meets our needs. For example, we don't test on windows 7 despite tens of thousands of users on that platform. The risk is if Android deprecated some stuff and we are developing and running CI on the newer versions, we could break things on the old version.

50% of our users are on android 14+15. I imagine 6 months from now that number will be >60%.

Comment 16

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

I have a try push with android 30. When we have a decision on what to run, I can add the toolchain tasks and then start greening things up.

Comment 17

[Ted Campbell [:tcampbell]]

Here was my API 30 try push which was a lot more Orange than I expected. https://treeherder.mozilla.org/jobs?repo=try&group_state=expanded&revision=52adcc7e05540377549e5fc3d58f81a6a8d3f6d6

Comment 18

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

that is greener than I would expect for an upgrade. I have tools to get the jobs greener (by editing manifests)- Assuming we have consensus on s/7.0/11.0/

Comment 19

[Julien Cristau [:jcristau]]

I've got a try push with android 15 (api 35) at https://treeherder.mozilla.org/jobs?repo=try&revision=933833b0d4089a50281607e9b70cc005d17b9c38 FWIW.
(It'd probably be less orange if I carried forward some of the existing android_version == 24 skip-if annotations, and if I had pushed with --no-artifact, but I thought I'd start simple.)

Comment 20

[Ted Campbell [:tcampbell]]

Since 11.0 is already five years old and the majority of users are newer, it probably is worth pushing forward to something newer.

We have been running API34 (Android 14) on Firebase for a while recently, and most of our perf test device fleet is on API34 as well. So I'm tempted to suggest we go with Android 14.0 for this round of updates.

I could see arguments for jumping to Android 15 since it is the fastest growing segment at the moment now that 15.0 updates are becoming available for phones.

Are folks okay with aiming for API 34 / Android 14.0 for this round?

Comment 21

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

I am fine with anything; This seems reasonable. I am sure by the end of the year more older versions will drop off. Will wait for some others to chime in.

Comment 22

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

no push back, shall we move forward with API34/Android 14.0 ?

Comment 23

[Ted Campbell [:tcampbell]]

Agree. Let's aim for Android 14.0 here

Comment 24

[Titouan Thibaud [:titouan]]

(In reply to Ted Campbell [:tcampbell] from comment #23)

Agree. Let's aim for Android 14.0 here

Just to make it clear in case it wasn't yet, we're not dropping support for Android versions under 14, we're only aiming for Android 14.0 for the emulators we'll be testing on.

Comment 25

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

Attachment: Bug 1969143 - Add support for Android 14 (api 34) emulator. r=tcampbell

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/d4858f917723
https://hg.mozilla.org/integration/autoland/rev/d7158d49cde5
Add support for Android 14 (api 34) emulator. r=tcampbell

Comment 27

[Joel Maher ( :jmaher ) (UTC -8) (PTO back normal Nov 17)]

from my initial list:

• base OS revision (currently ubuntu 18.04) - this has libraries, kernel versions, etc.; we are working to get Ubuntu 24.04 available for unittests.
• emulation software (I assume there are newer versions, maybe other flags/config options)
• Android version. We need a decision here, do we need the minimum version, or something more modern like 15
• other ancillary tools; python version, scripts, etc. (minor concern, but we shouldn't assume there are no issues)
• tests working on new base os/emulator/tools/android_version. It often takes 1 month just to get tests annotated and odd stuff worked out. Tests might run slower on a new system, have focus issues, file paths might have changed, permissions need adjustment, etc.;

I don't think the emulation software needs an upgrade, so this will end up being the last 2 items on the list.

Comment 28

[Ted Campbell [:tcampbell]]

For the actual emulator (QEMU-derived), CI is using 32.1.15 (released Aug 23 ) while the latest stable offered from Android SDK is 35.5.10. The existing emulator is from around the time of Android 14.0 so I expect it works fine for our needs for now.

Eventually it might be nice to update emulator version in case more recent crash fixes help with some of our stability issues, that but that is out of scope for this CI system image update.

Comment 29

[agoloman]

https://hg.mozilla.org/mozilla-central/rev/d7158d49cde5