1982921 - Add a universal APK for Fenix to the release archive

Status: RESOLVED FIXED

(Firefox Build System :: Android Studio and Gradle Integration, task)

Description

[John Oberhauser]

We currently upload 4 APKs for specific architectures and an AAB to the Mozilla archive
https://archive.mozilla.org/pub/fenix/releases/

I'm working with an app distributor and they need a universal APK that will work on any architecture. Adding a universal APK to the archive whenever there is a new release would be ideal.

Comment 1

[Ryan VanderMeulen [:RyanVM]]

We can pretty trivially add universalApk true to Fenix' build.gradle, but getting that wired up through the rest of the build system is a bit trickier because universal APKs violate some assumptions we have around things like versionCode setting and artifacts that need uploading by Taskcluster.

I'm starting to poke at this on Try.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Also, from my Try pushes, universal APKs are around 330MB - very similar in size to app bundles. Is that going to be a concern from a storage cost standpoint if we start adding these to every release we upload to archive.m.o?

Comment 3

[Ryan VanderMeulen [:RyanVM]]

Right now, the code I'm writing for this uses the same versionCode for both the universal APK and the app bundle. It makes the logic a lot simpler to implement and I think it's a safe thing to do as we're not intending to push these APKs to any app stores anyway (and if we did, I can't foresee any circumstance where we'd be uploading both a universal APK and an app bundle for the same product - it wouldn't make any sense to do so).

Nick, do you see any potential problems with that? I'll upload the patch shortly if you want to see what this looks like in practice too.

Comment 4

[Ryan VanderMeulen [:RyanVM]]

Attachment: Bug 1982921 - Create universal Fenix APKs during the build. r=#android-reviewers

Comment 5

[Ryan VanderMeulen [:RyanVM]]

Try push with some extra logging included to show the versionCodes in the various build logs.
https://treeherder.mozilla.org/jobs?repo=try&group_state=expanded&revision=47d31e44ad34d586d08e6188742d037850b7aa2d

Comment 6

[Johan Lorenzo [:jlorenzo]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #2)

Also, from my Try pushes, universal APKs are around 330MB - very similar in size to app bundles. Is that going to be a concern from a storage cost standpoint if we start adding these to every release we upload to archive.m.o?

Given that's a binary that we ship to partners (like partner repacks), I believe we need to have universal APKs on archive.m.o. Maybe we don't want them uploaded on nightly or beta but that'll prevent from finding regression windows. I'd say, let's make a decision after a year or so. This way, we can assess how often we needed these universal binary.

How does that sound?

Comment 7

[Ryan VanderMeulen [:RyanVM]]

Patch works when I trigger beetmover jobs on my Try push to push to stage:
https://ftp.stage.mozaws.net/pub/fenix/candidates/143.0a1-candidates/build1/android/fenix-143.0a1-android/

Comment 8

[Nick Alexander :nalexander [he/him]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #3)

Right now, the code I'm writing for this uses the same versionCode for both the universal APK and the app bundle. It makes the logic a lot simpler to implement and I think it's a safe thing to do as we're not intending to push these APKs to any app stores anyway (and if we did, I can't foresee any circumstance where we'd be uploading both a universal APK and an app bundle for the same product - it wouldn't make any sense to do so).

Nick, do you see any potential problems with that? I'll upload the patch shortly if you want to see what this looks like in practice too.

No. I think we'd just want to use the lowest versionCode -- i.e., not setting 64-bit or x86 bits -- for these universal APKs. I'll read the patch now.

Comment 9

[Ryan VanderMeulen [:RyanVM]]

Attachment: Bug 1982921 - Publish universal APKs to archive.mozilla.org. r=#taskgraph-reviewers

Comment 10

[Ryan VanderMeulen [:RyanVM]]

I'm splitting out the publishing part to a separate patch so we can test these APKs out a bit before we start pushing them to archive.mozilla.org.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/e1f49e7dc9c6
https://hg.mozilla.org/integration/autoland/rev/8142eb1ec05d
Create universal Fenix APKs during the build. r=android-reviewers,taskgraph-reviewers,hneiva,nalexander

Comment 12

[pstanciu]

https://hg.mozilla.org/mozilla-central/rev/8142eb1ec05d

Comment 13

[Ryan VanderMeulen [:RyanVM]]

For round 1 of my testing, I was able to manually install the signed universal APK of this morning's Nightly build onto my phone over top of my existing Google Play install and everything appears to be working fine afterwards. Next I want to confirm that this afternoon's Nightly build (which should be the next one to be submitted to the Play Store) is able to update via Google Play over top of that tomorrow after it gets through review.

If both of those steps work, I think that should give us enough confidence that these won't cause issues to move forward with publishing the APKs to archive.m.o and uplifting to Beta & Release and unblocking the partner work.

Comment 14

[Ryan VanderMeulen [:RyanVM]]

One issue this patch brings to light is that it brings the total size of APKs generated by each build job to over 900MB in total, which is getting awful close to autograph's 1GB size limit. We actually went over this temporarily after bug 1981499 landed on autoland and bloated the APK sizes even further, causing Fenix signing jobs to fail.

2025-08-20 01:20:19,655 - signingscript.sign - ERROR - Autograph response: 400, request exceeds max size of 1GB

Julien, do you have thoughts on what our options are for this? Presumably it's only a matter of time until this becomes a problem even if we do manage to fix the bloat issue in bug 1981499. I'm guessing that increasing the limit on the autograph end isn't a preferable choice here. I guess splitting the universal APK into its own job is another option, but that seems like it'll make this more complicated overall.

Though I guess one mitigating factor is that we're probably going to be dropping 32-bit x86 build variants soon, so that'll buy us ~200MB.

Comment 15

[Julien Cristau [:jcristau]]

From https://firefoxci.taskcluster-artifacts.net/Hl3RBNUVRe-Vx6hN6Fg-LA/0/public/logs/live_backing.log:

2025-08-20 00:59:12,363 - signingscript.task - INFO - sign(): Signing 798765919 bytes in /app/workdir/public/build/target.universal.apk with gcp_prod_autograph_apk...

The file then gets base64 encoded in the request to autograph, which adds ~33% and makes it too large.

The limit comes from https://github.com/mozilla-services/autograph/blob/fbc8e43747ac776ba949d002bc53bc6677011d7f/handlers.go#L129-133

Keeping the NI as I think about this...

Comment 16

[Ryan VanderMeulen [:RyanVM]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #13)

Next I want to confirm that this afternoon's Nightly build (which should be the next one to be submitted to the Play Store) is able to update via Google Play over top of that tomorrow after it gets through review.

Confirming that I was successfully updated from Google Play over top of my universal APK install with the latest Nightly build. I think we're good here! I'm going to land the patch to start publishing to archive.m.o.

(In reply to Julien Cristau [:jcristau] from comment #15)

Keeping the NI as I think about this...

Maybe less important now that we realized the root cause of the APK size bloat and worked around it at the source.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/27b9ebee45dc
https://hg.mozilla.org/integration/autoland/rev/546f5600b60c
Publish universal APKs to archive.mozilla.org. r=releng-reviewers,jcristau

Comment 18

[Ryan VanderMeulen [:RyanVM]]

Attachment: Bug 1982921 - Create and publish universal Fenix APKs during the build.

Comment 19

[Ryan VanderMeulen [:RyanVM]]

Attachment: Bug 1982921 - Create and publish universal Fenix APKs during the build.

Comment 20

[Phabricator Automation]

firefox-beta Uplift Approval Request

• User impact if declined: None, these aren't builds we ship to users
• Code covered by automated testing: no
• Fix verified in Nightly: yes
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: n/a
• Risk associated with taking this patch: Low
• Explanation of risk level: Doesn't affect builds we ship to the stores
• String changes made/needed: none
• Is Android affected?: yes

Comment 21

[Phabricator Automation]

firefox-release Uplift Approval Request

• User impact if declined: None, these aren't builds we ship to users
• Code covered by automated testing: no
• Fix verified in Nightly: yes
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: n/a
• Risk associated with taking this patch: Low
• Explanation of risk level: Doesn't affect builds we ship to the stores
• String changes made/needed: none
• Is Android affected?: yes

Comment 22

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/546f5600b60c

Comment 23

[Pulsebot]

https://github.com/mozilla-firefox/firefox/commit/6e62ac952922
https://hg.mozilla.org/releases/mozilla-beta/rev/1075c81389f0

Comment 24

[Pulsebot]

https://github.com/mozilla-firefox/firefox/commit/61104afa349c
https://hg.mozilla.org/releases/mozilla-release/rev/832d74eb4a2f

Comment 25

[Julien Cristau [:jcristau]]

(In reply to Julien Cristau [:jcristau] from comment #15)

Keeping the NI as I think about this...

Maybe less important now that we realized the root cause of the APK size bloat and worked around it at the source.

Alright, let's punt then :)