1529082 - Support dynamic delivery / Android App Bundle

Status: RESOLVED FIXED

(GeckoView :: General, enhancement, P1)

Description

[Nevin Chen(Not active on Bugzilla)]

Dynamic delivery empowers the users to have a smaller first install APK size, and then install the dynamic features as extra APKs if they want to use that feature. see AAB and Dynamic delivery

The user will first get the base.apk, then download splits_feature_module_name.apk from Play Store

Our current GeckoView code (e.g. NSSBridge / GeckoThread / GeckoJarReader) will need the apk path to load related libraries, but this won't work dynamic delivery world cause the APK name will change according to abi/laguage/density splits.

Here's one example of only splits the abi with one dynamic feature

"/data/app/org.mozilla.rocket.debug.firebase-n8F5hLsnz5J5dNOuixTgDQ==/base.apk"
"/data/app/org.mozilla.rocket.debug.firebase-n8F5hLsnz5J5dNOuixTgDQ==/split_geckopower.apk"
"/data/app/org.mozilla.rocket.debug.firebase-n8F5hLsnz5J5dNOuixTgDQ==/split_geckopower.config.x86.apk"

I suggest to replace context.getPackageResourcePath() to context.applicationInfo.splitSourceDirs and context.applicationInfo.publicSourceDir to get both base and split APKs.

Every Java class has it's ClassLoader(Android's PathClassLoader) and has its DexPathList contains all the dex files, but I can't find an api to get them

DexPathList[[zip file "/data/app/org.mozilla.rocket.debug.firebase-n8F5hLsnz5J5dNOuixTgDQ==/base.apk", zip file "/data/app/org.mozilla.rocket.debug.firebase-n8F5hLsnz5J5dNOuixTgDQ==/split_geckopower.apk", zip file "/data/app/org.mozilla.rocket.debug.firebase-n8F5hLsnz5J5dNOuixTgDQ==/split_geckopower.config.x86.apk"],nativeLibraryDirectories=[/data/app/org.mozilla.rocket.debug.firebase-n8F5hLsnz5J5dNOuixTgDQ==/lib/x86, /data/app/org.mozilla.rocket.debug.firebase-n8F5hLsnz5J5dNOuixTgDQ==/base.apk!/lib/x86, /data/app/org.mozilla.rocket.debug.firebase-n8F5hLsnz5J5dNOuixTgDQ==/split_geckopower.apk!/lib/x86, /data/app/org.mozilla.rocket.debug.firebase-n8F5hLsnz5J5dNOuixTgDQ==/split_geckopower.config.x86.apk!/lib/x86, /system/lib]]

Currently if we really want to work around this, we need to have a our own NSSBridge/GeckoThread/GeckoJarReader implementation and use it like this . But this approach isn't maintanable.

It'll be great to have dynamic delivery support in GeckoView.

Comment 1

[Chris Peterson [:cpeterson]]

App Bundles might be a good solution for bug 1548679.

Comment 2

[[ex-Mozilla] Agi Sferro | :agi]

Resetting priority so we can re-prioritize this. This is the only blocker for shipping GV as an app bundle.

Comment 3

[David Bolter [:davidb] (NeedInfo me for attention)]

Needs a priority. NI'ing Andreas to help.

Comment 4

[Chris Peterson [:cpeterson]]

Firefox Lite would like this feature.

Comment 5

[Emily Toop (:fluffyemily)]

How soon is Firefox Lite wanting to have this? None of our other consumers (FxR, Fenix) are asking for this yet, and it would be good to know so that we can properly prioritise.

Comment 6

[Joe Cheng [:jcheng] (please needinfo)]

Hi Emily, the Firefox Lite team would like to have this feature available from your team in Q4 so we can work on it and deliver features based on it in Q1 2020. Thanks

Comment 7

[Chris Peterson [:cpeterson]]

I'm editing a bunch of GeckoView bugs. If you'd like to filter all this bugmail, search and destroy emails containing this UUID:

e88a5094-0fc0-4b7c-b7c5-aef00a11dbc9

Comment 8

[Andreas Bovens [:abovens]]

Hey Joe, can you create a 1-pager with justification and rationale? That will help us with prioritization.

Comment 10

[Joe Cheng [:jcheng] (please needinfo)]

Here it is https://docs.google.com/document/d/1KR0lEc5ieA7MEL9jfdLxoL_b6FpEVDN5aKr0Bx-uIKE/edit

Comment 11

[widevel.com]

Still not working in 80.0.20200903001900 stable version

Comment 12

[Bob]

This issue makes it impossible for Geckoview to be used in new projects since App Bundles are now (August 2021) required for new Android Apps (see https://android-developers.googleblog.com/2021/06/the-future-of-android-app-bundles-is.html). Meaning the projects using Geckoview can only decrease from here.

Comment 14

[[ex-Mozilla] Agi Sferro | :agi]

Attachment: Bug 1529082 - Locate lib folder within the APK.

When using App Bundles, the so files will be in the APK splits, instead of
being in the folder specified by the native lib.

dlopen and dlsym do the right thing on Android and automatically figure out
where the library is.

I'm not sure if this is a good approach (e.g. is it safe? could another app
potentially hijack our libraries?). If it isn't we could find the right path
using some code that chromium has.

Comment 15

[[ex-Mozilla] Agi Sferro | :agi]

Attachment: Bug 1529082 - Locate lib folder within the APK.

Before this patch, we would load mozglue using the unqualified
System.loadLibrary("mozglue") and then we would try to load all the other
native libraries from the path returned from nativeLibraryDir.

Note that since the path is unqualified, we might end up loading a library
that's not inside the APK, potentially a malicious one.

Android App Bundles (or AAB) is a new feature of Android where apps can be
split in multiple APKs (called "splits"). Each APK split contains a functinal
part of the app, e.g. a specific resolution asset folder, all the native
libraries for one CPU architecture etc.

For AAB, nativeLibraryDir returns an empty folder as the libraries are
stored, uncompressed, inside one of the APK splits, so GeckoView fails to load
libxul and other libraries when starting up.

The current code also doesn't work for any case where System.loadLibrary
fails to load mozglue (we do have code that accounts for that, but I'm not sure
how much of it is funcitonal).

To fix that, we locate the mozglue using code ported from Chromium and use that
to locate all the other native libraries.

Note this still doesn't fix the problem of loading an unqualified mozglue
library, but there's not a lot we can do about that right now (and that's not a
new problem, so we can address it in a separate patch).

Comment 16

[[ex-Mozilla] Agi Sferro | :agi]

Attachment: Bug 1529082 - Add bundletool to Android's bootstrap

This tool is used to install AAB packages to devices.

AAB is the new package format for Android Apps and will supersede APK.

Comment 17

[[ex-Mozilla] Agi Sferro | :agi]

Attachment: Bug 1529082 - Add device.install_app_bundle

Comment 18

[[ex-Mozilla] Agi Sferro | :agi]

Attachment: Bug 1529082 - Move TestRunnerActivity to test_runner.

The fact that the test runner app is defined inside the geckoview test package
has always felt like a hack to me. I've mistakenly thought that
TestRunnerActivity was used in GeckoView's junit tests many times (even though
that's not the case).

From what I can see, there's no way to generate an AAB package for androidTest,
so to be able to run Gecko tests as AAB we finally need to define the
TestRunner as an ordinary package instead.

Comment 19

[[ex-Mozilla] Agi Sferro | :agi]

Attachment: Bug 1529082 - Run set_target after set_variant.

This allows variants to override targets.

Comment 20

[[ex-Mozilla] Agi Sferro | :agi]

Attachment: Bug 1529082 - Use bundletool in CI to install AAB targets.

Comment 21

[[ex-Mozilla] Agi Sferro | :agi]

Attachment: Bug 1529082 - Use bundletool to install apps as aab packages.

This patch adds a new command line argument --aab which allows users to install
GVE as an AAB.

This will also be used in a future patch to install the test runner as AAB.

Comment 22

[[ex-Mozilla] Agi Sferro | :agi]

Attachment: Bug 1529082 - Add mochitest variant that runs using AAB test_runner.

This patch adds a new Mochitest variant which installs the test runner app as
AAB.

Comment 23

[Rahul Shyokand]

Is there any production-ready solution for uploading GeckoView based .aab android app to Google PlayStore without getting the following crash?

021-10-26 22:09:37.959 32298-3601/? E/AndroidRuntime: FATAL EXCEPTION: Gecko
    Process: com.example.geckoviewapp, PID: 32298
    java.lang.Exception: Error loading sqlite libraries

Since other variants of Mozilla Browser also use GeckoView and they update the Google Playstore App regularly without getting this crash.
How can we build .aab in a way that it is distributable through Google Playstore without Crash?

This may be a serious concern for developers who used GeckoView in their App and now are unable to publish the app to Google Playstore.
Your help and guidance will be appreciated.

Comment 24

[[ex-Mozilla] Agi Sferro | :agi]

(In reply to Rahul Shyokand from comment #23)

Is there any production-ready solution for uploading GeckoView based .aab android app to Google PlayStore without getting the following crash?

021-10-26 22:09:37.959 32298-3601/? E/AndroidRuntime: FATAL EXCEPTION: Gecko
    Process: com.example.geckoviewapp, PID: 32298
    java.lang.Exception: Error loading sqlite libraries

Since other variants of Mozilla Browser also use GeckoView and they update the Google Playstore App regularly without getting this crash.
How can we build .aab in a way that it is distributable through Google Playstore without Crash?

This may be a serious concern for developers who used GeckoView in their App and now are unable to publish the app to Google Playstore.
Your help and guidance will be appreciated.

Hi there, the fix will be available in GeckoView 96. Meanwhile you should be able to set extractNativeLibs = true in AndroidManifest.xml and android.bundle.enableUncompressedNativeLibs = false in gradle.properties, see the following:

https://developer.android.com/guide/topics/manifest/application-element#extractNativeLibs
https://developer.android.com/studio/releases/gradle-plugin#behavior-changes

Comment 25

[[ex-Mozilla] Agi Sferro | :agi]

Attachment: Bug 1529082 - Resolve keyed target in get_mobile_project.

Comment 26

[[ex-Mozilla] Agi Sferro | :agi]

Attachment: Bug 1529082 - Fix formatting in wpt/run.py.

./mach lint --outgoing --fix keeps changing this for me.

Comment 27

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/94012b2abf6f
Locate lib folder within the APK. r=owlish
https://hg.mozilla.org/integration/autoland/rev/67001525a262
Add bundletool to Android's bootstrap r=jmaher
https://hg.mozilla.org/integration/autoland/rev/28475cfdf30b
Add device.install_app_bundle r=jmaher
https://hg.mozilla.org/integration/autoland/rev/c03034fa7d47
Move TestRunnerActivity to test_runner. r=ahal,owlish
https://hg.mozilla.org/integration/autoland/rev/86c4c2070378
Resolve keyed target in get_mobile_project. r=jmaher
https://hg.mozilla.org/integration/autoland/rev/69bd7293edb8
Use bundletool in CI to install AAB targets. r=ahal
https://hg.mozilla.org/integration/autoland/rev/f7888063e67e
Use bundletool to install apps as aab packages. r=ahal,owlish
https://hg.mozilla.org/integration/autoland/rev/44a00ee73528
Add mochitest variant that runs using AAB test_runner. r=gbrown

Comment 28

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/31475 for changes under testing/web-platform/tests

Comment 29

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/18607955e8b2
Fix formatting in wpt/run.py. r=jmaher

Comment 30

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/94012b2abf6f
https://hg.mozilla.org/mozilla-central/rev/67001525a262
https://hg.mozilla.org/mozilla-central/rev/28475cfdf30b
https://hg.mozilla.org/mozilla-central/rev/c03034fa7d47
https://hg.mozilla.org/mozilla-central/rev/86c4c2070378
https://hg.mozilla.org/mozilla-central/rev/69bd7293edb8
https://hg.mozilla.org/mozilla-central/rev/f7888063e67e
https://hg.mozilla.org/mozilla-central/rev/44a00ee73528

Comment 31

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/18607955e8b2

Comment 32

[Rahul Shyokand]

(In reply to Agi Sferro | :agi | ni? for questions | ⏰ PST | he/him from comment #24)

(In reply to Rahul Shyokand from comment #23)

Is there any production-ready solution for uploading GeckoView based .aab android app to Google PlayStore without getting the following crash?

021-10-26 22:09:37.959 32298-3601/? E/AndroidRuntime: FATAL EXCEPTION: Gecko
    Process: com.example.geckoviewapp, PID: 32298
    java.lang.Exception: Error loading sqlite libraries

Since other variants of Mozilla Browser also use GeckoView and they update the Google Playstore App regularly without getting this crash.
How can we build .aab in a way that it is distributable through Google Playstore without Crash?

This may be a serious concern for developers who used GeckoView in their App and now are unable to publish the app to Google Playstore.
Your help and guidance will be appreciated.

Hi there, the fix will be available in GeckoView 96. Meanwhile you should be able to set extractNativeLibs = true in AndroidManifest.xml and android.bundle.enableUncompressedNativeLibs = false in gradle.properties, see the following:

https://developer.android.com/guide/topics/manifest/application-element#extractNativeLibs
https://developer.android.com/studio/releases/gradle-plugin#behavior-changes

Thank you very much.
This worked when I added android.bundle.enableUncompressedNativeLibs = false in gradle.properties.
Now Gecko Libs are able to load with a .aab bundle distributed through Google Playstore.

Comment 33

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged by moz-wptsync-bot