Closed Bug 1039789 (splitapk) Opened 6 years ago Closed 9 months ago

[meta] Build and deploy two tiers of APKs for different device capabilities

Categories

(Firefox Build System :: Android Studio and Gradle Integration, defect)

All
Android
defect
Not set

Tracking

(firefox36-, fennec-)

RESOLVED FIXED
Tracking Status
firefox36 - ---
fennec - ---

People

(Reporter: rnewman, Unassigned)

References

(Depends on 1 open bug, Blocks 2 open bugs)

Details

(Keywords: meta)

Attachments

(4 obsolete files)

Facebook ships 15 different APKs to the Play store, with the express goal of shrinking the downloaded APK for each device by eliminating unnecessary resources.

Our current Nightly APK, with 50+ locales, is 29MB.

Our drawable-xhdpi directory alone is 580KB. drawable-hdpi is 604KB. If you're on a low-end phone, you're downloading and storing over 1MB of assets that your device will never use.

Similarly, we shouldn't be shipping tablet resources to, e.g., ARMv6 phones.

We also have code that is runtime conditionalized on Android API levels and features. It might be possible to delete that dead code.


We should consider generating different APKs to match the various filtering capabilities available to us in the Play Store.
Two comments:

* this would also allow us to trim omni.ja for GeckoView, specifically remove fonts;

* this is tricky but already kind-of supported:

We'd need to run |aapt| during the packaging phase for each configuration, but we already basically do that!  We just require that the Android resource set doesn't change between build and package.  We could make the packaging step more clever and allow the resource set to change; or we might get lucky and not even need that.  This would be a fun thing to build, tbh.
Let's do some manual test builds so we can evaluate how things might turn out. There might be some areas that are more high value than others.
I removed a bunch of resources from m/a/b, aiming for an SDK9-only build. I also deleted some consequently-dead code, but had to return a few hdpi resources without which the build failed -- including, curiously, some styles that are API11+, but are required by unqualified layouts. Bizarre.

Performing only a single ProGuard pass:

Pre:
-rw-r--r--     3 rnewman  rnewman  30667581 Jul 16 15:58 fennec-33.0a1.en-US.android-arm.apk

Post:
-rw-r--r--     1 rnewman  rnewman  30300165 Jul 16 19:41 fennec-33.0a1.en-US.android-arm.apk


so that's about 360KB just from stripping out some drawables and a little bit of code.

Doing a more thorough job -- compile-time max-SDK checking instead of our current runtime checking, combined with aggressive ProGuard dead code elimination -- should increase this, as should creating mdpi resources for those xhdpi ones that I had to re-include, and cutting those dependencies where they don't make sense.

We can also get some wins from omitting features that can't possibly work on lower API versions -- e.g., Android Beam.

If I get more free time this week I'll consider the automated approach I outline above; a great deal of this task should be offloadable to ProGuard, which is also able to strip unused resources.
My initial plan of attack for this is to explore having two APKs, splitting at Honeycomb.

The "resource constrained" APK will exclude all tablet code, all tablet resources, all high-DPI resources, Kindle-specific changes, and perhaps even exclude the fonts that we ship (which will save us a few MB alone).

This should be an achievable automation target (basically do this instead of an ARMv6 build), and not an undue burden on the human parts of this machine.

My hope is that if we knock multiple megabytes off the APK size, we'll see a spike of upgrades from old Firefox versions on <11 devices.
Assignee: nobody → rnewman
Status: NEW → ASSIGNED
This is a good starting point, allowing us to both define the correct manifest output and also make static code decisions based on supported version ranges.

Next step is to add an additional flag for resource constraints. With those two we can specify the desired behavior for each APK.

Tested the default build (same as now -- 9+) and with

ac_add_options --enable-android-min-sdk=11
ac_add_options --enable-android-max-sdk=15

which yielded the expected results.
Attachment #8459794 - Flags: review?(nalexander)
Comment on attachment 8459794 [details] [diff] [review]
Part 1: allow minimum and maximum SDK versions to be specified. v1

Review of attachment 8459794 [details] [diff] [review]:
-----------------------------------------------------------------

It's fine by me, but I'm going to defer to glandium because I don't know best practices for adding configure.in --enable flags.

::: configure.in
@@ +4848,5 @@
>  AC_SUBST(MOZ_DBUS_GLIB_CFLAGS)
>  AC_SUBST(MOZ_DBUS_GLIB_LIBS)
>  
>  dnl ========================================================
> +dnl = Allow imposed min and max Android SDK versions

This is tricky 'cuz b2g also builds on Android and has it's own mechanism.  Be clear that this is mobile/android only.  (I've been using MOZ_ANDROID_* to mark that, but it's not explicit.)

::: mobile/android/confvars.sh
@@ +12,5 @@
>  MOZ_OFFICIAL_BRANDING_DIRECTORY=mobile/android/branding/official
>  # MOZ_APP_DISPLAYNAME is set by branding/configure.sh
>  
> +# We support 9 and up by default.
> +# See the --enable-android-min-sdk and --enable-android-max-sdk arguments.

nit: Android SDK version 9.

nit: point at configure.in.
Attachment #8459794 - Flags: review?(nalexander) → review?(mh+mozilla)
Patch for discussion. This is incomplete, but shows the approach.

This builds a small collection of version-related flags inside AppConstants. These will be simplified at compile time, based on the min/max version flags, and ProGuard will eliminate whole clauses as a result.

We can also get some handy simplifications along the way: if an APK is built with a high enough min version, we can do away with runtime checks altogether.

Note that I also began conditionalizing whole features based on known API support -- e.g., MOZ_ANDROID_BEAM is now equivalent to "we built with beaming on, and we're allowing this APK to be installed on ICS+ devices, and this is an ICS+ device". This makes actual feature code much less error prone.
This hits essentially every place we check an SDK_INT value, switching to an approach that allows for compile-time optimization.
Attachment #8459867 - Attachment is obsolete: true
Self-review => finding fencepost errors.
Attachment #8460017 - Attachment is obsolete: true
Broad-spectrum build: 30758189
SDK 9 only:           30723145
SDK 13+ only:         30753333

The big size win will come when we can spot inaccessible resources (Bug 792151).
Alias: splitapk
Blocks: moreapks, 856791
No longer blocks: 856791
Blocks: 1042363
No longer blocks: fatfennec
Depends on: 1042382
Comment on attachment 8459794 [details] [diff] [review]
Part 1: allow minimum and maximum SDK versions to be specified. v1

Morphing this bug to be about splitting the APKs, not the optimization itself.

See Bug 1042383 for that.
Attachment #8459794 - Attachment is obsolete: true
Attachment #8459794 - Flags: review?(mh+mozilla)
Attachment #8460035 - Attachment is obsolete: true
Assignee: rnewman → nobody
Status: ASSIGNED → NEW
Depends on: 1044334
Saving knowledge: aapt has arguments for exactly this.

+               --min-sdk-version 9 \
+               --max-res-version 9 \
Depends on: 1063109
Depends on: 1063643
Depends on: 1063867
Depends on: 1063868
Summary: Build and deploy multiple APKs for different device capabilities → Build and deploy two tiers of APKs for different device capabilities
Depends on: 1063873
Depends on: 1065190
Depends on: 1072423
QA Contact: flaviu.cos
What support are you looking for from the a-team with this change? Is it anticipated that the test manifests will need to be updated? If we hit new failures, is the plan to just disable the newly failing tests? Thanks!
Flags: needinfo?(rnewman)
(In reply to Dan Minor [:dminor] from comment #14)
> What support are you looking for from the a-team with this change? Is it
> anticipated that the test manifests will need to be updated? If we hit new
> failures, is the plan to just disable the newly failing tests? Thanks!

This is mostly releng work: defining a new config and making it build on builders.

As I understand the role of your team, the only change here is that we want our current Android 2.3 tests to run against an 'android-9-constrained' build (which will be a copy of our 'android' config with a few extra lines), not against 'android'.

Then we'll make 'android' 10+ for ARM. (x86 will remain 9+.)

(You'll know if you get it right, because the 2.3 APK will fail to install on 4.0, and the now-10+ APK will fail to install on 2.3.)

If tests fail in new ways, we would need to fix either the test or the code, because it would be a real regression.
Flags: needinfo?(rnewman)
I'll be looking at above said releng work. I should pick this up next week.
Assignee: nobody → jlund
My hope here is that split apks will give us a much better EOL story for mobile versions than we had for 2.2/ArmV6, i.e. olders apks can simply ride the trains to EOL.
Depends on: 1073772
Jordan: I forked the releng parts of this into Bug 1073772, just so we don't get confused by having this meta bug marked as RESOLVED when there are still bits to do.

(Please check that I got the right component for you!)
Assignee: jlund → nobody
Keywords: meta
(In reply to Chris Cooper [:coop] from comment #17)
> My hope here is that split apks will give us a much better EOL story for
> mobile versions than we had for 2.2/ArmV6, i.e. olders apks can simply ride
> the trains to EOL.

Yes, absolutely. We can just leave old APKs in the store, filtered for limited API versions. (In fact, we could do that right now.)

But the topic of what we can do once we don't ship monolithic APKs is an interesting one, so I'll dump some thoughts here.

In cases where we don't want *users* to pay a price -- e.g., shipping fat compatibility libraries that are only needed for certain API ranges -- being able to generate and deploy specific APKs will avoid us having to choose between those two sets of users.

In cases where *we* don't want to pay the price -- e.g., maintaining lots of additional code that we don't/can't test, as with API 8 and 11-12 -- this doesn't really change our calculus.

Indeed, soon after we split these APKs into [9], [10+] we will be faced with the attractive prospect of making that [9], [13+].

10-12 are barely used, buggy, aren't tested, and have a disproportionate support burden (e.g., Bug 1067429, Bug 1019981, Bug 1065531). Trying to reform GeckoPreferences so it contains fewer "here be dragons" essentially requires abandoning 11 & 12, and split APKs allows us to do that more easily.

EOLing those releases could be accomplished by leaving an orphan APK in Play.
Depends on: 1073474
Depends on: 1065494
Depends on: 1078265
Depends on: 1078266
(In reply to Richard Newman [:rnewman] from comment #15)
> This is mostly releng work: defining a new config and making it build on
> builders.

I agree that the implementation is mostly releng work. We should shortly have the ability to automatically publish APKs to Google Play, removing any additional work from relman as well. The team that I expect will be most affected by this change is QE as they will need to test various configurations on an ongoing basis.
Depends on: 1093394
Depends on: 1093708
Depends on: 1093826
No longer depends on: 1093826
Blocks: 1100361
[Tracking Requested - why for this release]: 2.3 users can't install Fennec
tracking-fennec: --- → ?
Tracked in the actual bug instead (bug 1100361)
we don't track meta bugs
tracking-fennec: ? → -
Depends on: 1108456
Depends on: 1108564
Depends on: 1118339
Depends on: 1120762
Depends on: 1122059
QA Contact: flaviu.cos → cristina.madaras
Depends on: 1122145
Depends on: 1135796
Depends on: 1137144
Depends on: 1137586
Depends on: 1137898
Depends on: 1122071
Depends on: 1139567
Depends on: 1150068
Product: Firefox for Android → Firefox Build System
Summary: Build and deploy two tiers of APKs for different device capabilities → [meta] Build and deploy two tiers of APKs for different device capabilities

We both did this and then stopped doing this. It's definitely no longer open!

Status: NEW → RESOLVED
Closed: 9 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.