1139567 - Figure out split apk + single locale stuff for Fennec releases

Status: RESOLVED WONTFIX

(Release Engineering :: Release Automation: Other, defect)

Description

[[e-mail] (:bhearsum)]

Split APK rode to Beta with Gecko 37 but it doesn't look like the release data submission works correctly. We have a single blob that only has api-11 and x86 in it. I'm guessing that we need to have a separate blob for api-9, but I'm not sure. Do we even do single locale repacks for api-9? Should we?

I don't think these updates are fully supported yet, so I've left the beta channel rule pointing at 36.0b10 for the time being.

Comment 1

[Kim Moir [:kmoir] ET]

That's strange.  x86 should not be in it, and I believe we need a separate blob for api-9. Investigating.

Comment 2

[Kim Moir [:kmoir] ET]

I think this is because I didn't add the rules and channels in the release configs for fennec on release because I was waiting for testing in bug 1117831 to confirm they worked in beta.  Will attach patches.

Comment 3

[Kim Moir [:kmoir] ET]

Ben, I added new rules for release on Fennec bug can't see the id in the balrog ui or by viewing source.  And I don't have the credentials to the balrog db. Is there another way to see the ids of the rules I just added so I can create patches?

Comment 4

[[e-mail] (:bhearsum)]

(In reply to Kim Moir [:kmoir] from comment #3)
> Ben, I added new rules for release on Fennec bug can't see the id in the
> balrog ui or by viewing source.  And I don't have the credentials to the
> balrog db. Is there another way to see the ids of the rules I just added so
> I can create patches?

Booooo, we should probably show this in the UI somewhere....I filed bug 1146496 for this. In the meantime, here's a dump from the db:
mysql> select rule_id, product, channel from rules where product="Fennec" order by product, channel;
+---------+---------+-------------------+
| rule_id | product | channel           |
+---------+---------+-------------------+
|       8 | Fennec  | aurora            |
|     106 | Fennec  | aurora            |
|      15 | Fennec  | auroratest        |
|      91 | Fennec  | beta              |
|     120 | Fennec  | beta*             |
|      90 | Fennec  | beta-cdntest      |
|      89 | Fennec  | beta-localtest    |
|       4 | Fennec  | nightly           |
|      87 | Fennec  | nightly           |
|      83 | Fennec  | nightly-ash       |
|      82 | Fennec  | nightly-ash       |
|       6 | Fennec  | nightly-cedar     |
|      86 | Fennec  | nightlytest       |
|      24 | Fennec  | nightlytest       |
|     121 | Fennec  | release*          |
|     125 | Fennec  | release-cdntest   |
|     124 | Fennec  | release-localtest |
+---------+---------+-------------------+

Comment 5

[Kim Moir [:kmoir] ET]

Attachment: bug1139567.patch

Comment 6

[[e-mail] (:bhearsum)]

Wait...I just realized this is about split apk stuff. I don't think we're done here. AFAIK, we need multiple blobs for split APK to work, because api-11 and api-9 both use the same build target. I suspect they're stomping on each other in the current Fennec blobs.

Comment 7

[Kim Moir [:kmoir] ET]

notes from #releng

kmoir: so, do we need to make sure that these rules stay up to date now? we've been ignoring them on the beta channel for releases....
	bhearsum	things have gotten more complicated with split apk, too
	jlund|mergeduty	hmm, because 'android' doesn't exist anywhere anymore? would have thought it still be there for esr so would still need to be in prod-masters
	rail	no android on esr
	bhearsum	i have no idea about thi sone
	kmoir	bhearsum: yes I think they should stay up to date	
	rail	jlund|mergeduty: maybe, I was just guessing
	rail	jlund|mergeduty: ah, yes, we updated the template, but not the config
	rail	release-fennec-mozilla-release.py.template vs release-fennec-mozilla-release.py
	rail	and the latter is what used by buildbot
	bhearsum	kmoir: i think we have a problem...
	jlund|mergeduty	right. verifyConfigs still shows android key
	kmoir	bhearsum: what's the problem?
	bhearsum	kmoir: AFAICT, both api-11 and api-9 send data to the same fennec blobs
	bhearsum	which means they override one another	
	bhearsum	on nightly and aurora, we have separate blobs for them IIRC
	kmoir	looks	
	bhearsum	eg:
	bhearsum	Fennec-mozilla-aurora-nightly-latest and
	bhearsum	Fennec-mozilla-central-api-9-nightly-latest
	bhearsum	or whatever, the branch names
	bhearsum	the fennec release blobs point at files like: "*": "http://download.mozilla.org/?product=fennec-36.0.2&os=%OS_BOUNCER%&lang=%LOCALE%"
	bhearsum	which...i have no idea what that is
	bhearsum	http://download.mozilla.org/?product=fennec-36.0.2&os=android&lang=eo is a 404
	bhearsum	and even if it wasn't, the we'd end up sending either api-11 or api-9 people to the wrong build....
	bhearsum	it sounds like the split apk stuff really threw a wrench in this plan
	jlund|mergeduty	bhearsum: right, this hack: http://mxr.mozilla.org/build/source/tools/lib/python/balrog/submitter/cli.py#269
	bhearsum	yes
	bhearsum	which we've never had to deal with for release builds before
	jlund|mergeduty	bhearsum: in other news, looks like we need to give this file http://mxr.mozilla.org/build/source/buildbot-configs/mozilla/release-fennec-mozilla-release.py the https://bug1119237.bugzilla.mozilla.org/attachment.cgi?id=8578880 treatment (we only patched the template)		
	kmoir	I'll fix my patches
	bhearsum	kmoir: i don't think it's as simple as changing buildbot-configs
	rail	I also notices that sentry was trying to get android files with :lang not replaced
	rail	*noticed
	bhearsum	there's a whole bunch of stuff that needs to change, and i'm not sure the best way to go about it
	bhearsum	release automation isn't going to cope with a system that needs multiple rule ids per channel right now

Comment 8

[[e-mail] (:bhearsum)]

I don't know what the right thing to do here is. The difficulty arises from the fact that we use the same build target for API 11 and API 9, and use OS Version to distinguish them. Balrog doesn't support multiple binaries for the same build target in the same blob. For nightly-style updates we use separate blobs for these. In theory, we could do the same for release builds, but release automation scripts currently have an assumption of one release name per release. Eg, both api-11 and api-9 builds submit to releases like Firefox-36.0.4-build1. We could submit data to two different releases easily enough, but getting release automation to point the correct rules at the correct releases could prove tricky. We could make humans do that for now, though it's far less than ideal.

Something we've talked about a little bit before is adding support for different OS Versions in the blob. We've never really scoped this out well, so it's not clear how much work it is. There's certainly going to be issues that arise from OS Versions needing to be partial matches (ie, "Android 4.2.2" and "Android 4.2.1" are to be treated the same, generally) instead of full matches like build targets.

No matter what we do with the Balrog side of things, we need to submit separate bouncer entries for api-11 and api-9. We currently submit things to products like Fennec-36.0.4 with OS "android", and point at locations like: /mobile/releases/36.0.4/android/:lang/fennec-36.0.4.:lang.android-arm.apk. These are 404s in a split apk world.

Comment 9

[Richard Newman [:rnewman]]

It might be worth considering as you scope out a solution for this that APK splits can, and eventually probably will, occur along multiple boundaries -- for example, we might go so far as to split on region, screen density, and form factor.

Facebook ships sixteen different APKs to Play for each of its releases, and they're not all API-level splits.

Of course, not all of these are currently reflected in the updater URL, and Play might well support options that our own updater infra will not (so we'd have to not do those splits, or ship different code to Play versus our own distributions).

A solution based on including the Android SDK int (I wouldn't call it "OS version") will suffice for now, but is definitely kicking the can down the road.

Comment 10

[[e-mail] (:bhearsum)]

(In reply to Richard Newman [:rnewman] from comment #9)
> It might be worth considering as you scope out a solution for this that APK
> splits can, and eventually probably will, occur along multiple boundaries --
> for example, we might go so far as to split on region, screen density, and
> form factor.

It's great to have some sort of idea of what the future looks like, thank you.

> Facebook ships sixteen different APKs to Play for each of its releases, and
> they're not all API-level splits.
> 
> Of course, not all of these are currently reflected in the updater URL, and
> Play might well support options that our own updater infra will not (so we'd
> have to not do those splits, or ship different code to Play versus our own
> distributions).
> 
> A solution based on including the Android SDK int (I wouldn't call it "OS
> version") will suffice for now, but is definitely kicking the can down the
> road.

So, Nick and I were chatting about this a bit last night, and it seems to us that the thing that best fits our existing model for updates is to ensure that we have a unique build target (that's the Android_arm-eabi-gcc3 bit) for each stream of builds. It's an assumption that the update code has made for quite some time, and the fact that we're using the same one for api-11 and api-9 right now is the main source of pain here. Had we gone we separate build targets for these, this bug would be trivial. Eg, something like:
Android_arm-eabi-gcc3-api9
Android_arm_eabi-gcc3-api11

What do you think about adjusting it, Richard? Obviously we'll need to do something special for the builds that have already shipped, but doing that as one-off isn't a big deal.

And on a sidenote, if there's things we may want to split on in the future, adding them to the update URL now would let us fork those users onto new streams of builds later without them needing to opt-in. We could have things like:
Android_arm-eabi-gcc3-api9-smallscreen
Android_arm-eabi-gcc3-api9-bigscreen
Android_arm-eabi-gcc3-api11-smallscreen
Android_arm-eabi-gcc3-api11-bigscreen
etc.

The only restriction here is that it would be a bad idea to use fully dynamic information to avoid exploding the number of possibilitie too muchs. Eg: the client could make the call about whether to send smallscreen or bigscreen based on dpi or something. Sending the raw dpi or actual screen size would give us 100s (if not thousands) of possibilities, which becomes unmanageable. We probably want to move this out to a different bug.

Comment 11

[Richard Newman [:rnewman]]

(In reply to Ben Hearsum [:bhearsum] from comment #10)

> Android_arm-eabi-gcc3-api9
> Android_arm_eabi-gcc3-api11
> 
> What do you think about adjusting it, Richard?

That seems pretty straightforward from an implementation standpoint. That string is computed here:

            .replace("%BUILD_TARGET%", "Android_" + AppConstants.MOZ_APP_ABI + pkgSpecial)

so I think this is as simple as putting something in AppConstants.MOZ_PKG_SPECIAL (and figuring out what scheme to use, obv.)


**However** -- and this is a big however -- the SDK version is something that we *want* to be dynamic. Users upgrade their device OS, at which point the updater needs to run and fetch a *different* build.

So it's not enough to say "here's the build I'm running", because that's not necessarily the build the user *should* be running.

(That's on reason why I suggested a few months ago ripping out the OS version from the updater path and replacing it with the SDK int. Same thing but more precise, easier to map to a build.)

Screen dimensions etc. should be immutable, and the EABI likely is, but some of these could vary.

That implies that a solution here is not a solution like a fixed file hierarchy, but a computed lookup. "Oh, you're using API21 on an ARMv7 tablet? Here's our standard API11+ build."

Comment 12

[[e-mail] (:bhearsum)]

(In reply to Richard Newman [:rnewman] from comment #11)
> (In reply to Ben Hearsum [:bhearsum] from comment #10)
> 
> > Android_arm-eabi-gcc3-api9
> > Android_arm_eabi-gcc3-api11
> > 
> > What do you think about adjusting it, Richard?
> 
> That seems pretty straightforward from an implementation standpoint. That
> string is computed here:
> 
>             .replace("%BUILD_TARGET%", "Android_" + AppConstants.MOZ_APP_ABI
> + pkgSpecial)
> 
> so I think this is as simple as putting something in
> AppConstants.MOZ_PKG_SPECIAL (and figuring out what scheme to use, obv.)
> 
> 
> **However** -- and this is a big however -- the SDK version is something
> that we *want* to be dynamic. Users upgrade their device OS, at which point
> the updater needs to run and fetch a *different* build.
> 
> So it's not enough to say "here's the build I'm running", because that's not
> necessarily the build the user *should* be running.
>
> (That's on reason why I suggested a few months ago ripping out the OS
> version from the updater path and replacing it with the SDK int. Same thing
> but more precise, easier to map to a build.)

Right, so the Build Target doesn't necessary have to be a static thing. I guess it's not quite accurate to say "one build target per stream of binaries", it's more like "no build target should be used for multiple *different* binaries".

The "api" specification should be dynamically calculated based on what the user is running. As an example, we do this for our Desktop Firefox OS X builds. The browser could send one of the following, depending on the arch of the platform its running on:
Darwin_x86_64-gcc3-u-i386-x86_64
Darwin_x86-gcc3-u-i386-x86_64

And I assume we could something similar here, where the updater decides whether to send api-9 or api-11 based on the Android version.

> That implies that a solution here is not a solution like a fixed file
> hierarchy, but a computed lookup. "Oh, you're using API21 on an ARMv7
> tablet? Here's our standard API11+ build."

I think we're on the same page here -- this is exactly what we want to do. We have the ability to do it with Build Target, but not with OS Version.

Comment 13

[Richard Newman [:rnewman]]

Gotcha. So if we sent

  Android_arm-eabi-gcc3_api21

would you expect that to work (i.e., to serve the API 11+ build)?


(The alternative is that the client is responsible for figuring out which build it should request, which of course will be wrong if we introduce or remove a split since that build was made.)

Comment 14

[[e-mail] (:bhearsum)]

(In reply to Richard Newman [:rnewman] from comment #13)
> Gotcha. So if we sent
> 
>   Android_arm-eabi-gcc3_api21
> 
> would you expect that to work (i.e., to serve the API 11+ build)?

We can make that work. We just need to know all of the possible build targets are (which is why I was talking about making them not "fully" dynamic in a prior comment). We map the build platforms to build targets over here: https://github.com/mozilla/build-tools/blob/master/lib/python/release/platforms.py#L22 -- so as long as those are set-up correctly we can serve the right update to each of them.

To be a little more explicit about this, I would imagine that the pseudocode would look something like:
if android_version < foo:
  api_level = 9
elif android_version < bar:
  api_level = 11
elif android_version < baz:
  api_level = 14
else:
  api_level = 16

...as opposed to:
api_level = android_version

It gets a bit more complicated when we add new api_level definitions, of course. The way we've dealt with this in the past is "watersheding" people through a version that sets them correctly. Eg, if we add api_level 18 in Firefox 40, users on 39 (or earlier) + api level 16 would first get an update to 40, and then their next update check would correctly specify api_level 16 or 18, and we'd offer them the correct update to 41, 42, 43, etc.

> 
> (The alternative is that the client is responsible for figuring out which
> build it should request, which of course will be wrong if we introduce or
> remove a split since that build was made.)

Yeah, this seems bad to me. We explicitly want to be able to tell the client what to do from the server, otherwise it's easy to get into cases where the client gets stuck.

Comment 15

[Richard Newman [:rnewman]]

(In reply to Ben Hearsum [:bhearsum] from comment #14)

> ...as opposed to:
> api_level = android_version

Makes perfect sense to me.

 
> It gets a bit more complicated when we add new api_level definitions, of
> course. The way we've dealt with this in the past is "watersheding" people
> through a version that sets them correctly. Eg, if we add api_level 18 in
> Firefox 40, users on 39 (or earlier) + api level 16 would first get an
> update to 40, and then their next update check would correctly specify
> api_level 16 or 18, and we'd offer them the correct update to 41, 42, 43,
> etc.

In theory this isn't necessary (and of course chained updates are a pain for users; each is a 35MB download!); so long as the device sends its api_level as part of the update request, it can jump right to the end state, regardless of how many splits we've added.

The only situation I think we need watershedding is if we change the update URL itself, or the semantics of its fields -- so if we added is_tablet, say, we'd need to get you to the first release that knows how to send that, then after the next update check you might get another build that switched on that flag.

If by "api_level definitions" you mean that kind of addition, rather than just new numeric API level splits, then I think we're saying the same thing :D

Comment 16

[Richard Newman [:rnewman]]

I'm gonna put together a list of dispatchable fields some time today.

Comment 17

[[e-mail] (:bhearsum)]

(In reply to Richard Newman [:rnewman] from comment #15)
> (In reply to Ben Hearsum [:bhearsum] from comment #14)
> 
> > ...as opposed to:
> > api_level = android_version
> 
> Makes perfect sense to me.
> 
>  
> > It gets a bit more complicated when we add new api_level definitions, of
> > course. The way we've dealt with this in the past is "watersheding" people
> > through a version that sets them correctly. Eg, if we add api_level 18 in
> > Firefox 40, users on 39 (or earlier) + api level 16 would first get an
> > update to 40, and then their next update check would correctly specify
> > api_level 16 or 18, and we'd offer them the correct update to 41, 42, 43,
> > etc.
> 
> In theory this isn't necessary (and of course chained updates are a pain for
> users; each is a 35MB download!); so long as the device sends its api_level
> as part of the update request, it can jump right to the end state,
> regardless of how many splits we've added.

Hmmm. I think the "else" case from my previous comments means that version higher than some value (the maximum one that we specify) will all be the same thing. Eg, api_level 16 in Firefox 38 might be the maximum one we define. This means that Firefox 38 running on Android with api level 16 or *higher* would look the same to the update server. If we want to give api level 18 users their own build, we'd have to ship the Firefox 38 users a build that can specify api level 18 (say, in Firefox 39), and then we'd be able to distinguish those two populations for Firefox 40.

Now, if we're forward looking enough that we define all the useful api levels ahead of time, this might not be an issue.

> The only situation I think we need watershedding is if we change the update
> URL itself, or the semantics of its fields -- so if we added is_tablet, say,
> we'd need to get you to the first release that knows how to send that, then
> after the next update check you might get another build that switched on
> that flag.
> 
> If by "api_level definitions" you mean that kind of addition, rather than
> just new numeric API level splits, then I think we're saying the same thing
> :D

api_level was just an example, actually. I'm talking about the situation I spelled out more above, or things like this where we add a totally new thing.

Comment 18

[Richard Newman [:rnewman]]

(In reply to Ben Hearsum [:bhearsum] from comment #17)

> Hmmm. I think the "else" case from my previous comments means that version
> higher than some value (the maximum one that we specify) will all be the
> same thing. Eg, api_level 16 in Firefox 38 might be the maximum one we
> define. This means that Firefox 38 running on Android with api level 16 or
> *higher* would look the same to the update server.

Doesn't have to. Remember, clients will all report their true API level. The update server can change over time, so right now we'll have:

  if x86
    apk = (x86, *)
  else if (sdk_int < 11)
    apk = (arm, 9)
  else
    apk = (arm, 11)

and we can seamlessly add new clauses whenever we have new split builds to offer.


> If we want to give api
> level 18 users their own build, we'd have to ship the Firefox 38 users a
> build that can specify api level 18 (say, in Firefox 39), and then we'd be
> able to distinguish those two populations for Firefox 40.

That's why I suggested that clients truthfully report their metadata, and we do the mapping in the update server. Adding an API 18 split simply means adding a clause to that lookup table above; the clients don't need to change at all.


On the topic of future splits, here are the splits that Play supports:

---

    Support different OpenGL texture compression formats with each APK.
    Support different screen sizes and densities with each APK.
    Support different device feature sets with each APK.
    Support different platform versions with each APK.
    Support different CPU architectures with each APK (such as for ARM, x86, and MIPS, when your app uses the Android NDK).

---

I've seen reports of people doing per-country distributions, but I believe this is only at the listing level. We could do it for our own updater if we wish.



The fields I think we should support eventually:

  - CPU arch. We already do.
  - Platform (SDK) version. This has been the subject so far.
  - Screen size. small, normal, large, xlarge. The first two mean phones, the last two mean tablets. We'll need to do a little work here to decide how to make this attribute match Play. Splitting between tablet and phone will buy us a fair amount of space. This split would only meaningfully affect 11+.
  - Screen density. In theory this allows us to craft APKs for each of ldpi up to xxxhdpi, and include only the necessary image resources.


This would make a request like:

   Android_arm_eabi-gcc3-api21-large-xxhdpi

for which we would currently return:

  http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-api-11/fennec-39.0a1.multi.android-arm.apk

and eventually perhaps return:

  http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-android-api-19-tablet-xxhdpi/fennec-42.0a1.multi.android-arm.apk


(I think the CPU specifier should really be Androidy -- "armeabi-v7a", but I guess we must already ship code to translate that into Mozilla-ey, or rely on it never changing.)


Does that make sense?

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

Could you remind me of the reward for providing single-locale apks ? There's a reasonable amount of work ahead to support multiple splits like this in the update server, and I just want to check there's enough bang for that time-buck, as it were.

Comment 20

[Kim Moir [:kmoir] ET]

The original request for single locale updates is documented here
https://bugzilla.mozilla.org/show_bug.cgi?id=1019724#c6

"More specifically, the Maithili community is planning on pointing users to single locale builds which will not have any update mechanism....

Beyond the Maithili community, non-play-store updates will allow us to address non-play-store devices like the Kindle Fires and phones in China."

Comment 21

[[e-mail] (:bhearsum)]

(In reply to Richard Newman [:rnewman] from comment #18)
> (In reply to Ben Hearsum [:bhearsum] from comment #17)
> 
> > Hmmm. I think the "else" case from my previous comments means that version
> > higher than some value (the maximum one that we specify) will all be the
> > same thing. Eg, api_level 16 in Firefox 38 might be the maximum one we
> > define. This means that Firefox 38 running on Android with api level 16 or
> > *higher* would look the same to the update server.
> 
> Doesn't have to. Remember, clients will all report their true API level. The
> update server can change over time, so right now we'll have:
> 
>   if x86
>     apk = (x86, *)
>   else if (sdk_int < 11)
>     apk = (arm, 9)
>   else
>     apk = (arm, 11)
> 
> and we can seamlessly add new clauses whenever we have new split builds to
> offer.
> 
> 
> > If we want to give api
> > level 18 users their own build, we'd have to ship the Firefox 38 users a
> > build that can specify api level 18 (say, in Firefox 39), and then we'd be
> > able to distinguish those two populations for Firefox 40.
> 
> That's why I suggested that clients truthfully report their metadata, and we
> do the mapping in the update server. Adding an API 18 split simply means
> adding a clause to that lookup table above; the clients don't need to change
> at all.

Ah, I see. I had it in my head that we'd map API levels to our own values. Not doing that should be fine as long as the list doesn't get too far out of control.

> I've seen reports of people doing per-country distributions, but I believe
> this is only at the listing level. We could do it for our own updater if we
> wish.

Per-country isn't something we support right now. That's bug 858114.


> The fields I think we should support eventually:
> 
>   - CPU arch. We already do.
>   - Platform (SDK) version. This has been the subject so far.
>   - Screen size. small, normal, large, xlarge. The first two mean phones,
> the last two mean tablets. We'll need to do a little work here to decide how
> to make this attribute match Play. Splitting between tablet and phone will
> buy us a fair amount of space. This split would only meaningfully affect 11+.
>   - Screen density. In theory this allows us to craft APKs for each of ldpi
> up to xxxhdpi, and include only the necessary image resources.
> 
> 
> This would make a request like:
> 
>    Android_arm_eabi-gcc3-api21-large-xxhdpi
> 
> for which we would currently return:
> 
>  
> http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-
> android-api-11/fennec-39.0a1.multi.android-arm.apk
> 
> and eventually perhaps return:
> 
>  
> http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-central-
> android-api-19-tablet-xxhdpi/fennec-42.0a1.multi.android-arm.apk
> 
> 
> (I think the CPU specifier should really be Androidy -- "armeabi-v7a", but I
> guess we must already ship code to translate that into Mozilla-ey, or rely
> on it never changing.)
> 
> 
> Does that make sense?

Yeah, sounds pretty reasonable to me.


(In reply to Nick Thomas [:nthomas] from comment #19)
> Could you remind me of the reward for providing single-locale apks ? There's
> a reasonable amount of work ahead to support multiple splits like this in
> the update server, and I just want to check there's enough bang for that
> time-buck, as it were.

This is a good point. I think we _do_ need to move the api level to the build target to fix this bug, but before we start adding all sorts of other bits we should probably get this answered.

Comment 22

[[e-mail] (:bhearsum)]

> (In reply to Nick Thomas [:nthomas] from comment #19)
> > Could you remind me of the reward for providing single-locale apks ? There's
> > a reasonable amount of work ahead to support multiple splits like this in
> > the update server, and I just want to check there's enough bang for that
> > time-buck, as it were.
> 
> This is a good point. I think we _do_ need to move the api level to the
> build target to fix this bug, but before we start adding all sorts of other
> bits we should probably get this answered.

Mark, can you answer this?

Comment 23

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #22)
> > (In reply to Nick Thomas [:nthomas] from comment #19)
> > > Could you remind me of the reward for providing single-locale apks ? There's
> > > a reasonable amount of work ahead to support multiple splits like this in
> > > the update server, and I just want to check there's enough bang for that
> > > time-buck, as it were.
> > 
> > This is a good point. I think we _do_ need to move the api level to the
> > build target to fix this bug, but before we start adding all sorts of other
> > bits we should probably get this answered.

Catlee pointed out that this potentially affects more than just single locale, because even multilocale builds on Nightly/Aurora update through AUS.

Comment 24

[Kim Moir [:kmoir] ET]

:mfinkle can you answer the questions in comment 22?

Comment 25

[Brad Lassey [:blassey] (use needinfo?)]

redirecting the NI to karen

Comment 26

[Karen Rudnitski [:kar]]

Fennec's APK size is becoming more and more of a barrier for download or even pre-install. 

1) Ordinary people are reticent in downloading Firefox on their mobile because of how much data they would consume by downloading it (as well as the regular updates) that it's becoming a decision factor in NOT trying out the app. This applies to pay-as-you-go users and other users who are much more price sensitive. We're hearing this in LatAm and also in our East Asian countries. 

2) Pre-install (even pay-to-play) opportunities are also scrutinizing the size of our APK as it means that other apps can't be loaded. Important for lower- and medium- speced phones, which is the vast majority of phones out there.

Splitting out the APK allows us the flexibility to target pre-install and download opportunities with a targeted offering whilst reducing the size of the APK itself (to not include bytes that will most likely never be used by that audience). This reduces one of our barriers to adoption.

Comment 27

[Kim Moir [:kmoir] ET]

I don't think this bug is relevant anymore