Closed Bug 1120762 Opened 9 years ago Closed 9 years ago

Test, solidify, and document Google Play distribution approach for split APKs

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
All
Android
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(firefox37+ fixed, fennec37+)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox37 + fixed
fennec 37+ ---

People

(Reporter: rnewman, Assigned: Sylvestre)

References

Details

Attachments

(3 files)

with-both.png
72.61 KB, image/png
Details
with-37.png
63.15 KB, image/png
Details
sc.png
18.27 KB, image/png
Details 
(Wild guess at the bug component; please move if there's a better place for release management bugs.)

Split APK work is merging to Aurora this week. Thanks to heroic efforts (go go jlund and crew!) we've got builds and tests on treeherder.

We now have six weeks to figure out how to get the two separate builds into Play without breaking anything.

Presumably we'll follow a very similar strategy to the ARMv7/x86 split; the manifests will take care of most of the work, so this should be a matter of uploading the right APKs and making sure they have the right version numbers, and are served correctly to new and updating devices of various API versions.

I confess to having no idea about where this is documented, so I hope someone on the CC list leaps up and fills in the gaps!

There are some open questions in this thread that need to be explored:

https://mail.mozilla.org/pipermail/mobile-firefox-dev/2015-January/001058.html

Particular areas for QA:

* Upgrading from a 36 Gingerbread device to 37
* ... and modern (HC, ICS, JB, KK, L)
* Fresh installs, likewise -- do you get the right APK?
* What happens if you install 37 on a Gingerbread device via Play, and then upgrade the device to a modern Android release?
  * Options: it gets uninstalled; it stays installed but crashes (see Bug 1119915); it stays installed but Play reports an available update; something surprising.
  * Conclusions: whether we use maxSdkVersion, version codes, or scream and run away.

I have my own Play account if we need a staging area for this work.
Blocks: 1063873
Depends on: 1122059
Blocks: 1122071
Depends on: 1122509
What's the next steps in this bug? We have an Aurora build on the Play Store for testing purposes. We could update that application with the two APKs and see how it works.

Sound like a plan?
Mark, we can try by hand, no worries. Let me know which apk you want me to upload.

For something more scalable, we need a manifest (bug 1122509) to know which files are part of a release, rename the various files to avoid confusion (bug 1122059) update two scripts (bug 1122071 and bug 1122076). Once we have the manifest, we can do it quickly.

FYI, we have the possibility to use also the Google play alpha and beta track if you want.
OK, so we have one course of action that's manual, and then automating.

Sylvestre, can you take on the task of manually uploading APKs?

These should do the trick for the Aurora app:

http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-aurora-android-api-9/
http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-aurora-android-api-11/
http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/latest-mozilla-aurora-android-x86/


It might be a good idea to coordinate with Aaron and Kevin in advance to make sure they can set up devices for a pre-split APK in order to test upgrades (see Comment 0).

For that you might need to upload a pre-split APK first, like these:

http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2014-12-31-00-40-08-mozilla-aurora-android/
http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2014-12-31-00-40-08-mozilla-aurora-android-x86/

do the installs, then upload the split APKs, and test away.



Coop, could you take a look into assigning someone to the bugs indicated in Comment 2, so we can move automation forward?
Assignee: nobody → sledru
Status: NEW → ASSIGNED
Flags: needinfo?(kbrosnan)
Flags: needinfo?(coop)
Flags: needinfo?(aaron.train)
(In reply to Richard Newman [:rnewman] from comment #3) 
> Coop, could you take a look into assigning someone to the bugs indicated in
> Comment 2, so we can move automation forward?

As a first step, I've put the bugs into more specific components. Finding owners may take a little longer.
Flags: needinfo?(coop)
Sure, I will (try to) work on that this week. Is that OK with you?
(In reply to Sylvestre Ledru [:sylvestre] from comment #5)
> Sure, I will (try to) work on that this week. Is that OK with you?

Fine by me. So long as we have all the answers and a plan in place in 19 calendar days, we'll be good to go!
Attached image with-both.png — 
Actually, I did it yesterday but I didn't take the time to give you feedback.

So, I uploaded both apk for 36 and the three 37 but only two 37 showed up (cf with-both.png)
after removing 36, google play is still complaining...  (with-37.png)
Attached image with-37.png — 

    
    

    
  
I think you might be mixing up the stages.

You need to upload the two 36 APKs, and only those.

Wait for them to propagate, install them on devices.

Upload the three 37 APKs, replacing the 36 APKs.

Wait for them to propagate, test installs and upgrades, making sure that Gingerbread devices upgrade to the right half of the split.

Does that make sense?

    
    

    
  
(In reply to Richard Newman [:rnewman] from comment #9)
> Wait for them to propagate, install them on devices.
Maybe I am missing a point but I don't see how can I do that with Google play without making aurora public.

    
    

    
  
Yes, it needs to be a public app. I have my own account and app for this purpose.

    
    

    
  
Richard, Lukas suggested that you could give us some permissions in your google play test account. Is it possible?
Flags: needinfo?(rnewman)

    
    

    
  
Invitation sent!

You'll need to create a new listing to test with; the Referrer Intent Test app there is signed with my keys.
Flags: needinfo?(rnewman)

    
    

    
  
We will need apps with a custom name though as the play store won't allow apps with duplicate IDs.

    
    

    
  
What is going on here? Need a plan before the 23rd.

    
  
Flags: needinfo?(sledru)

    
    

    
  
I am working on that tomorrow. I have access to Richard's application.
Flags: needinfo?(sledru)

    
    

    
  
Richard, do you see a workaround for this?
"You need to use a different package name because "org.mozilla.firefox_beta" already exists in Google Play."
Flags: needinfo?(rnewman)

    
    

    
  
Yes: create a build with a different package name. You won't be able to upload a signed build straight from FTP.

You should be able to unpack the gecko-unsigned-unaligned APK, modify its manifest, repackage it, and re-align and re-sign before upload.

If that doesn't work, you'll need to make a build with a different ID at an earlier step in the process. 

You can either do that locally (which is what I do with Referrer Intent Test) or by making a branding change just like you use to get org.mozilla.firefox{_beta,_aurora,} and pushing that to try, once on top of 36 and once on top of 37.
Flags: needinfo?(rnewman)

    
    

    
  
I think I will need your help.
So, I unpacked the apk, updated package-name.txt, resigned the apk (http://developer.android.com/tools/publishing/app-signing.html), realigned the zip and upload the file.
The verification is complaining about the incorrect checksum of package-name.txt (as expected) and the upload is rejecting.
META-INF/MANIFEST.MF is correct (openssl sha1 -binary package-name.txt |openssl base64) but not META-INF/RELEASE.SF
how this is regenerated?

    
    

    
  
(In reply to Sylvestre Ledru [:sylvestre] from comment #19)
> I think I will need your help.
> So, I unpacked the apk, updated package-name.txt, resigned the apk
> (http://developer.android.com/tools/publishing/app-signing.html), realigned
> the zip and upload the file.
> The verification is complaining about the incorrect checksum of
> package-name.txt (as expected) and the upload is rejecting.
> META-INF/MANIFEST.MF is correct (openssl sha1 -binary package-name.txt
> |openssl base64) but not META-INF/RELEASE.SF
> how this is regenerated?

Updating package-name.txt won't do anything.  That's a vestige of something -- I've never really known what.

rnewman, repackaging an APK like you suggest is not trivial.  The APK contains only a compiled binary manifest -- does aapt support rebadging in this way?  I am quite convinced that the package name (ANDROID_PACKAGE_NAME) gets baked into C++ code, even if it's not ever used, so I'm not confident this will ever work.  On the other hand, I've been developing Fennec with downloaded org.mozilla.fennec binaries locally for months and had no problems.

I suggest either a custom local build or a custom try build with different package names rather than repackaging :(

    
    

    
  
(In reply to Nick Alexander :nalexander from comment #20)

> I suggest either a custom local build or a custom try build with different
> package names rather than repackaging :(

Yes, that's my recommended approach.

If I don't totally crash and burn this evening (I've been working since about 3am), I'll try to kick off beta-esque try builds for 36 and 37 with a custom package name and the splits needed in Comment 9.

Sylvestre, if you can do that, great, let me know.

If not, please post the config patch you usually use to test a beta build on Try, so I don't have to figure it out from first principles!

    
    

    
  
(In reply to Richard Newman [:rnewman] from comment #21)

> If not, please post the config patch you usually use to test a beta build on
> Try, so I don't have to figure it out from first principles!
I think you are mistaking about my responsibilities (and skills ;). I am doing release management, not release engineering. The most important thing that I am doing with the APK is to upload them to Google play :)

This bug seems to require more skills than expected. I think we should ask someone to someone who knows!

    
    

    
  
(In reply to Richard Newman [:rnewman] from comment #21)
> (In reply to Nick Alexander :nalexander from comment #20)
> 
> > I suggest either a custom local build or a custom try build with different
> > package names rather than repackaging :(
> 
> Yes, that's my recommended approach.
> 
> If I don't totally crash and burn this evening (I've been working since
> about 3am), I'll try to kick off beta-esque try builds for 36 and 37 with a
> custom package name and the splits needed in Comment 9.
> 
> Sylvestre, if you can do that, great, let me know.
> 
> If not, please post the config patch you usually use to test a beta build on
> Try, so I don't have to figure it out from first principles!

Should we be changing the package name on per-checkin continuous integration builds for split-apk?

bhearsum, do you know how test beta release builds on try ^ and how they are configured? I'd like to help but I neglect the familiarity with the release bits and it sounds like what's needed is out of relman's scope.
Flags: needinfo?(bhearsum)

    
    

    
  
I'm unable to make these builds, because I can't get Aurora to build with a release mozconfig; it uses the same hard-coded /tools/ GCC path as Beta, but Aurora barfs where Beta does not.

https://rnewman.pastebin.mozilla.org/8822764

    
    

    
  
Attempting to create appropriate builds via try, specifying a package name and all applicable Android targets.

Beta:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=807941650b09

Aurora:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=40349a0c775c

    
    

    
  
Judging by the beta build:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=807941650b09

it doesn't look like I can make an all-version build any more. Including 'android' in the trychooser string doesn't help; we only get 9-10/11+ builds. This is no use for testing.

jlund, bhearsum: any ideas how I can get a custom version of our current beta build configs (i.e., one APK for API 9+) using try?
Flags: needinfo?(jlund)

    
    

    
  
Hacking away at mozconfigs to try to make the API 11+ build a 9+ build, to try to ape the old world.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=8e9e32568af0

    
    

    
  
(In reply to Richard Newman [:rnewman] from comment #27)
> Judging by the beta build:
> 
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=807941650b09
> 
> it doesn't look like I can make an all-version build any more. Including
> 'android' in the trychooser string doesn't help; we only get 9-10/11+
> builds. This is no use for testing.
> 
> jlund, bhearsum: any ideas how I can get a custom version of our current
> beta build configs (i.e., one APK for API 9+) using try?

This is often a problem when we have changes that depend on Buildbot that ride the trains. Try is configured like mozilla-central, so this gets tricky.

Your best bet is likely to make an api 9 or 11 build, but adjust the in-tree mozconfig (http://mxr.mozilla.org/mozilla-central/source/mobile/android/config/mozconfigs/android-api-11/nightly or http://mxr.mozilla.org/mozilla-central/source/mobile/android/config/mozconfigs/android-api-9-10-constrained/nightly) to do whatever kind of build you want.
Flags: needinfo?(bhearsum)
Flags: needinfo?(jlund)

    
    

    
  
(In reply to Richard Newman [:rnewman] from comment #30)
> OK, Sylvestre, give this a shot.
After uploading the 36 arm apk, when I tried to upload the i386, I am getting:

Your APK needs to have the package name org.twinql.firefox_beta.

    
    

    
  
Maybe I missed that line in the x86 mozconfig.

Never mind -- we shouldn't need to test x86 here. Roll on!

    
    

    
  
OK. Good. I finished https://play.google.com/apps/publish/?dev_acc=12491135504504423873#PricingPlace:p=org.twinql.firefox_beta
I haven't uploaded 37 because I guess we want to try the update on a device.

It is "Pending publication" for now. I will let you know when it is live (or maybe Richard can tell us)

    
    

    
  
I have installed https://play.google.com/store/apps/details?id=org.twinql.firefox_beta on several Android devices including 3 Android 2.3.x phones.

Marc do you have time to install on some of the highlighted devices?
https://docs.google.com/a/mozilla.com/spreadsheets/d/1IXEx2Ul6gF73IHN76lc5AjF7VL395DhZnYZTrQFLcEk/edit#gid=0

Aaron if you have some time installing on any Android 2.3.x devices you have available would be helpful.
Flags: needinfo?(kbrosnan) → needinfo?(mschifer)

    
    

    
  
I believe we are targeting Thursday for a release of 37b1.

    
    

    
  
We've successfully installed https://play.google.com/store/apps/details?id=org.twinql.firefox_beta on: 
HTC Desire S   (2.3.3)
Samsung R      (2.3.4)
HTC Desire HD  (2.3.5)
Acer Iconia    (3.2.1)
Galaxy Note 3  (4.4.2)

    
    

    
  
OK, let me know when you want me to upload 37 to test the update.

    
    

    
  
Couple devices I ran this morning with a successful installation of https://play.google.com/store/apps/details?id=org.twinql.firefox_beta
   
   Nexus S      (2.3.6)
   Nexus 6      (5.0.1)
   Galaxy SII   (4.1.2)
   Galaxy Note  (4.1.2)
   Galaxy Tab 2 (4.2)


Note: I did not test x86.

What are we doing in regard to x86 as well?
Flags: needinfo?(aaron.train)

    
    

    
  
same twinql build as beta was installed by me on:

htc desire hd a9191, Gingerbread 2.3.5

(In reply to Aaron Train [:aaronmt] from comment #39)
> Couple devices I ran this morning with a successful installation of
> https://play.google.com/store/apps/details?id=org.twinql.firefox_beta
>    
>    Nexus S      (2.3.6)
>    Nexus 6      (5.0.1)
>    Galaxy SII   (4.1.2)
>    Galaxy Note  (4.1.2)
>    Galaxy Tab 2 (4.2)
> 
>

    
    

    
  
We are ignoring x86 in this test. As long as the build ids are correct, with x86 > arm it will just work.

    
    

    
  
Well, the package name is set in mozconfig.common.override, the same as for the beta build I pushed, so I have no idea why that's happening. Maybe one of the releng folks does?
Flags: needinfo?(rnewman)

    
    

    
  
There are two override files. This push changes them both.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=96b1e9451cb3

    
    

    
  
Richard, same with these files.
"Your APK needs to have the package name org.twinql.firefox_beta."

(we really have to rename these files, that is really confusing to have the same file name for two different APKs).
Flags: needinfo?(sledru) → needinfo?(rnewman)

    
    

    
  
Nothing I try can override the package name or app name on 37 or higher. jlund, gps, any ideas?

https://treeherder.mozilla.org/#/jobs?repo=try&revision=19d1d72f4a9d
Flags: needinfo?(rnewman)
Flags: needinfo?(jlund)
Flags: needinfo?(gps)

    
    

    
  
Trying again, with an additional hacky approach courtesy of nalexander:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ac861a907525
Flags: needinfo?(jlund)
Flags: needinfo?(gps)
Flags: needinfo?(mschifer)

    
    

    
  
Those two APKs have the exact same version code. That probably screws things up.

    
    

    
  
If I had to guess, I'd say the version we want you to run should have the higher version code. That means the 11+, because it'll be inapplicable to lower versions.

Here's the chunk of Makefile for this:

ifeq (,$(ANDROID_VERSION_CODE))
ifeq ($(CPU_ARCH),arm)
ifeq ($(MIN_CPU_VERSION),7)
ANDROID_VERSION_CODE=$(shell cat $(DEPTH)/config/buildid | cut -c1-10)
else
# decrement the version code by 1 for armv6 builds so armv7 builds will win any compatability ties
ANDROID_VERSION_CODE=$(shell echo $$((`cat $(DEPTH)/config/buildid | cut -c1-10` - 1)))
endif
else #not arm, so x86
# increment the version code by 1 for x86 builds so they are offered to x86 phones that have arm emulators
ANDROID_VERSION_CODE=$(shell echo $$((`cat $(DEPTH)/config/buildid | cut -c1-10` + 1)))
endif
endif



I suspect what we want to do here is:

* Add the MIN_SDK_VERSION to the version code. That'll give us a gap of 2.
* Additionally increment the 9-23 x86 version by 3, so that the 9-23 APK is higher than the 11-23 ARMv7.
* Eliminate the ARMv6 logic.

Nick, others: opinions?
Flags: needinfo?(nalexander)

    
    

    
  
Here's a try build that might exercise this.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=17d440d94ed2

    
    

    
  
https://treeherder.mozilla.org/#/jobs?repo=try&revision=5c9654b605e1

API 9:

package: name='org.twinql.firefox_beta' versionCode='2015022629' versionName='37.0' platformBuildVersionName='5.0-1521886'

API 11:

package: name='org.twinql.firefox_beta' versionCode='2015022631' versionName='37.0' platformBuildVersionName='5.0-1521886'

x86:

package: name='org.twinql.firefox_beta' versionCode='2015022632' versionName='37.0' platformBuildVersionName='5.0-1521886'



When uploaded to Play, these three APKs yield this warning:

---
    A device with API levels in range 11+ is eligible to receive version 2015022631, which is optimized for higher API levels, but actually receives version 2015022632 because it has a higher version code. This would occur when
    Screen layouts containing any of [small, normal, large, xlarge] and
    Native platforms containing any of [x86 (armeabi-v7a)*, x86_64 (arm64-v8a)*, x86_64 (armeabi-v7a)*] and
    OpenGL ES versions in range 2.0+ and
    Features containing all of [android.hardware.TOUCHSCREEN, android.hardware.WIFI].

    A device upgrading from API levels in range 9-10 to API levels in range 11+ would become eligible to receive version 2015022631, which is optimized for higher API levels, but would actually receive version 2015022632 because it has a higher version code. This would occur when
    Screen layouts containing any of [small, normal, large, xlarge] and
    Native platforms containing any of [x86 (armeabi-v7a)*, x86_64 (arm64-v8a)*, x86_64 (armeabi-v7a)*] and
    OpenGL ES versions in range 2.0+ and
    Features containing all of [android.hardware.TOUCHSCREEN, android.hardware.WIFI].
    Some devices are eligible to run multiple APKs. In such a scenario, the device will receive the APK with the higher version code.
---

These warnings are actually exactly what we want to see, as far as I can tell: this is saying that an x86-capable device will get the x86 9+ APK instead of the ARMv7 11+ APK, regardless of whether it's 11+ initially or upgrades to 11+.


I hit PUBLISH on this configuration, so we can try out those updates in the morning!
Flags: needinfo?(rtanglao)
Flags: needinfo?(kbrosnan)
Flags: needinfo?(aaron.train)

    
  
Depends on: 1137586

    
    

    
  
I filed Bug 1137586 to cover version code shifting, and attached a patch for Nick to review.

    
    

    
  
"This app is incompatible with all of your devices."
Flags: needinfo?(aaron.train)

    
    

    
  
This app got pulled from the Play Store for using Firefox iconography.

*tableflip*

    
    

    
  
Based on the warnings provided by Play, I would be fairly confident in moving forward on Beta once Bug 1137586 is reviewed and landed.

If you want to test the upgrade path (which we should, but I no longer have the patience for), Sylvestre will have to upload the APKs for 36, then the ones from Comment 53, in a new app listing in the Mozilla Play account.

    
  
Depends on: 1137898

    
    

    
  
Bug 1137898 landed on beta.

    
    

    
  
My feedback was incorporated into the patch of Bug 1137586 and into the follow-up ticket Bug 1137898.
Flags: needinfo?(nalexander)

    
    

    
  

      
      
      
      

        Attached image
          
        sc.png
        — 
      

    


  
So, Kevin & I will try to use the beta channel of firefox beta to test that.

For the record
$ wget http://ftp.mozilla.org/pub/mozilla.org/mobile/releases/37.0b2/android-api-9/multi/fennec-37.0b2.multi.android-arm.apk http://ftp.mozilla.org/pub/mozilla.org/mobile/releases/37.0b2/android-api-11/multi/fennec-37.0b2.multi.android-arm.apk http://ftp.mozilla.org/pub/mozilla.org/mobile/releases/37.0b2/android-x86/multi/fennec-37.0b2.multi.android-i386.apk

In the beta tab in the google play interface, I uploaded the three apk and on this side, this worked perfectly! cf screenshot

    
    

    
  
Looks like a group needs to be created and assign some google accounts to it. https://support.google.com/googleplay/android-developer/answer/3131213?hl=en

I don't have access to this.
Flags: needinfo?(kbrosnan)

    
    

    
  
OK. I created a group (firefox-beta-program@@googlegroups.com) using the same account that we use to publish the APK (so that the rest of the release management team have access to it).

I send an invitation to a few people (Kevin, Richard, aaron, release management)

I am supposed to share this URL:
https://play.google.com/apps/testing/org.mozilla.firefox_beta

    
    

    
  
Looks like you must click the link to become a beta tester.

    
    

    
  
This looks good on my end. If you get a chance to get Aaron configured properly in the morning that would be good to have a second set of eyes on this. Else I'll be on line at 08:30 Pacific.

    
    

    
  
OK. As the 37 cycle is short, I pushed it to the prod cycle now. Thanks


FYI, I reported bug 1139329 to document that.

    
    

    
  
We've deployed this change to the Play store and have tested in Yandex and T-Store as well. If the docs have been updated, I think we can resolve this bug.

    
    

    
  
i was on PTO last Monday-Thursday so I missed this. But all is well that ends well. If you need further Gingerbread testing please needinfo me on this or future bugs.
Flags: needinfo?(rtanglao)

    
    

    
  
OK, let's close this. Great work, folks!
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED

    
  
Blocks: 1137898
No longer depends on: 1137898

    
  
Blocks: 1122509
No longer depends on: 1122509

    
  
Blocks: 1122059
No longer depends on: 1122059
QA Contact: pmoore → mshal

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: