1042128 - Figure out Play Store dynamics for Android 2.2/ARMv6 EOL support releases

Status: RESOLVED DUPLICATE of bug 1040319

(Release Engineering :: Release Automation: Other, defect)

Description

[Chris Cooper [:coop] (he/him)]

We're EOLing Android 2.2, but will continue to provide security releases until January 2015. 

We will be releasing these security releases off of the ESR branch we cut for Firefox 31, but we want to update existing release channel users in the Play Store. We need to make sure that the build IDs for the various mobile builds are manipulated in such a way that users receive the correct update.

catlee: can you point Pete at the current logic for manipulating the build IDs, please? I'm sure he'll have some follow-up questions as well.

Comment 1

[Chris AtLee [:catlee]]

The buildid manipulation is done here: http://mxr.mozilla.org/mozilla-central/source/mobile/android/base/Makefile.in#17

For releases, we generate the buildid as part of buildbot scheduling and pass that down to the build in the MOZ_BUILD_DATE environment variable. This normally ensures that all builds in a release use the same buildid (or the same base buildid in the case of these mobile builds).

One complication here is that we'll be triggering parallel release automation for ESR, so will probably need to adjust the offsets on ESR to be much less than the ones built off of release.

Comment 2

[Chris Cooper [:coop] (he/him)]

(In reply to Chris AtLee [:catlee] from comment #1) 
> One complication here is that we'll be triggering parallel release
> automation for ESR, so will probably need to adjust the offsets on ESR to be
> much less than the ones built off of release.

Yes, we had talked last week about subtracting a week to deal with respins and such.

Do we still have fake products in the Play Store where we can test versioning issues like this?

Comment 3

[Brad Lassey [:blassey] (use needinfo?)]

(In reply to Chris Cooper [:coop] from comment #2)
> 
> Do we still have fake products in the Play Store where we can test
> versioning issues like this?

Yup, org.mozilla.fennec_blassey is in the play store disabled

Comment 4

[Pete Moore [:pmoore][:pete]]

So I've started looking at this today. I have some points of confusion.

I see from comment 0, and other sources such as https://bug767864.bugzilla.mozilla.org/attachment.cgi?id=636891 that the Play Store seems not to be entirely responsible for the update process for delivering new Fennec binaries, which is what I had previously thought (I figured we serve updates for Desktop, but all Fennec updates would be managed via the Play Store integration with the OS and Fennec itself would not query aus2/aus3/aus4 etc for updates). I'd be interested to understand better how this interplay between the Play Store and the native Firefox update mechanisms (aus2/3/4) work.

I'm also aware when we request updates from aus, there are a bunch of parameters we pass (e.g. looking at about:config we have something like app.update.url='https://aus3.mozilla.org/update/3/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml' for desktop, and something similar for mobile). I see http://mxr.mozilla.org/mozilla-central/source/mobile/android/base/Makefile.in#17 sets ANDROID_VERSION_CODE based on the buildid first 10 digits, with an offset which favours x86 (+1 hour), followed by arm7 (+0), then arm6 (-1 hour) - so presumably we publish three separate builds (x86, arm7, arm6) to the Play Store, which all potentially advertise themselves as "available" to an updating device, and the device chooses the one with the highest ANDROID_VERSION_CODE (however this appears in the final apk or apk meta data). What I didn't really understand is why arm6/arm7/x86 updates don't share the same id, but something else determines which is the best binary architecture to serve, rather than (slightly misleadingly) adjusting the id (e.g. having the architecture as part of %BUILD_TARGET% name or a separate parameter).

I think what would be useful is to talk through this with someone that understands the Fennec update process well, as I don't quite yet have all the pieces I need to solve the problem, and a walk-through would be really helpful in this case.

Another question: from tbpl I only see opt builds for Android 2.2 (arm) and Android 4.2 (x86). Are the 2.2 builds for arm7, and we no longer performing CI for arm6? But don't we need to release arm6 binaries until January too for Android 2.2? I'm confused what the relationship is between Android OS versions (2.2, 2.3, 4.0, 4.2) and supported architectures (arm6, arm7 and x86). Is there a lookup table somewhere?

Many thanks,
Pete

Comment 5

[Brad Lassey [:blassey] (use needinfo?)]

(In reply to Pete Moore [:pete][:pmoore] from comment #4)
> So I've started looking at this today. I have some points of confusion.
> 
> I see from comment 0, and other sources such as
> https://bug767864.bugzilla.mozilla.org/attachment.cgi?id=636891 that the
> Play Store seems not to be entirely responsible for the update process for
> delivering new Fennec binaries, which is what I had previously thought (I
> figured we serve updates for Desktop, but all Fennec updates would be
> managed via the Play Store integration with the OS and Fennec itself would
> not query aus2/aus3/aus4 etc for updates). I'd be interested to understand
> better how this interplay between the Play Store and the native Firefox
> update mechanisms (aus2/3/4) work.
We don't use aus for builds delivered through the play store

 
> I'm also aware when we request updates from aus, there are a bunch of
> parameters we pass (e.g. looking at about:config we have something like
> app.update.url='https://aus3.mozilla.org/update/3/%PRODUCT%/%VERSION%/
> %BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/%OS_VERSION%/%DISTRIBUTION%/
> %DISTRIBUTION_VERSION%/update.xml' for desktop, and something similar for
> mobile). I see
> http://mxr.mozilla.org/mozilla-central/source/mobile/android/base/Makefile.
> in#17 sets ANDROID_VERSION_CODE based on the buildid first 10 digits, with
> an offset which favours x86 (+1 hour), followed by arm7 (+0), then arm6 (-1
> hour) - so presumably we publish three separate builds (x86, arm7, arm6) to
> the Play Store, which all potentially advertise themselves as "available" to
> an updating device, and the device chooses the one with the highest
> ANDROID_VERSION_CODE (however this appears in the final apk or apk meta
> data). What I didn't really understand is why arm6/arm7/x86 updates don't
> share the same id, but something else determines which is the best binary
> architecture to serve, rather than (slightly misleadingly) adjusting the id
> (e.g. having the architecture as part of %BUILD_TARGET% name or a separate
> parameter).
I don't think any of this matters for play store builds since we're not using aus

> I think what would be useful is to talk through this with someone that
> understands the Fennec update process well, as I don't quite yet have all
> the pieces I need to solve the problem, and a walk-through would be really
> helpful in this case.
> 
> Another question: from tbpl I only see opt builds for Android 2.2 (arm) and
> Android 4.2 (x86). Are the 2.2 builds for arm7, and we no longer performing
> CI for arm6? But don't we need to release arm6 binaries until January too
> for Android 2.2? I'm confused what the relationship is between Android OS
> versions (2.2, 2.3, 4.0, 4.2) and supported architectures (arm6, arm7 and
> x86). Is there a lookup table somewhere?
The renaming of Android 2.2 to Android 2.3 is being taken care of in another bug.

Comment 6

[Pete Moore [:pmoore][:pete]]

(In reply to Pete Moore [:pete][:pmoore] from comment #4)

> I see
> http://mxr.mozilla.org/mozilla-central/source/mobile/android/base/Makefile.in#17 
> sets ANDROID_VERSION_CODE based on the buildid first 10 digits, with
> an offset which favours x86 (+1 hour), followed by arm7 (+0), then arm6 (-1
> hour) - so presumably we publish three separate builds (x86, arm7, arm6) to
> the Play Store, which all potentially advertise themselves as "available" to
> an updating device, and the device chooses the one with the highest
> ANDROID_VERSION_CODE (however this appears in the final apk or apk meta
> data).

I've seen from http://hg.mozilla.org/mozilla-central/file/71497ed2e0db/mobile/android/base/AndroidManifest.xml.in#l6 that ANDROID_VERSION_CODE ends up as:
http://developer.android.com/guide/topics/manifest/manifest-element.html#vcode

So it looks like it is just a number, such that the release with the highest available value should be the one offered to the user.

Comment 7

[Pete Moore [:pmoore][:pete]]

Callek has also pointed out to me that nightly and aurora-nightlies are served by aus, and beta/release are served by Play Store. That explains why there are two mechanisms. I also understand bug

Aside: presumably when we have integrated with the Play Store upload API, we would be able to switch nightly and aurora-nightlies also to be play-store-managed, to be consistent.

I can't find any Android builds here:
https://secure.pub.build.mozilla.org/buildapi/self-serve/mozilla-esr31?numbuilds=5000

I was hoping to find arm6, arm7 and x86 Android builds here... Maybe because we've only just cut the esr31 branch?

Tomorrow I'll take apart some published apk's to check how they are bundled. I found, for example:

 * ftp://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/32.0b2-candidates/build1/android-armv6/multi/fennec-32.0b2.multi.android-arm-armv6.apk
 * ftp://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/32.0b2-candidates/build1/android/multi/fennec-32.0b2.multi.android-arm.apk
 * ftp://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/32.0b2-candidates/build1/android-x86/multi/fennec-32.0b2.multi.android-i386.apk

So I see we did build all 3 architectures for 32.0b2, for example. I'll see what is in the App manifests (http://developer.android.com/guide/topics/manifest/manifest-intro.html) and then maybe it will be clearer why we have to adjust the versionCode, and why arm7 devices don't just look at the arm7 manifest, arm6 versions look at the arm6 manifest, and x86 devices don't just look at the x86 manifest.

Brad, would you be able to send me the details (perhaps via email) with how I can log into the play store and publish some fake products under org.mozilla.fennec_blassey? No doubt it will be useful to play around with this in the next couple of days. Many thanks in advance!

Pete

Comment 8

[Pete Moore [:pmoore][:pete]]

s/ I also understand bug// :)

Comment 9

[Brad Lassey [:blassey] (use needinfo?)]

(In reply to Pete Moore [:pete][:pmoore] from comment #7)
> Callek has also pointed out to me that nightly and aurora-nightlies are
> served by aus, and beta/release are served by Play Store. That explains why
> there are two mechanisms. I also understand bug
> 
> Aside: presumably when we have integrated with the Play Store upload API, we
> would be able to switch nightly and aurora-nightlies also to be
> play-store-managed, to be consistent.
I wouldn't assume that, we ship to stores that are not the play store as well.
> 
> I can't find any Android builds here:
> https://secure.pub.build.mozilla.org/buildapi/self-serve/mozilla-
> esr31?numbuilds=5000
> 
> I was hoping to find arm6, arm7 and x86 Android builds here... Maybe because
> we've only just cut the esr31 branch?
> 
> Tomorrow I'll take apart some published apk's to check how they are bundled.
> I found, for example:
> 
>  *
> ftp://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/32.0b2-candidates/
> build1/android-armv6/multi/fennec-32.0b2.multi.android-arm-armv6.apk
>  *
> ftp://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/32.0b2-candidates/
> build1/android/multi/fennec-32.0b2.multi.android-arm.apk
>  *
> ftp://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/32.0b2-candidates/
> build1/android-x86/multi/fennec-32.0b2.multi.android-i386.apk
> 
> So I see we did build all 3 architectures for 32.0b2, for example. I'll see
> what is in the App manifests
> (http://developer.android.com/guide/topics/manifest/manifest-intro.html) and
> then maybe it will be clearer why we have to adjust the versionCode, and why
> arm7 devices don't just look at the arm7 manifest, arm6 versions look at the
> arm6 manifest, and x86 devices don't just look at the x86 manifest.
> 
> Brad, would you be able to send me the details (perhaps via email) with how
> I can log into the play store and publish some fake products under
> org.mozilla.fennec_blassey? No doubt it will be useful to play around with
> this in the next couple of days. Many thanks in advance!

You log in using your mozilla gApps account. RelEng owns managing which accounts have access to our products.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Pete Moore [:pete][:pmoore] from comment #7)
> Callek has also pointed out to me that nightly and aurora-nightlies are
> served by aus, and beta/release are served by Play Store. That explains why
> there are two mechanisms.

I agree with this and blassey's comment. But, bug 1019724 will muddy those waters a bit, adding single locale builds on beta/release which are updated by us. AIUI the multi locale will stay on Google Play, since that's the premier app store for Android. I don't think it makes any difference to this bug.

> I can't find any Android builds here:
> https://secure.pub.build.mozilla.org/buildapi/self-serve/mozilla-
> esr31?numbuilds=5000
> 
> I was hoping to find arm6, arm7 and x86 Android builds here... Maybe because
> we've only just cut the esr31 branch?

We haven't got to that point of turning them on, it depends if we can solve this bug first. (Bug 1042835 turned things off elsewhere.) We'd probably only enable armv6 on mozilla-esr31.

> So I see we did build all 3 architectures for 32.0b2, for example. 

The release config has been updated since then, and there won't be a armv6 build for today's 32.0b4.

Comment 11

[Pete Moore [:pmoore][:pete]]

Cool, I've had a chat with :bhearsum and :catlee and they have clarified the issue for me, which I will now try to summarise in case it is useful for others that stumble on this bug.

When we make a support release for armv6, it will be made off the ESR31 branch.
Independently, we will be making releases for armv7 and x86 of the release branch.

Potentially there are devices, which could "take" either of these builds, e.g. see:
http://mxr.mozilla.org/mozilla-central/source/mobile/android/base/Makefile.in#22
http://mxr.mozilla.org/mozilla-central/source/mobile/android/base/Makefile.in#26

For *a device that supports armv6 plus either armv7 and/or x86* it will have the choice of taking the armv6 release, or one of the others. Both it will see as "compatible". Then it will just be a case that the one with the highest versionCode (http://developer.android.com/guide/topics/manifest/manifest-element.html#vcode) wins.

Therefore, since we only want armv6 devices to "have to" take the less feature-full armv6 version, we want to make sure in such a "competition" between apks, that armv7/x86 builds *win* and that armv6 apks have a lower versionCode and always come in last place in the rankings.

So offsetting e.g. a week is also not sufficient. Imagine we have to make a security fix for armv6 (esr31 branch) in three months time, and it gets a buildid of three weeks' time from now minus one week - that will still be "ahead" of any release we make today on the release branch. Thus, this security fix would come out, and armv7/x86 devices that can run armv6 code would see "ah there is a higher version number out than what I have - I'll take that". Then they would migrate away from the release branch to this older less feature-rich support version. Baaaaad.

So my proposal is this:

There is no requirement that the apk versionCode is based on the build timestamp. Indeed, it is just treated as an integer, and of all the compatible builds as potential upgrades - the one with the highest integer will be upgraded to. Therefore let's adapt the model used for armv6 - we take the latest versionCode for the most current armv6 release, and increment it by *1* for each release that we make. We can use e.g. a database sequence which starts at the most recent release's versionCode plus 1.

The versionCode is 10 characters, when taken from the buildid includes the year, month, day and *hour* of the release. In other words, every new release we make from the ESR branch, we are bumping the apparent "time" of the ESR release forward by one hour. Therefore even if we make 24 releases, the apparent buildid shown in the versionCode will only be one day later, so as long as we release from release branch at least one day after making the armv6 release, we still have the opportunity to deliver 24 security fixes *at any time in the future* on the ESR branch, without the versionCode ever hitting the value in the release branch release.

I hope this makes sense.

If not, I can attach a doc where I fill in a dummy scenario with made-up buildids/versionCodes etc.

Can't ask for feedback on a bug comment, so doing a "needinfo" instead - guys let me know if you are happy with this strategy! :D

Pete

Comment 12

[Pete Moore [:pmoore][:pete]]

Regarding incrementing a counter for the armv6 versionCode, we would need a way to discover the most recently used version. Options I see are:

 1) committing a file during the release process of building the apk, which contains the latest versionCode, so the next time we can "add one" to it
 2) having a web service that returns an incrementing counter (like "page hits") - but would need to only be accessible from build network so it did not get sabotaged
 3) manually checking in a file prior to making release (same as 1) but manual)
 4) querying the playstore to find out latest versionCode for armv6, and incrementing
 5) storing in logs, and parsing logs from e.g. ftp for previous run
 6) is there already a mechanism to pass an extra parameter in from ship it, and relman to do it manually? (error prone and risky)

The simplest I guess is number 1).

Comment 13

[Pete Moore [:pmoore][:pete]]

Started work on this here: https://github.com/petemoore/gecko-dev/compare/esr31

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Pete Moore [:pete][:pmoore] from comment #11)
> So offsetting e.g. a week is also not sufficient. Imagine we have to make a
> security fix for armv6 (esr31 branch) in three months time, and it gets a
> buildid of three weeks' time from now minus one week - that will still be

typo - s/three weeks/three months/

> "ahead" of any release we make today on the release branch. Thus, this
> security fix would come out, and armv7/x86 devices that can run armv6 code
> would see "ah there is a higher version number out than what I have - I'll
> take that". Then they would migrate away from the release branch to this
> older less feature-rich support version. Baaaaad.

I think it's fairly unlikely that we want to build an armv6 build only; don't recall having ever done that before because we want a fix in all builds. As m-r and m-esr31 diverge it's possible it will become an issue, either through issues avoided via structural change in m-r, or bad merges to esr31 that require further fixes.

> Therefore let's adapt the model used for armv6 - we
> take the latest versionCode for the most current armv6 release, and
> increment it by *1* for each release that we make. We can use e.g. a
> database sequence which starts at the most recent release's versionCode plus
> 1.

It would be helpful to jump to something implausible like 2014080001, just to make it plain whenever someone looks at the code (harder to spot last digit changing). Then increment from there.

(In reply to Pete Moore [:pete][:pmoore] from comment #12)
> Regarding incrementing a counter for the armv6 versionCode, we would need a
> way to discover the most recently used version. Options I see are:
> 
>  1) committing a file during the release process of building the apk, which
> contains the latest versionCode, so the next time we can "add one" to it
> not get sabotaged
>  3) manually checking in a file prior to making release (same as 1) but
> manual)

I suggest doing 3 as part of tagging. We already have machinery there to bump versions, so it's a short step to incrementing an integer. eg
http://hg.mozilla.org/build/buildbot-configs/file/default/mozilla/release-firefox-mozilla-release.py#l61
http://hg.mozilla.org/build/tools/file/default/scripts/release/tag-release.py

Comment 15

[Pete Moore [:pmoore][:pete]]

(In reply to Nick Thomas [:nthomas] from comment #14)
> (In reply to Pete Moore [:pete][:pmoore] from comment #11)
> > So offsetting e.g. a week is also not sufficient. Imagine we have to make a
> > security fix for armv6 (esr31 branch) in three months time, and it gets a
> > buildid of three weeks' time from now minus one week - that will still be
> 
> typo - s/three weeks/three months/
> 
> > "ahead" of any release we make today on the release branch. Thus, this
> > security fix would come out, and armv7/x86 devices that can run armv6 code
> > would see "ah there is a higher version number out than what I have - I'll
> > take that". Then they would migrate away from the release branch to this
> > older less feature-rich support version. Baaaaad.
> 
> I think it's fairly unlikely that we want to build an armv6 build only;
> don't recall having ever done that before because we want a fix in all
> builds. As m-r and m-esr31 diverge it's possible it will become an issue,
> either through issues avoided via structural change in m-r, or bad merges to
> esr31 that require further fixes.

I wasn't sure from above if you think it makes sense to account for this situation, or if it is overkill. In general, are you happy with the proposal of incrementing an integer, or do you think it would be better to do it differently? I want to do something as less-hacky as possible, but something that covers the use cases as best as possible, not relying too much on human intervention which may introduce risk of errors. At the same time having something non-intuitive can also give rise to human error in troubleshooting/fixing etc. So I appreciate any feedback you have if you can think of a better way to address the problem. This was just my first-stab at what I considered a reasonable solution, but don't have the same war wounds others may have suffered. =)

> 
> > Therefore let's adapt the model used for armv6 - we
> > take the latest versionCode for the most current armv6 release, and
> > increment it by *1* for each release that we make. We can use e.g. a
> > database sequence which starts at the most recent release's versionCode plus
> > 1.
> 
> It would be helpful to jump to something implausible like 2014080001, just
> to make it plain whenever someone looks at the code (harder to spot last
> digit changing). Then increment from there.

Very good point - agree 100%. Practical thinking; like it!

> 
> (In reply to Pete Moore [:pete][:pmoore] from comment #12)
> > Regarding incrementing a counter for the armv6 versionCode, we would need a
> > way to discover the most recently used version. Options I see are:
> > 
> >  1) committing a file during the release process of building the apk, which
> > contains the latest versionCode, so the next time we can "add one" to it
> > not get sabotaged
> >  3) manually checking in a file prior to making release (same as 1) but
> > manual)
> 
> I suggest doing 3 as part of tagging. We already have machinery there to
> bump versions, so it's a short step to incrementing an integer. eg
> http://hg.mozilla.org/build/buildbot-configs/file/default/mozilla/release-
> firefox-mozilla-release.py#l61
> http://hg.mozilla.org/build/tools/file/default/scripts/release/tag-release.py

Thanks for the links, I'll plug in there!

This has been a great bug to learn about the release process. Thanks for your help!

Pete

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Pete Moore [:pete][:pmoore] from comment #15)
> I wasn't sure from above if you think it makes sense to account for this
> situation, or if it is overkill. In general, are you happy with the proposal
> of incrementing an integer, or do you think it would be better to do it
> differently? I want to do something as less-hacky as possible, but something
> that covers the use cases as best as possible, not relying too much on human
> intervention which may introduce risk of errors.

Bumping an integer is fine, if we can coerce the bumping code to do that in a reasonable length of time. In retrospect, it's not immediately obvious to me how to do that without being disruptive to existing code. Maybe something like 201408-31.1, 201408-31.2, can work (ie some prefix with the actual version in it for easy replacement).

I'd much rather try for automation than have a human have to remember to bump manually.

Comment 18

[Pete Moore [:pmoore][:pete]]

From :bhearsum

I think this is something that's quite easy to automate. We have a fairly generic "bumpFiles" list in each release config: https://hg.mozilla.org/build/buildbot-configs/file/production/mozilla/release-firefox-mozilla-esr31.py.template#l50. Those eventually get passed to https://github.com/mozilla/build-tools/blob/master/lib/python/build/versions.py#L19. If the release config is updated, and the BUMP_FILES dict in versions.py is updated, this _should_ just work. It's easy to test independently of release automation, too.

I'm on PTO for the rest of today, but hope to muddle through this on Monday, and have something for testing by the end of the day.

Rail, Coop suggested I contact you and ask if you have a reasonably recent staging environment for testing release changes that I might be able to run this through (or that maybe you might be able to run it through). Might that be a possibility? Hope to get a patch out for review at the beginning of next week (week 18-22 August).

Pete

Comment 19

[Rail Aliiev [:rail]]

Unfortunately I don't have anything fresh, but mgerva may have since he's on the hook for the next beta1.