1137898 - Use a hierarchical pattern for computing version codes

Status: RESOLVED FIXED

(Firefox Build System :: Android Studio and Gradle Integration, defect)

Description

[Richard Newman [:rnewman]]

In Bug 1137586 we discovered that version code computation is fragile, and doesn't really scale well to multiple APKs.

In summary: we should reserve the last N bits of a 32-bit signed integer for something like a build timestamp, and use some of the first N bits to provide an absolute ranking for APKs: processor, then API range/capability division.

1 bit for ARM vs x86 leaves five bits, which allows 32 intra-division APKs. If we want to save one pad bit, 16 sub-APKs.

Making our timestamps more coarse gives us more bits.

Unfortunately, our current version codes occupy all but the sign bit, so we might be doomed to losing most of the first byte to just "later than our current scheme", which sucks.


IRC log:

<@rnewman> the other is that we're doing it so all builds in the same hour get the same 'root'
<@rnewman> which would scare me
<@nalexander> rnewman: by my count, it's yyyymmddhh
<@nalexander> rnewman: oh god, you're right.
<@nalexander> rnewman: the stamps are based on the builder picking up the job, IIRC.
<@rnewman> so we trust that x86 and ARM go to build at about the same time
<@nalexander> rnewman: God this is so fragile.
<@rnewman> so the "add the minSDKVersion" actually makes it less fragile
<@rnewman> but our ability to restrict fragility is limited by builds per day
<@rnewman> e.g., we should really add 21 for x86
<@rnewman> or we can assume that we won't ship Fennec for 10,000 years, and we should add a leading '1' for x86
<@nalexander> rnewman: yeah, that was my thinking.  We're doing this relative shifting within a build, but we have to cmpare across builds.
<@nalexander> rnewman: I thought about doing that (digits correspond to features) but assumed we had a reason not to.
<@rnewman> this logic should really be broken up into fields in a 32-bit int
<@nalexander> rnewman: yes.
<@nalexander> rnewman: that's where I started.
<@rnewman> so... we can do that if you want, but we have to make sure it works, is documented and future-proof, and we can land it so we can ship Beta 37
<@nalexander> rnewman: looks like the version code is freeform; Google suggests mapping into upper and lower 16 bits here: http://developer.android.com/guide/topics/manifest/manifest-element.html#vcode
<@rnewman> aye
<@nalexander> rnewman: well, I'm cool to ship what you have for now.  We're trying to do this yesterday, right?
<@rnewman> YYYYMMDDHHMM is already >32
<@nalexander> rnewman: oh, right.
<@rnewman> dropping minutes gets us into 32bit
<@rnewman> if we switched to Unix timestamps we can buy some bits
<@nalexander> rnewman: looks like that's a bad idea: http://developer.android.com/reference/android/content/pm/PackageInfo.html#versionCode
<@nalexander> rnewman: expects an int.
<@nalexander> rnewman: so that's why cut 10.
<@rnewman> yeah, so we get a signed 32-bit int
<@rnewman> which buys us second-precision for unix timestamps
<@rnewman> if we truncate that to minutes (now = 23751212), we have 7 bits free before sign
<@rnewman> sorry, 6
<@rnewman> so we could *gain* granularity and buy 6 bits for ranking by using a non-lexical date format
<@nalexander> rnewman: but we have legacy versions, so we can't go backwards too far.
<@nalexander> rnewman: we'd lose a bit to bump ahead of legacy, wouldn't we?
<@nalexander> rnewman: in any case, I think we should push on with your patch right now.
<@rnewman> :/
<@rnewman> yeah, current version code is 2015022632, so we'd need to gain a digit or two by padding out those flag bits
<@nalexander> rnewman: could you file a ticket for actually doing this in a sensible manner?
<@rnewman> sure
<@nalexander> rnewman: the thing that's loopy is that we're not keying the version code by anything stable; it's based on "when the builder picks up the job".
<@nalexander> rnewman: god forbid we trigger a build at midnight.
<@rnewman> at an hour boundary!
<@nalexander> rnewman: oh, right, even worse.
<@rnewman> day boundary = x86 users are fucked

Comment 1

[Nick Alexander :nalexander [he/him]]

If and when the time comes, some of these free bits should go into other attributes including device screen size (DPI).  See the discussion at http://tools.android.com/tech-docs/new-build-system/user-guide/apk-splits.

Comment 2

[Richard Newman [:rnewman]]

Taking 2015070105 as the number we have to beat:

0111 1000 0001 1011 1000 0111 1001 1001

0111 1                                   -- fixed to be larger than we have now.
      000 0001                           -- at least one has to flip
               1011 1000 0111 1001 1001  -- 20 bits to use for individual builds.


That gives us 1,048,575 possible individual build events, with about sixty potential categories (or the ability to 'buy' new build events).

The trick is mapping build instants into build events in a monotonically increasing way.

We could take the current timestamp (1433868032715), subtract a fixed instant (1433868000000), and convert to minutes; that gives us 728 days, which isn't enough.

We could add another bit to get us up to four years in this scheme, or we could alter granularity.

I'm keen to get this ball rolling, because when we flip over to 2016 we're going to lose another bit at the top end.

Nick, thoughts?


(Why not just flip the top bit? I hear you ask. Because Android parses this as an int, and does so with parseInt. That means a signed 32-bit int, so flipping the top bit gives us negative versions. Yay.)

Comment 3

[Nick Alexander :nalexander [he/him]]

Attachment: MozReview Request: Bug 1137898 - Migrate to android:versionCode scheme v1. r?rnewman,snorp

Bug 1137898 - Pre: Remove unused UA_BUILDID. r=rnewman

Comment 4

[Nick Alexander :nalexander [he/him]]

Attachment: MozReview Request: Bug 1137898 - Pre: Remove unused MIN_CPU_VERSION. r=rnewman

Bug 1137898 - Pre: Remove unused MIN_CPU_VERSION. r=rnewman

We migrated to use MOZ_MIN_CPU_VERSION in moz.build some time back.

Comment 5

[Nick Alexander :nalexander [he/him]]

Attachment: MozReview Request: Bug 1137898 - Part 1: Compute Android version codes in Python. r=gps

Bug 1137898 - Part 1: Compute Android version codes in Python. r=gps

I considered three ways to do this:

* one, as a Python script executed with $(shell);
* two, as a Python script that writes an include file for the preprocessor;
* three, as a function exposed to the moz.build sandbox.

I rejected two because it's both tied to the preprocessor, and awkward
to make handle the dependency on the buildid (in a file) and
additional build defines (in config.status).

I rejected three because I know of no precedent for this approach, and
it hides the dependency on the buildid.

One doesn't handle failures in the script gracefully, but neither did
the existing approach.  This patch is at least testable.

Comment 6

[Nick Alexander :nalexander [he/him]]

rnewman: I pushed some build/Python bits to make this easier to do the actual work on top of.  I'll try to get to it myself but you could jump in here and implement the actual approach...

Comment 7

[Gregory Szorc [:gps]]

Comment on attachment 8631190 [details]
MozReview Request: Bug 1137898 - Part 1: Compute Android version codes in Python. r=gps

https://reviewboard.mozilla.org/r/12853/#review11585

The build system bits look fine. I didn't really pay too close attention to the versioning foo: you should ask rnewman to verify this behaves as it is supposed to.

Comment 8

[Nick Alexander :nalexander [he/him]]

(In reply to Richard Newman [:rnewman] from comment #2)
> Taking 2015070105 as the number we have to beat:
> 
> 0111 1000 0001 1011 1000 0111 1001 1001
> 
> 0111 1                                   -- fixed to be larger than we have
> now.
>       000 0001                           -- at least one has to flip
>                1011 1000 0111 1001 1001  -- 20 bits to use for individual
> builds.
> 
> 
> That gives us 1,048,575 possible individual build events, with about sixty
> potential categories (or the ability to 'buy' new build events).

I think you're taking the 6 "second highest" bit for those "about sixty", right?  This scares me.  I want the "feature bits" to be low order bits, so that we can remap what the feature bits mean over time.  That is, I want:

SIGIL | DATE | FEATURE

If we do instead

SIGIL | FEATURE | DATE

then we might find an obsolete feature always winning the race.

> The trick is mapping build instants into build events in a monotonically
> increasing way.
> 
> We could take the current timestamp (1433868032715), subtract a fixed
> instant (1433868000000), and convert to minutes; that gives us 728 days,
> which isn't enough.

We could do mathematically interesting things, too: since most Nightly builds take place at 2am PST, we could dilate the granularity throughout the day.  That might give us minutes where it counts, and hours where it doesn't.  Tricky over time, though.

> We could add another bit to get us up to four years in this scheme, or we
> could alter granularity.

I think we should consider granularity by release channel.  We expect Release to update more slowly than Beta, etc.  We could chunk by day, I expect, for Release.  (Historical data might be available to guide us here.)  Perhaps by 15 minutes for Nightly.  Perhaps by minute locally.

There's a nice discussion at https://software.intel.com/en-us/blogs/2012/11/12/how-to-publish-your-apps-on-google-play-for-x86-based-android-devices-using, which gives me another idea.  We have a naming pattern for release builds (38.0, 38.0.1, etc).  Do we have a naming pattern for beta builds (38.0.rc1, 38.1.rc1)?  We could go away from the build ID and encode the actual version information in many fewer bits.  For release, and beta, perhaps; this breaks down for Aurora and Nightly builds.

> I'm keen to get this ball rolling, because when we flip over to 2016 we're
> going to lose another bit at the top end.

Yes.

> (Why not just flip the top bit? I hear you ask. Because Android parses this
> as an int, and does so with parseInt. That means a signed 32-bit int, so
> flipping the top bit gives us negative versions. Yay.)

This sucks.  Just one more bit would save us easily.

Comment 9

[Kevin Brosnan [Ex-Mozilla]]

Release builds have "builds", ex 36.0 build 1, 36.0 build 2 however in product there is no knowledge of the two builds. ftp://ftp.mozilla.org/pub/mobile/candidates/archived/36.0-candidates/ multiple builds is rather common for release builds.

Beta builds look like 36.0b1 build 1. Though once again in product has no knowledge if there was more than one build attempt. Betas 1, 2, 4, 6, 8, and 10 are the normal builds for mobile. With enough reason there can be odd betas or numbers above 10.

Comment 10

[Richard Newman [:rnewman]]

(In reply to Nick Alexander :nalexander (PTO July 17-27) from comment #8)

> I think you're taking the 6 "second highest" bit for those "about sixty",
> right?  This scares me.  I want the "feature bits" to be low order bits, so
> that we can remap what the feature bits mean over time.  That is, I want:

I think you're right. I didn't put too much importance on in which sequence to put the groups of bits; the important thing is that there's a strict ordering.

(It's an interesting point, too. I think it comes down to: are these features or are they requirements that partition the version space? Does it make sense to obsolete one of these partitions? Do we stand a chance of ordering them and EOLing them correctly?)

Comment 11

[Richard Newman [:rnewman]]

(In reply to Nick Alexander :nalexander (PTO July 17-27) from comment #8)

> I think we should consider granularity by release channel.  We expect
> Release to update more slowly than Beta, etc.  We could chunk by day, I
> expect, for Release.  (Historical data might be available to guide us here.)
> Perhaps by 15 minutes for Nightly.  Perhaps by minute locally.

To state the unstated assumption: version codes don't need to compete across channels, because our channels are partitioned by id (e.g., org.mozilla.firefox_beta).

Indeed, more broadly we don't need a single version code scheme at all; it can vary by ID.

One useful consequence of this: we could, if necessary, manually assign version codes to each release (and beta) build.

If we never ship more than one release or beta per day (or even per hour), we have quite a lot of bits. 


> We have a naming pattern for release builds (38.0, 38.0.1, etc).  Do
> we have a naming pattern for beta builds (38.0.rc1, 38.1.rc1)?

This is a kind of manual version code assignment. I'd be a little concerned about encoding these rules too firmly, because rules are human and they change.



> This sucks.  Just one more bit would save us easily.

\o/ android \o/

Comment 12

[Richard Newman [:rnewman]]

Comment on attachment 8631188 [details]
MozReview Request: Bug 1137898 - Migrate to android:versionCode scheme v1. r?rnewman,snorp

https://reviewboard.mozilla.org/r/12849/#review11789

Ship It!

Comment 13

[Richard Newman [:rnewman]]

Comment on attachment 8631189 [details]
MozReview Request: Bug 1137898 - Pre: Remove unused MIN_CPU_VERSION. r=rnewman

https://reviewboard.mozilla.org/r/12851/#review11791

Ship It!

Comment 14

[Richard Newman [:rnewman]]

Comment on attachment 8631190 [details]
MozReview Request: Bug 1137898 - Part 1: Compute Android version codes in Python. r=gps

https://reviewboard.mozilla.org/r/12853/#review11793

Ship It!

Comment 15

[Richard Newman [:rnewman]]

Flipping these dependency bits.

Comment 16

[Richard Newman [:rnewman]]

(In reply to Nick Alexander :nalexander (PTO July 17-27) from comment #6)
> rnewman: I pushed some build/Python bits to make this easier to do the
> actual work on top of.  I'll try to get to it myself but you could jump in
> here and implement the actual approach...

Most likely this is on you; at the very least if I built it it would need review by you, so not sure that would save us :)

If I get cramps from bookmark sync, I'll let you know.

Comment 17

[Nick Alexander :nalexander [he/him]]

rnewman: the following is a Python comment, with some Emacs-lisp for inline evaluation.  And some hand editing.  My analysis diverges from yours; please consider carefully and correct me.

'''(defun b (x)
  (require 'calc-bin) (let ((calc-number-radix 2)) (concat "0" (math-format-radix x))))

Taking 2015090101 as the number we have to beat:

(b 2015090101) "0111 1000 0001 1011 1101 0101 1011 0101"

it's worth noting that flipping the next high order bit gives

(b 2015363072) "0111 1000 0010 0000 0000 0000 0000 0000"

Since we use the first 10 digits of the build ID as the base right now
(YYYYMMDDHH), the current scheme loses a bit only in 2016.  That is:

(b 2015123123) "0111 1000 0001 1100 0101 0110 1011 0011" and
(b 2016010100) "0111 1000 0010 1001 1101 1111 0111 0100"

The new scheme is as follows:

The top bit must be 0, since Android version codes are 32-bit *signed* integers,
which in Java means the top bit is the sign bit.

Keep the next 4 bits as 1111, since that's what we already have.

That leaves 32 - 1 - 4 = 27 bits to play with.  However, only the first 5 bits
are open -- the 6th bit must be 1, to be at least as large as 2015090101.

The high order bits are *valuable*, since they partition the space of builds.
That is, they can correspond to "version scheme changes" like the change from
the old to the new build versioning scheme.  They are our get of jail cards!

The most conservative assignment is therefore

0111 1000 0010 0000 0000 0000 0000 0000

Which leaves us with the low order 21 bits to play with.

If we go for hourly builds, we get

(/ (expt 2 21) (* 24.0 366))

238.74681238615665 years of hourly builds.

A finer build resolution would be *nice*, but it doesn't need to be too fine --
5, 10, 30 minutes.

A few low order bits for "feature flags" would be nice too.  Feature flags don't
partition the space of builds; they partition the space of each build.  They
correspond to our existing x86/ARM split, and the Android SDK version APK splits
(API 11+/API 9-10).

If we took 3 bits for feature flags, we'd have

(/ (expt 2 21) (* 24.0 366 (expt 2 3))) 29.84335154826958

years of hourly builds with 3 bits of feature flags.  (The old scheme uses only
2 bits for x86/API 11+/API 9-10: adding 3 and assuming that builds only occur
every 4 hours is equivalent to flipping the bottom two low order bits.)

If we instead took half-hourly builds and took 4 bits for feature flags/APK splits, we'd have 

(/ (expt 2 21) (* 2 24.0 366 (expt 2 4))) 7.460837887067395

years of builds.  I think that's probably conservative -- we hope Fennec to
exist and still ship in 7+ years, but we expect to have to make a version scheme
change before then.  Unfortunately, 3.75 is a little too close to the present
for my comfort.

The maximally conservative scheme would be to take

0111 1000 001x x000 0000 0000 0000 00yy

with the xx bits as 0 for now, and the yy bits as the flags in the current
scheme.  That leaves 17 bits for builds, which is

(/ (expt 2 17) (* 2 24.0 366)) 7.460837887067395

years of half-hourly build events, with the current flags, *and* the option to
add a flag without losing a high order bit.  If we ran through the allotted ~7.5
years, we could always take an additional bit for build IDs.  Since there are
two such xx bits, we get up to ~30 years of half-hourly build events.  After 25
years our Android signing certificate will expire, so we're definitely due for a
version scheme change before then!
'''

Comment 18

[Richard Newman [:rnewman]]

This analysis makes sense to me.

I propose two changes:

1. Use hourly builds, with four reserve bits, for MOZ_OFFICIAL builds. That gives us 15 years, but more importantly it gives us seven years to figure out if we need the high bit as an additional flag!

I do not expect us to cut *and ship* new release builds to Play more than once per hour, because it takes three hours for them to propagate. Worst-case we can fake it to the next hour if needed.

2. Use two high flag bits and no low flag bits for local builds. There should be no need for local builds to have tie-break local flags, so instead we can buy extra granularity in the timestamp.

So we'd have JS:


function hoursSince20150901(millisSince1970) {
  let zero = 1441090800000;

// Uncomment this to pretend we switched August 19th.
//  let zero = 1440040800000;

  let since = millisSince1970 - zero;
  let hours = since / (1000 * 60 * 60);
  return hours;
}

function officialBuildID() {
  // Bitshift implies floor.
  return 2015363072         // 0111 1000 0010 0000 0000 0000 0000 0000
       | hoursSince20150901(Date.now()) << 2;   // Leave two bits at the end.
}

function unofficialBuildID() {
  // Multiply to preserve more precision.
  return 2015363072         // 0111 1000 0010 0000 0000 0000 0000 0000
       | hoursSince20150901(Date.now()) * 4;
}

Comment 19

[Nick Alexander :nalexander [he/him]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=2e9853da63bc

Comment 20

[Nick Alexander :nalexander [he/him]]

Comment on attachment 8631188 [details]
MozReview Request: Bug 1137898 - Migrate to android:versionCode scheme v1. r?rnewman,snorp

Bug 1137898 - Migrate to android:versionCode scheme v1. r?rnewman,snorp

Android version codes serve multiple masters.  They indicate newer
versions, yes; but they also break ties between versions with
different features and requirements. High order bits effectively
partition the space of versions and are valuable.  Since Android
version codes are signed Java integers, we have 31 bits to work with.
Mozilla's traditional build ID is YYYYMMDDhhmmss.  This was chopped to
ten characters (YYYYMMDDhh, i.e., hourly build IDs) and then converted
to a decimal.  This took many high order bits.  We will lose another
high order bit in the 36th month of 2015 -- i.e., as soon as the year
rolls over to 2016.  If we waited to lose the next higher order bit,
we'd lose that one in the 46th month of 2017 -- i.e., as soon as the
year rolls over to 2018.

The following patch sacrifices a high order bit to change the version
scheme, winning us roughly 15 years of hourly build IDs before we are
forced to lose another high order bit.  So it's clearly to our
advantage to change the scheme sooner rather than later -- we will
sacrifice 1 bit for 15 years of build IDs, rather than keeping the
current scheme and sacrificing (say) 2 bits for 3 years of build IDs.

The resulting scheme produces build IDs that look like (in binary):

0111 1000 0010 xxxx xxxx xxxx xxxx xyzw

The meaning of these build IDs is documented in the Python source code
that generates them.

Comment 21

[Nick Alexander :nalexander [he/him]]

Comment on attachment 8631188 [details]
MozReview Request: Bug 1137898 - Migrate to android:versionCode scheme v1. r?rnewman,snorp

gps: could you look at this with your experienced Python eye?  Thanks!

Comment 22

[James Willcox (:snorp) ([e-mail]) (he/him)]

Comment on attachment 8631188 [details]
MozReview Request: Bug 1137898 - Migrate to android:versionCode scheme v1. r?rnewman,snorp

https://reviewboard.mozilla.org/r/12849/#review15609

My main concern here is that we will end up needing more 'flags'. For instance, we will probably ship a aarch64 build next year. Or maybe we will do another APK split. How would those affect this? Would we just have to sacrifice more high-order bits? How many of those can we spare?

Comment 23

[Nick Alexander :nalexander [he/him]]

https://reviewboard.mozilla.org/r/12849/#review15609

We have options.  The nuclear option is to take a real high order bit.  We can only do that 2^5 = 32 times.

The second-most nuclear option is to take the highest middle-order bit, effectively changing the versioning scheme but not losing the most valuable bits.  That's less awesome because you can't roll back from the flagged builds.  For example, if we used the reserved middle bit for aarch64 (so it wins over both x86 and ARM) we couldn't transition aarch devices back to ARM automatically.

We have a free flag bit (low order bit) for feature flags or new architectures right now.  We can re-work these low order bits as necessary.

(This is all discussed in the Python code comment.)  Counter suggestions appreciated, but there are no obvious wins: I took 3 of 4 low order bits for flags, assuming that APK splits (and arch tie breakers) were more likely than version changes.  We could take 4 of 4 low order bits (and we have a 5th high-order bit for 7.5 years, if we really need it).  Hard to think it's better to not leave a guaranteed get of jail card.

Comment 24

[Richard Newman [:rnewman]]

https://reviewboard.mozilla.org/r/12847/#review15789

This looks good to me, but I have some suggestions.

::: python/mozbuild/mozbuild/android_version_code.py:38
(Diff revision 2)
> +    0111 1000 0010 xxxx xxxx xxxx xxxx xyzw

Might I suggest

0111 1000 0010 tttt tttt tttt tttt txpg

't' for time, leaving us 'x' for x86/ARM, 'p' for placeholder, and 'g' for Gingerbread?

::: python/mozbuild/mozbuild/android_version_code.py:45
(Diff revision 2)
> +    The bits labelled 'y', 'z', and 'w' are feature flags.  

Nit: trailing ws.

::: python/mozbuild/mozbuild/android_version_code.py:63
(Diff revision 2)
> +    N.B.: it the reserved 0 bit to the left of the highest order 'x' bit can,

s/it the/the

::: python/mozbuild/mozbuild/android_version_code.py:67
(Diff revision 2)
> +    without losing a (valuable) high order bit.

Add a sentence: "The important consideration is that build IDs be monotonically increasing for all published builds."

::: python/mozbuild/mozbuild/android_version_code.py:95
(Diff revision 2)
> +    version = 0b1111000001 << 21

Can we just spell this whole thing out so it matches the comment?

::: python/mozbuild/mozbuild/android_version_code.py:115
(Diff revision 2)
> +        raise ValueError("Don't know how to compute android:versionCode "

This raises an interesting point: the sub-bits are really scoped by the higher-order bits.

For example, x86 isn't split, and can support a different API range -- Bug 1062537.

So I think lines 109-116 should move to line 102, inside the conditional. That means if we deprecate Honeycomb support for x86 earlier than for ARM, we don't confuse all of this.

::: python/mozbuild/mozbuild/test/test_android_version_code.py:22
(Diff revision 2)
> -        self.assertEqual(android_version_code(buildid, cpu_arch='x86', min_sdk=9, max_sdk=None), x86_api9)
> +        self.assertEqual(android_version_code_v0(buildid, cpu_arch='x86', min_sdk=9, max_sdk=None), x86_api9)

Double-check: I think max_sdk might be set to 999.

Comment 25

[Gregory Szorc [:gps]]

Comment on attachment 8631188 [details]
MozReview Request: Bug 1137898 - Migrate to android:versionCode scheme v1. r?rnewman,snorp

https://reviewboard.mozilla.org/r/12849/#review15897

You don't need a build peer review on this despite the file path. I defer to rnewman. But flag me again if you want someone actively doing Python to give it a sanity check.

Comment 26

[Richard Newman [:rnewman]]

Back to Nick for my comments, then. Thanks, Greg!

Comment 27

[Pulsebot]

https://hg.mozilla.org/integration/fx-team/rev/5f17da82fc01

Comment 28

[Nick Alexander :nalexander [he/him]]

Landed with rnewman's comments addressed.

sylvestre: you have done a lot of work for the mobile team getting Fennec into the Play Store.  I'd like you to be aware of this change, which could impact our play distribution; and I'd like you to connect me with the current release manager(s), so they are aware as well.

Comment 29

[Nick Alexander :nalexander [he/him]]

Comment on attachment 8631188 [details]
MozReview Request: Bug 1137898 - Migrate to android:versionCode scheme v1. r?rnewman,snorp

Approval Request Comment
[Feature/regressing bug #]: none.

[User impact if declined]: slower path to get this version code stuff to the release channel.  I see that Firefox 43 hits the release channel on 2015-12-15 at the earliest.  If something were to go wrong (and not be seen in Beta), that would give us 2 weeks before we lost a high-order bit.  I'd be a lot more comfortable having this in Firefox 42, having it hit release with 2 months to fix our high-order bit.

[Describe test coverage new/current, TreeHerder]: manual.

[Risks and why]: there are definitely risks.  For Nightly and Aurora, we'd serve updates that get rejected on the device.  The rejections might be silent; that would be unfortunate.  Presumably testers would notice that the app was stale eventually.  In the Google Play store, we'd see no updates at all, since Google would not push an older APK version to devices.

All such risks are addressed the same way: push an APK with a larger version number.  Given how conservative we've been, we're not concerned about that nuclear option, if it is needed.

[String/UUID change made/needed]: none.

Comment 30

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/5f17da82fc01

Comment 31

[Nick Alexander :nalexander [he/him]]

Checking status:

From the first mozilla-central build (non-Nightly) with the patch:

http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android-api-11/1441278445/mozilla-central-android-api-11-bm73-build1-build101.txt.gz

DANDROID_VERSION_CODE=2015369441
-DMOZ_APP_BUILDID=20150903040725

From the last few Nightlies without the patch:

http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2015/09/2015-09-03-03-02-25-mozilla-central-android-api-11/mozilla-central-android-api-11-nightly-bm74-build1-build2.txt.gz

-DMOZ_APP_BUILDID=20150903030225
-DANDROID_VERSION_CODE=2015090314

http://ftp.mozilla.org/pub/mozilla.org/mobile/nightly/2015/09/2015-09-02-03-02-29-mozilla-central-android-api-11/mozilla-central-android-api-11-nightly-bm74-build1-build1.txt.gz

-DANDROID_VERSION_CODE=2015090214
-DMOZ_APP_BUILDID=20150902030229

From this, we can clearly see the change in version code formatting, and we can see that we have increased the version code.  Everything's looking good so far.

Comment 32

[Sylvestre Ledru [:Sylvestre]]

Comment on attachment 8631188 [details]
MozReview Request: Bug 1137898 - Migrate to android:versionCode scheme v1. r?rnewman,snorp

OK, let's try that then! We will be watching potential regressions.
Thanks for the ni.

Comment 33

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/648c31e78168

Comment 34

[Johan Lorenzo [:jlorenzo]]

Hello there!

This numbering scheme was discussed in [1] and :nalexander suggested we archive the result of this conversation back in Bugzilla.

The original issue was that the scheme doesn't allow 2 nightlies to be shipped within the same hour. I don't think this is a big problem since it's the first time we hear about it in 7 years.

Here are the important comments of that GH issue:

ncalexan commented at 2022-10-07T21:12:30Z

Thank you so much for digging that bug out, @kbrosnan!

If I understand this sentence correctly [...], we will be running out of numbers in July or August 2030.

I might have misunderstood the sentence after all. This part of bug 1137898 comment 20 is interesting:

The following patch sacrifices a high order bit to change the version
scheme, winning us roughly 15 years of hourly build IDs before we are
forced to lose another high order bit. So it's clearly to our
advantage to change the scheme sooner rather than later -- we will
sacrifice 1 bit for 15 years of build IDs, rather than keeping the
current scheme and sacrificing (say) 2 bits for 3 years of build IDs.

Hey @ncalexan 👋 I know we're half-way through the 15 years and there is no rush here. That said, I prefer to ask now: do we have to do anything once first period of 15 years is over and we lose another bit? In other words, will we have to change the number scheme to accommodate the loss of a high order bit?

Hi @JohanLorenzo -- sorry for the delayed reply: travel, the All-Hands week, and my current project, have made life very busy :)

Let me answer the immediate question first. No, we do not have to change the numbering scheme. When we get close to running out of the existing 17 bits, we can elect to overflow into the reserved high-order 0 bit and continue more or less as we are. This would give another 2**17 / (366 * 24) = ~14.92 years of hourly build IDs.

Can we do better? I think we probably can, if we can incorporate some additional information (from outside the build proper) into the calculation. Two thoughts here:

1. We could make the version numbers dense by having a service that provided "next version numbers".  This would give, say `2**17` builds.  Were we to do say 4 Nightly builds a day, that would give us roughly `2 ** 17 / (366 * 4) ~= 60` years of builds.  However, builds are no longer "isolated" in the manner that Mozilla has historically espoused.

2. We could make the version numbers denser, but not maximally dense, by incorporating, say, days since January 1, 2030 and then a daily build counter.

There's some continuum here where we get a dense build number sequence but require centralization through denser but still sparse requiring less centralization all the way to our current sparse build number sequence that requires no centralization (but must discretize the build number space, leading to the "one build per hour" issue).

Now, do we need to do better? I expect not: Android has accommodated additional high order bits with versionCodeMajor in Android 28. I have no idea how quickly we could depend on Android 28, but I'd be inclined to do the simple thing: continue to exhaust our bit space with the current scheme for 6 years, evaluate, and then take our next 15 years, then transition to versionCodeMajor=1 (assuming Android 28 is reasonable, which in 20 years, I expect it will be).

Oh: when we roll into the high-order bit, we could also take some of the unused lower order bits -- perhaps the x x86 bit and the g API version bit and have a larger space.

jlorenzo commented at 2022-10-12T10:10:44Z

Thank you for your thorough answer, Nick!

TIL versionCodeMajor! I definitely agree on your latter strategy. Android API 28 was branded Android 9 (Pie), which was released 4 years ago and Google no longer provides security updates for it. I sincerely hope Firefox will be past that Android version 6 years from now 🙂

In any case, thank you for the details about what we could do instead. At the moment, I believe we keep the current numbering scheme since scheduling 2 nightlies during the same hour occurs rarely. I'm going to close this issue, but feel free to reopen if you all disagree 🙂

[1] https://github.com/mozilla-mobile/fenix/issues/27197