1045304 - B2G Updates are broken when transitioning off Mozilla-Aurora to versioned Gecko

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[John Ford [:jhford] CET/CEST Berlin Time]

When a version of Gecko moves from mozilla-aurora to a versioned gecko branch, we stop getting updates for flames that were on Aurora.

The update channel is baked into the build as 'aurora' before the aurora->b2g32 transition.  The problem is that the last build on the aurora branch stops getting updates automatically until we have new builds on aurora.

An example is v2.0.  The last v2.0 build on aurora was done around July 21 and has an update channel of 'aurora'.  This build hits AUS at: 

https://aus4.mozilla.org/update/3/B2G/32.0a2/20140721000201/flame/en-US/aurora/Boot2Gecko%202.0.0.0-prerelease/default/default/update.xml?force=1

Currently, that gives me <update></update>

When I change the channel in the URL from aurora to nightly-b2g32, like this: 

https://aus4.mozilla.org/update/3/B2G/32.0a2/20140721000201/flame/en-US/nightly-b2g32/Boot2Gecko%202.0.0.0-prerelease/default/default/update.xml?force=1

I get a real update:

<updates>
    <update type="minor" displayVersion="32.0" appVersion="32.0" platformVersion="32.0" buildID="20140728000238">
        <patch type="complete" URL="http://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/2014/07/2014-07-28-00-02-38-mozilla-b2g32_v2_0-flame/b2g-flame-gecko-update.mar" hashFunction="sha512" hashValue="d40265dc1783e303e89d83948eb97c1cd74c8819f350a09b331ef8c9e20bbeb1a67e601958c9ed33587becd7601079896061dc3693e3fd0718f7e5841790231a" size="67033463"/>
    </update>
</updates>


We shouldn't be in a state where users are running really out of date builds for entire cycle.  For this specific case, are we able to put out a one time update that moves people from 'aurora' to 'nightly-b2g32'?

In the future, instead of linking the update channel to the gecko branch name, can we link them to the Firefox OS version?  Maybe we could use a channel name like "nightly-%s" % gaia_branch_name -- example: nightly-v2.0, nightly-v1.3t.

Failing that, can we alias the aurora channel in balrog with nightly-b2g32 until 2.1 goes to aurora?

I've tried using the developer menu to change the update channel, but that's broken for me and I'm filing a bug for that now.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to John Ford [:jhford] -- please use 'needinfo?' instead of a CC from comment #0)
> In the future, instead of linking the update channel to the gecko branch
> name, can we link them to the Firefox OS version?  Maybe we could use a
> channel name like "nightly-%s" % gaia_branch_name -- example: nightly-v2.0,
> nightly-v1.3t.

I think something like this would be better, assuming the config changes we need at merge time are automatable. We need to be aware of gaia version changes (eg 1.5 -> 2.0) which might disrupt this.
 
> Failing that, can we alias the aurora channel in balrog with nightly-b2g32
> until 2.1 goes to aurora?

Yes, probably. What do you think bhearsum ?

Comment 2

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #1)
> (In reply to John Ford [:jhford] -- please use 'needinfo?' instead of a CC
> from comment #0)
> > In the future, instead of linking the update channel to the gecko branch
> > name, can we link them to the Firefox OS version?  Maybe we could use a
> > channel name like "nightly-%s" % gaia_branch_name -- example: nightly-v2.0,
> > nightly-v1.3t.
> 
> I think something like this would be better, assuming the config changes we
> need at merge time are automatable. We need to be aware of gaia version
> changes (eg 1.5 -> 2.0) which might disrupt this.

If we're going to have updates track b2g versions, I agree.

> > Failing that, can we alias the aurora channel in balrog with nightly-b2g32
> > until 2.1 goes to aurora?
> 
> Yes, probably. What do you think bhearsum ?

Also doable.


It seems I may have misunderstood the lay of the land when I transitioned us to Balrog. I assumed that like Desktop, there are B2G users that want to track a specific channel. Ie, stay permanently on "nightly" or "aurora", regardless of what version it may be. If that's not a thing we care about, the above sounds fine. If we do care about this type of use case, things are a lot trickier because it's not possible to distinguish between users who want a channel and those who want a specific version.

Comment 3

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Ben Hearsum [:bhearsum] from comment #2)
> It seems I may have misunderstood the lay of the land when I transitioned us
> to Balrog. I assumed that like Desktop, there are B2G users that want to
> track a specific channel. Ie, stay permanently on "nightly" or "aurora",
> regardless of what version it may be. If that's not a thing we care about,
> the above sounds fine. If we do care about this type of use case, things are
> a lot trickier because it's not possible to distinguish between users who
> want a channel and those who want a specific version.

I think your assumption was a valid one based on how channels work for desktop.  I'm not really the person to decide whether updates track a channel or a version.  I do know that I personally prefer to track updates by version number.

Comment 4

[Kevin Grandon :kgrandon]

Hey guys - Who can alias the aurora channel to nightly-b2g32 for the time being? Several dogfooders are currently blocked from upgrading because of this. Any chance we could do this soon?

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Turns out b2g is different to desktop (shocking, I know), and the app.update.channel setting is stored in defaults/pref/b2g.js, inside of omni.ja, eg 
  pref("app.update.channel", "nightly-b2g32");
in http://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/latest-mozilla-b2g32_v2_0-flame/b2g-32.0.en-US.android-arm.tar.gz. On desktop this file is left out of updates.

Need to verify this, but if we do the alias of aurora to nightly-b2g32, then users will also change channel in the update. Will test to verify. Also neatly avoids the question of what happens at the next merge and aurora starts up again.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Verified the channel is changed as expected, so I've changed the rule for B2G requests with channel aurora to point at the B2G-mozilla-b2g32_v2_0-nightly-latest blob. 

This means users will get transferred to the latest gecko+gaia from mozilla-b2g32_v2_0, then keep up to date as more builds come along.

Comment 7

[[e-mail] (:bhearsum)]

Have we settled on this process going forward? Specifically:
* On merge days that disable aurora updates, B2G aurora updates should be pointed to the new b2g branch that is created at the same time.
* On merge days that re-enable aurora updates, B2G aurora updates should be pointed back to the "aurora" channel.

If so, I'll update the docs and close the bug.

Comment 8

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Ben Hearsum [:bhearsum] from comment #7)
> Have we settled on this process going forward? Specifically:
> * On merge days that disable aurora updates, B2G aurora updates should be
> pointed to the new b2g branch that is created at the same time.

Yes, this sounds right.

> * On merge days that re-enable aurora updates, B2G aurora updates should be
> pointed back to the "aurora" channel.

If the people who were on that aurora that get moved to the versioned channel aren't moved back to the 'new' aurora, this sounds good.  In other words, if I set up the device with aurora that's 2.0, when 2.1 moves to aurora, will I stay on 2.0?

> If so, I'll update the docs and close the bug.

Comment 9

[[e-mail] (:bhearsum)]

(In reply to John Ford [:jhford] -- please use 'needinfo?' instead of a CC from comment #8)
> (In reply to Ben Hearsum [:bhearsum] from comment #7)
> > * On merge days that re-enable aurora updates, B2G aurora updates should be
> > pointed back to the "aurora" channel.
> 
> If the people who were on that aurora that get moved to the versioned
> channel aren't moved back to the 'new' aurora, this sounds good.  In other
> words, if I set up the device with aurora that's 2.0, when 2.1 moves to
> aurora, will I stay on 2.0?

Based on Nick's finding that the update channel pref is included with the MARs, I believe this to be the case:
(In reply to Nick Thomas [:nthomas] from comment #6)
> Verified the channel is changed as expected, so I've changed the rule for
> B2G requests with channel aurora to point at the
> B2G-mozilla-b2g32_v2_0-nightly-latest blob. 
> 
> This means users will get transferred to the latest gecko+gaia from
> mozilla-b2g32_v2_0, then keep up to date as more builds come along.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Sounds good to me too. For the nightly/mozilla-central case we're leaving people where they are, and requiring an explicit action to follow a version ?

Comment 11

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] from comment #10)
> Sounds good to me too. For the nightly/mozilla-central case we're leaving
> people where they are, and requiring an explicit action to follow a version ?

That's what we're doing now. I seem to recall somebody saying that this is fine for nightly, though I can't locate where...

I'm closing this because I'm pretty sure that's right. John, can you confirm?

Comment 12

[[e-mail] (:bhearsum)]

Updated docs:
https://wiki.mozilla.org/ReleaseEngineering/How_To/Adjust_B2G_Update_Channels_on_Merge_Day
https://wiki.mozilla.org/index.php?title=ReleaseEngineering%2FMerge_Duty%2FCentral_will_become_an_odd_numbered_Gecko_version&diff=1007088&oldid=999524
https://wiki.mozilla.org/index.php?title=ReleaseEngineering%2FMerge_Duty%2FCentral_will_become_an_even_numbered_Gecko_version&diff=1007092&oldid=999523

Comment 13

[John Ford [:jhford] CET/CEST Berlin Time]

That sounds right.  Thanks for fixing this.