1310849 - Test that we can successfully update a Mac Universal build to a 64-bit opt build

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris Cooper [:coop] (he/him)]

Since we plan to stop shipping Universal builds soon (bug 1231164), we should test that we are able to update a Universal build to the equivalent 64-bit opt build.

Comment 1

[[e-mail] (:bhearsum)]

Are we talking about verifying that a 64-bit-only MAR applies to a universal build, or that we can configure Balrog to do this? Or both?

Comment 2

[Chris Cooper [:coop] (he/him)]

(In reply to Ben Hearsum (:bhearsum) from comment #1)
> Are we talking about verifying that a 64-bit-only MAR applies to a universal
> build, or that we can configure Balrog to do this? Or both?

Both. I'm reasonably confident in the ability to apply a random MAR, but getting Balrog to serve it up appropriately is the key. Can we redirect 32-bit Mac users to ESR, while simultaneously moving 64-bit users from Universal to 64-bit only?

Comment 3

[[e-mail] (:bhearsum)]

(In reply to Chris Cooper [:coop] from comment #2)
> (In reply to Ben Hearsum (:bhearsum) from comment #1)
> > Are we talking about verifying that a 64-bit-only MAR applies to a universal
> > build, or that we can configure Balrog to do this? Or both?
> 
> Both. I'm reasonably confident in the ability to apply a random MAR, but
> getting Balrog to serve it up appropriately is the key. Can we redirect
> 32-bit Mac users to ESR, while simultaneously moving 64-bit users from
> Universal to 64-bit only?

I think these two things are quite detached from a Balrog perspective. For the 64-bit, part, it shouldn't be a problem, but it's something that we should set-up some test Rules for to verify. We'll want ongoing release channel Releases to have a platforms section like:
"platforms": {
  "Darwin_x86_64-gcc3": { ...pointers to actual bits go here...},
  "Darwin_x86_64-gcc3-u-i386-x86_64": {"alias": "Darwin_x86_64-gcc3"},
}

This will point users on old Universal builds and the 64-bit only ones to 64-bit only ones.

The tricky part about generating Releases will be the tooling. Platform alias' are set-up based on the data in https://github.com/mozilla/build-tools/blob/master/lib/python/release/platforms.py#L22, so we'll need to adjust it in some way. The last time we changed the Mac platform configuration we used a second Buildbot platform: https://github.com/mozilla/build-tools/blob/a3a0fad643cbd95ea05d3c21e33069a976dcb8cc/lib/python/release/platforms.py#L16. Hopefully we can do similar here. We should be very careful to make sure we don't mess up other branches when changing this data structure.


To move 32-bit usrers to ESR, it sounds like we'll want a watershed that points 32-bit users at a channel changing MAR. The rule for that will need to specify Darwin_x86-gcc3-u-i386-x86_64 as its build target. I believe Nick (or maybe Rail) was uncovering some potential issues here around mar channel IDs, so this may be more complicated.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Ben Hearsum (:bhearsum) from comment #3)
> To move 32-bit usrers to ESR, it sounds like we'll want a watershed that
> points 32-bit users at a channel changing MAR. The rule for that will need
> to specify Darwin_x86-gcc3-u-i386-x86_64 as its build target. I believe Nick
> (or maybe Rail) was uncovering some potential issues here around mar channel
> IDs, so this may be more complicated.

Yes, there are three things in play there: channel, channel id, and version. I'm going to assume we are only going to move 32bit-mac users on release to ESR52. A desupport notice would be more appropriate for nightly/aurora/beta, I think. Some ADI numbers, or Balrog log analysis, would useful to check that's valid. We need some blocking rules in place *before* the version bumps though, otherwise we break people.

Channel is the simplest - we swap the users from 'release' to 'esr', to keep the rules simple in Balrog. This would be done by overwriting Contents/Resources/defaults/pref/channel-prefs.js.

The other two are related to metadata stored in the mar file - channel-id and version. Release builds only accept mar files with channel-id of 'firefox-mozilla-release', via
  http://hg.mozilla.org/releases/mozilla-release/file/default/browser/confvars.sh#l51
The equivalent for esr is 'firefox-mozilla-esr'. This shows up in the package at Contents/Resources/update-settings.ini. The updater doesn't allow version downgrades, and checks the version stored on the mar is >= the current version.

It's possible to lie about channel-id and version if you pass -H and -V args to the mar utility when packing the mar file (then sign with the release cert for mars). For version, it would be much much nicer to make sure that users never update to 53 before we try to send them to ESR52 (ie blocking rules)

Worth thinking about the updates from a user point of view:
* for people that are up to date on 51.0.x: a custom mar swaps them to the esr channel and accepts esr mar files. On the next query they find a partial to 52.0esr. Regular esr updates from there
* not up to date users who check for an update: same as above, except it's a complete update

The latter only goes back to the last watershed, I suppose. Do we need to have a first run page to tell users what we've done, and how long they can expect support for ? We could probably do that at the same time as the channel/channel-id change.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

The change in bug 1295375 landed, so I've blocked all mac updates (until we test), see bug 1295375 comment #38. We don't actually have an update to test yet, since CI builds don't create or publish a complete mar. We could wait until the patch is merged to m-c and a nightly is done. If testing fails the strategy could be to back out bug 1295375, respin, re-enable updates.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

I pushed to Oak so that we can get a nightly earlier:
https://treeherder.mozilla.org/#/jobs?repo=oak&revision=aa48e7657d908a4481938f6306ef0a330fb6947c

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

The complete update works properly:
* took the latest universal m-c nightly, which did an update query with [1]
* that downloaded the Oak nightly [2] with buildID 20161208173609
* update applied OK
* Nightly started up OK (testing on a MacBook Pro, Mid 2012), appeared to be working fine on a few websites
* downloaded the dmg [3] and unpacked
* did diff between "m-c updated to oak" and "oak dmg", no differences found   --> SUCCESS

Should test the partial update too, but partial generation is failing for an unrelated reason. Backed out on Oak (6b3623a4ed23) and triggered a universal nightly, then relanded (783b6154bf74) and did another nightly.


[1] https://aus5.mozilla.org/update/6/Firefox/53.0a1/20161208075029/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/nightly-oak/Darwin%2015.6.0/NA/default/default/update.xml
[2] https://archive.mozilla.org/pub/firefox/nightly/2016/12/2016-12-08-17-36-09-oak/firefox-53.0a1.en-US.mac64.complete.mar
[3] https://archive.mozilla.org/pub/firefox/nightly/2016/12/2016-12-08-17-36-09-oak/firefox-53.0a1.en-US.mac64.dmg

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

The two Oak nightlies finished in reverse order to how they started (didn't leave enough time to allow for the longer running universal build). Rerunning the single arch build to generate a partial, I hope.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

mshal and I did some testing with the Oak nightlies, which all worked fine. We checked
* universal to single arch was OK for partial and complete (no differences found between updated build and the dmg from the destination nightly
* partial update worked ok for win64 -> win64

So we've adjusted the rules (id 481 & 482) so that they affect Darwin_x86-gcc3-u-i386-x86_64 only, so the 32bit arch stops at 20161208075029 which 64bit carries on to the latest build. Verified with urls

Old 64bit build --> latest
https://aus4.mozilla.org/update/3/Firefox/53.0a1/20161201000000/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/nightly/Darwin%2014.1.0/default/default/update.xml

Recent 64bit build --> latest (partial & complete at time of writing)
https://aus4.mozilla.org/update/3/Firefox/53.0a1/20161208075029/Darwin_x86_64-gcc3-u-i386-x86_64/en-US/nightly/Darwin%2014.1.0/default/default/update.xml

Old 32bit build --> 20161208075029
https://aus4.mozilla.org/update/3/Firefox/53.0a1/20161201000000/Darwin_x86-gcc3-u-i386-x86_64/en-US/nightly/Darwin%2014.1.0/default/default/update.xml

20161208075029 32bit --> nothing
https://aus4.mozilla.org/update/3/Firefox/53.0a1/20161208075029/Darwin_x86-gcc3-u-i386-x86_64/en-US/nightly/Darwin%2014.1.0/default/default/update.xml