1639199 - Add a custom distribution_id to MacOS 10.9/10.10/10.11 users moved to ESR 78

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, task)

Description

[Romain Testard [:RT]]

MacOS 10.9/10.10/10.11 users on release will be moved to ESR 78 with bug 1637533.
In order to help segment Enterprise users from users moved to ESR as part of an OS deprecation we need to identify these users with a custom distribution_id.

Proposed distribution_id: mozilla-mac-eol-esr1

Comment 1

[Eric Rahm [:erahm]]

Not sure if there's a better spot for distribution_id configuration, moving to the same component as bug 1637533 for now.

Comment 2

[Stephen A Pohl [:spohl]]

I believe this will have to be handled on the build system side of things, not widget:cocoa. I'm not sure what the correct component is however.

Comment 3

[Justin Wood (:Callek)]

Mike, Molly

Are there any concerns/explicit points to be aware of from your seat on this?

Comment 4

[Justin Wood (:Callek)]

Romain, did we ever determine if this is necessary for new downloads as well, or are users updated from release ok to be the only ones with the distribution id set?

Comment 5

[Mike Kaply [:mkaply]]

So the biggest thing from my end is that this will require a custom update MAR to update to this distribution (at least the first time). So we'll need to think about how that is done

Plus the bouncer URL will need to go to the correct version as well.

Comment 6

[Molly Howell (she/her) (mostly inactive)]

Yeah, the only issue I can think of is the same one Mike's pointed out. We'd have to either set that custom MAR as a watershed for affected users, or otherwise keep generating custom MARs for every update until support does end (which I guess means through the ESR 78 cycle).

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Callek has been thinking about a custom mar to change the channel so adding a distribution is probably only an incremental requirement, but I don't recall doing this before. There might be issues we don't know about, eg adding distribution/distribution.ini when distribution/ won't exist (this looks hopeful). The update manifest changes should be fairly straightforward, just some platform detection using an existing file

add-if "platform.ini" "distribution/distribution.ini"
add-if "Contents/Resources/platform.ini" "Contents/Resources/distribution/distribution.ini"

The download case is probably more work to solve. Given where bug 1619353 is at we'd probably have to adjust the automation to create a partner repack, copy the files around, set up bouncer products etc. IIRC we didn't do this for XP on ESR, is there sufficient value to implement here ?

Comment 8

[Romain Testard [:RT]]

(In reply to Justin Wood (:Callek) from comment #4)

Romain, did we ever determine if this is necessary for new downloads as well, or are users updated from release ok to be the only ones with the distribution id set?

Lets not do the distributiuon_id for new profiles because we have very low numbers of MacOS 10.9/10.10/10.11 (about 24k MAU) per https://sql.telemetry.mozilla.org/queries/71673/source#179920 - we'll be able to assume that all new profiles created on 10.9/10.10/10.11 past migration point are consumers.

Comment 9

[Justin Wood (:Callek)]

Attachment: unsigned esr78-test.mar

This is a channel-switch from beta to esr, and sets the distribution ID appropriately via a distribution.ini

This .mar is unsigned, intended to be tested from an early-cycle 77 beta

Comment 10

[Aki Sasaki (not active)]

Attachment: signed esr78_switch_test.mar

Comment 11

[Justin Wood (:Callek)]

Attachment: unsigned esr78-test.mar (take 2)

This sets the changed channel to esr-localtest and it does not yet remove the about string in the distribution.ini since that is only supported on 78 as of b8 (which isn't even built yet)

This mar is targetted as a beta build and not intended to be shipped to users.

Comment 12

[Justin Wood (:Callek)]

Attachment: Signed esr78_switch_test.mar (take 2)

Comment 13

[Justin Wood (:Callek)]

For this (take 2) test I set it up to support a switch from ANY OSX (on beta versions) to explicitly esr-localtest in order to make the manual work simpler.

The attached mar was created to support Linux and Windows as well if it was so desired to test there (for 78 we're not doing that, but leaves us breadcrumbs on how to do this if we desupport another class of OS's) -- I also did NOT build this mar in a way that supports Bug 1644184 - because beta 8 is the first beta that supports a distribution.ini without about (otherwise the distribution.ini is invalid) and I wanted to double check that the attribution information is reported out to balrog on update.

In order to test I setup a github repo to serve update xml out of, because Balrog itself will not tolerate Bugzilla as a venue to serve binaries, and getting this exact mar hosted someplace supported wasn't worth it, since we can't use this exact one for the formal release population.

Steps to test:

• First download (and install) 77.0b8
• Then enable enterprise policies: sudo defaults write /Library/Preferences/org.mozilla.firefox EnterprisePoliciesEnabled -bool TRUE
• Then point updates at my github repo: sudo defaults write /Library/Preferences/org.mozilla.firefox AppUpdateURL -string "https://github.com/Callek/firefox-update-osx-test/raw/master/update/6/%PRODUCT%/%VERSION%/%BUILD_ID%/%BUILD_TARGET%/%LOCALE%/%CHANNEL%/os_version/system_capabilities/%DISTRIBUTION%/%DISTRIBUTION_VERSION%/update.xml"
• Launch firefox with a brand new profile
• Check for updates [should get a small instant update]
• Restart Firefox
• Exit Firefox
• Disable enterprise policies/that policy: sudo defaults delete /Library/Preferences/org.mozilla.firefox AppUpdateURL and sudo defaults delete /Library/Preferences/org.mozilla.firefox EnterprisePoliciesEnabled
• Launch Firefox with same profile again
• Perform two consecutive check for updates with restarts, first should
  be normal and get you 78.0b5 second should show the distribution info in
  the about dialog and get you 78.0b8

Once we have a 78.0buildN off ESR we can test a real/formal mar. And once the above is validated I can create that formal mar and have it properly hosted. The next test will be restricted to the OSX Versions and not require any manual Enterprise Policy setting.

Comment 14

[Alexandru Trif, Desktop Test Engineering [:atrif]]

Hello! We conducted several update tests using the above steps on macOS 10.9.5/10.10.5 and 10.11.6 with .dmg and .pkg files. All updates work as expected and as described above. The distribution info is also shown in about dialog on 78.0b8 and 78.0b5 (confirmed as expected on slack).

Also, we ran a spot check on macOS 10.12.6/10.13.6/10.14.5 and macOS 10.15.6beta2 and there were no issues encountered and the updates are still respecting the above rules. One thing to mention here is that on macOS 10.12.6 and 10.13.6 we used policies.json to change the update URL channel, for some reason the terminal command didn’t activate policies. Thank you!

Comment 15

[Justin Wood (:Callek)]

Attachment: unsigned esr78-switch.mar (final [A])

I believe this will be the final mar switch we'll need for this.

sha256: fbb46db8c8d593d2c1efa1a9b21acdaecf1ddb1b31bf7674a5e190f27f84343c
size: 2155

Comment 16

[Justin Wood (:Callek)]

Attachment: signed esr78-switch.mar (final [A])

Sha256: 0a10e6e9ca1c7615b83cecf195e66bd0cba1e38dcfd9f8d0185da09f14deef16
sha512: 2ed56a6f0e63739dd96e3177215af4e30faa18c1795a05e1eaeb7e6f8df71ebe79a77f878393fa84e4b02db6161009ea2e2666a2a9b670b67da9834aa59e5f34

size: 2675

Comment 17

[[e-mail] (:bhearsum)]

For posterity, we didn't get the Balrog rules in place for this prior to 78 shipping. From roughly July 1 to July 6th, OS X 10.9-10.11 users on the release channel may have been offered updates to Firefox 78 instead of switched to esr. (It's a bit more subtle than that because of backgroundRate, but it was possible in various circumstances.)

Comment 18

[Mike Kaply [:mkaply]]

Were they offered 79 or 80 updates? Can we somehow move them frmo 78 to 78 esr with a full update?

Comment 19

[[e-mail] (:bhearsum)]

(In reply to Mike Kaply [:mkaply] from comment #18)

Were they offered 79 or 80 updates? Can we somehow move them frmo 78 to 78 esr with a full update?

As far as I can tell they were never offered updates past 78.0.1 or 78.0.2. Over in https://bugzilla.mozilla.org/show_bug.cgi?id=1661047, I'm trying to get them moved over to ESR (which I think we can do with the existing MAR). There's a test set-up on release-cdntest if you want to have a look.

Comment 20

[[e-mail] (:bhearsum)]

(In reply to [e-mail] (:bhearsum) from comment #17)

For posterity, we didn't get the Balrog rules in place for this prior to 78 shipping. From roughly July 1 to July 6th, OS X 10.9-10.11 users on the release channel may have been offered updates to Firefox 78 instead of switched to esr. (It's a bit more subtle than that because of backgroundRate, but it was possible in various circumstances.)

Again for posterity, I was told elsewhere that my read on this was wrong, and that we did have the proper rules in place.

Comment 21

[Julien Cristau [:jcristau]]

I guess we're not doing this after all.

Comment 22

[Julien Cristau [:jcristau]]

This was actually done with the 78 release, updating resolution.