Closed Bug 1804303 Opened 1 year ago Closed 1 year ago

force an update to channel-prefs.js in 109 for release users on mac

Categories

(Release Engineering :: Release Requests, task)

Product:
Release Engineering
Component:
Release Requests
Type:
task

Tracking

(firefox-esr102 verified, firefox109+ verified, firefox110 verified, firefox111 verified)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox-esr102 --- verified
firefox109 + verified
firefox110 --- verified
firefox111 --- verified

People

(Reporter: bhearsum, Assigned: jcristau)

References

Details

(Keywords: leave-open)

Attachments

(3 files)

Bug 1804303 - force update to channel-prefs.js on mac.
48 bytes, text/x-phabricator-request
pascalc
: approval-mozilla-beta+
RyanVM
: approval-mozilla-release+
pascalc
: approval-mozilla-esr102+
Details | Review
Bug 1804303 - add 109.0.1 to the list of watershed releases
48 bytes, text/x-phabricator-request
Details | Review
Backed out changeset a4d8b4c3d321 (bug 1804303) as it is no longer necessary. r?bhearsum
48 bytes, text/x-phabricator-request
Details | Review

https://bugzilla.mozilla.org/show_bug.cgi?id=1799332 describes an issue affected macOS users on 13.0 (Ventura) that we believe affects anyone that has ever had their channel-prefs.js file modified (something that we did in Firefox 70 -- so any installs dating back that far are susceptible to this issue).

It's not entirely clear to me that everyone will be affected by it -- we seem to have fairly healthy uptake of 13.0 which I would not expect if this was 100% reproducible -- but nonetheless it's quite clear that it's possible to hit.

As a precautionary measure, I propose that we force an unconditional update to channel-prefs.js when we ship Firefox 108 to release users. This will ensure that they have the right version of the file, and that the code signature will be valid.

Notably, we cannot ship a MAR that contains this to the Beta channel, as it would cause those users to switch to Release -- so this probably requires some custom work for RelEng.

[Tracking Requested - why for this release]: potential late release-automation changes, related to newest macos version

tracking-firefox108: --- → ?

Julien, :bytesized, and I just talked about this and decided that due to both the very tight timeline here, and the fact that the data we have seems to show that not as many users on macOS 13.0 are affected by this, that we ought to do this for 109 instead.

Summary: force an update to channel-prefs.js in 108 for release users → force an update to channel-prefs.js in 109 for release users

A couple of other notes:

(In reply to bhearsum@mozilla.com (:bhearsum) from comment #3)

I don't think this will work quite as easily: we need something that results in a different update URL, so balrog knows whether a request is coming from a client before or after this change, and serve the right update. Changing the channel name achieves that, but just changing comments in channel-prefs.js doesn't, so I think it will likely need to be combined with an actual app update (so there's a new buildID).

Assignee: nobody → jcristau
Status: NEW → ASSIGNED

Ryan, chatting with Ben we came up with the following plan around this:

  1. land in nightly (some time next week after the bump to 111?) for at least one build, then back it out. Set up a mac watershed on that build
  2. monitor update telemetry to try and catch unexpected failures
  3. land on mozilla-release for 109.0.1
  4. set up the mac watershed on release once 109.0.1 is at 100%
  5. back out of m-r
    It'd probably be good to also do this on beta and aurora, perhaps with one of the late 110 builds?

How does this sound?

Flags: needinfo?(ryanvm)

I should note, I've verified in a staging release with this patch that both partial and complete mar files end up with add "Contents/Resources/defaults/pref/channel-prefs.js" in the update manifest instead of add-if-not (meaning we always replace that file with the one from the new version, instead of just creating it if it doesn't already exist).

(In reply to Julien Cristau [:jcristau] from comment #6)

Your proposal sounds good to me. Agreed that we should do it on Beta/Aurora as well. QA also needs a heads-up for update testing.

Flags: needinfo?(ryanvm)
Keywords: leave-open
Attachment #9311070 - Attachment description: Bug 1804303 - force update to channel-prefs.js → Bug 1804303 - force update to channel-prefs.js on mac.
Pushed by jcristau@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a4d8b4c3d321
force update to channel-prefs.js on mac. r=bhearsum DONTBUILD

Backed out changeset a4d8b4c3d321 (bug 1804303)

We needed one nightly build that updates channel-prefs.js. Now that
it's done, and a watershed is in place, we can restore mar file
generation to its normal logic.

Attachment #9312619 - Attachment description: Backed out 1 changesets (bug 1804303) as it is no longer necessary. r?bhearsum → Backed out changeset a4d8b4c3d321 (bug 1804303) as it is no longer necessary. r?bhearsum
Backout by jcristau@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/2de29d32476a
Backed out changeset a4d8b4c3d321 as it is no longer necessary. r=bhearsum DONTBUILD

Comment on attachment 9311070 [details]
Bug 1804303 - force update to channel-prefs.js on mac.

Beta/Release Uplift Approval Request

  • User impact if declined: signature verification failures on mac under unknown circumstances for installations pre-dating firefox 70
  • Is this code covered by automated tests?: No
  • Has the fix been verified in Nightly?: Yes
  • Needs manual test from QE?: Yes
  • If yes, steps to reproduce: covered in QA-1732
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): The update-packaging code is old and kind of quirky but the logic here is well understood and we've made and verified this change in nightly already.
  • String changes made/needed: n/a
  • Is Android affected?: No
Attachment #9311070 - Flags: approval-mozilla-release?

Comment on attachment 9311070 [details]
Bug 1804303 - force update to channel-prefs.js on mac.

Let's put this on beta also. We will need to revert it again before RC, though.

Attachment #9311070 - Flags: approval-mozilla-beta?

Comment on attachment 9311070 [details]
Bug 1804303 - force update to channel-prefs.js on mac.

Approved for 110 beta 5, thanks.

Attachment #9311070 - Flags: approval-mozilla-beta? → approval-mozilla-beta+

We just finished the updates spotcheck for Firefox 110.0b5. They passed as expected and the default channel.prefs.js is restored. We covered macOS 13.2 Intel and ARM and macOS 13.0.1 Intel. We updated from 105.0b5, 107.0b5 (incl. DevEd build), 109.0b5 and 110.0b4 and some DE/FR locales. We will test the watershed as well when 110.0b6 is available.

Comment on attachment 9311070 [details]
Bug 1804303 - force update to channel-prefs.js on mac.

Approved for 109.0.1

Attachment #9311070 - Flags: approval-mozilla-release? → approval-mozilla-release+

https://hg.mozilla.org/releases/mozilla-release/rev/fd28e59cfc4f

Are we going to need to do something for ESR also?

status-firefox109: --- → fixed
status-firefox111: --- → fixed
status-firefox-esr102: --- → ?
Flags: needinfo?(jcristau)

Tested the updates with watershed for Firefox 110.0b5. They passed as expected and the default channel-prefs.js is restored to the original state. We covered macOS 13.2 Intel and ARM and macOS 13.0.1 Intel. We updated from 105.0b5, 107.0b5 (incl. DevEd build), 109.0b5 and 110.0b4 and some DE/FR locales. We did some quick checks (updated from 109.0b9 en-US) on macOS 10.15, 11 and 12 and they had the same behavior with the watershed and restored channel prefs.

Comment on attachment 9311070 [details]
Bug 1804303 - force update to channel-prefs.js on mac.

We might as well, yes.

Flags: needinfo?(jcristau)
Attachment #9311070 - Flags: approval-mozilla-esr102?
Attachment #9311070 - Flags: approval-mozilla-esr102? → approval-mozilla-esr102+

We finished the initial update testing for Firefox 109.0.1 (release-cdntest channel). They passed as expected and the default channel.prefs.js is restored. We covered macOS 13.2 Intel and ARM and macOS 13.0.1 Intel (and quick spotcheck on older versions of macOS as well), more details for this testing can be found here.

We finished testing on the release channel as well. Tests passed as expected. More informations of the testing can be found in the link provided in Comment 26.

Pushed by jcristau@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/62d3f4facb14
add 109.0.1 to the list of watershed releases r=bhearsum DONTBUILD

I've verified with a staging esr release that the complete and partial mar files end up with add "Contents/Resources/defaults/pref/channel-prefs.js" in the manifest, but otherwise look the same.

We finished testing the updates to 110.0 (release-localtest channel) with 109.0.1 as the watershed (including resetting the channel-prefs.js) and we tested the updates for 102.8.0esr as well with no issues found. Test results for Firefox 110.0 can be seen here and for 102.8.0esr here.

We finished testing the updates to 110.0-build3 (release-cdntest channel) with 109.0.1 as the watershed (including resetting the channel-prefs.js) and the updates for 102.8.0esr-build2 (esr-cdntest) as well with no issues found. Test results for Firefox 110.0 can be seen here and for 102.8.0esr here.

I was pondering what to do for ESR.. One possibility is to back the patch out, and keep 102.8.0esr as a mac watershed, but we could also just leave the change in place and either keep the last 102.X release as a watershed for updates to 115.x, or (more likely) have bug 1799332 make this irrelevant. What do you think?

Flags: needinfo?(bhearsum)

We finished testing the updates to 110.0-build3 on release channel with 109.0.1 as the watershed (including resetting the channel-prefs.js) and the updates for 102.8.0esr-build2 on esr channel as well with no issues found. Test results for Firefox 110.0 can be seen here and for 102.8.0esr here.

status-firefox109: fixed → verified
status-firefox110: fixed → verified
status-firefox111: fixed → verified
status-firefox-esr102: fixed → verified

(In reply to Julien Cristau [:jcristau] from comment #34)

I was pondering what to do for ESR.. One possibility is to back the patch out, and keep 102.8.0esr as a mac watershed, but we could also just leave the change in place and either keep the last 102.X release as a watershed for updates to 115.x, or (more likely) have bug 1799332 make this irrelevant. What do you think?

Seems fine to me to keep the patch in place on esr102. We should make a note somewhere to make sure that we don't forget to revisit this when 115 spins up. (I agree that it's probably a non-issue at that point...but we should definitely make a concious decision here...)

Flags: needinfo?(bhearsum)

Thanks. I'll call this one fixed then. :)

Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Summary: force an update to channel-prefs.js in 109 for release users → force an update to channel-prefs.js in 109 for release users on mac
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: