1890528 - On macOS, update channel is changed to release for updates to RC 125 (on beta and release-localtest channels)

Status: RESOLVED FIXED

(Toolkit :: Application Update, defect, P1)

Description

[Brindusa Tot, Desktop QA]

Found in
RC 125.0

Affected versions
macOS

Tested platforms

• Affected platforms: all macOS
• Unaffected platforms: Windows, Ubuntu

Steps to reproduce

1. Install older version than 125 of firefox (e.g RC 123.0)
2. In channel-pref.js, change the update channel from 'release' to 'release-localtest'.
3. Start firefox.
4. Visit About Firefox to trigger the update & restart the browser;

Expected result

• Firefox should be updated to latest 125 RC version, and update channel should remain 'release-localtest'

Actual result

• Firefox is updated to latest 125 RC version, but update channel is changed back to 'release'.

Regression range

• Starting with RC 125.

Additional notes

• This is also reproducible also on the update is done on the beta channel.
• This might be related to the changes landed in bug 1799332

Comment 1

[Ryan VanderMeulen [:RyanVM]]

I have all updates on the Beta channel halted across the board for now until we can get a mac-only rule in place pinning their updates to 125.0b9.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Updates have been re-enabled for the Beta channel with a rule in place pinning all Mac updates to 125.0b9.

Comment 3

[[e-mail] (:bhearsum)]

MARs being served to Beta and Release populations both contain the release ChannelPrefs framework with an add-if-not instruction. This means that if the framework is not present on disk, it will be created. This instruction is correct, but the Beta MARs should contain the beta ChannelPrefs framework.

124.0 and earlier installs pre-date this framework, so they end up installing it, which causes the install to become a release channel install regardless of what channel-prefs.js wanted.

125.0 already has a ChannelPrefs framework, so its channel remains unchanged.

This appears to be an oversight - I don't see any chatter or plans to ensure that non-release channel MARs contain different ChannelPrefs frameworks in https://bugzilla.mozilla.org/show_bug.cgi?id=1799332 or related bugs.

Comment 4

[[e-mail] (:bhearsum)]

Our immediate remediation for this was to remove the rule blocking all updates on Beta to the RC, and replace it with a watershed rule that forced all Beta users <124.0 through 125.0b9 before getting the RC. This allows us to get the RC to Beta users immediately, and will function as a long term fix if we don't decide to do anything else.

There is another potential way to fix this that doesn't require us to keep the watershed. If we updated our processes to produce separate MARs for Beta users that contained the Beta ChannelPrefs.framework and UpdateSettings.framework, we could remove the above watershed. However, creating these has some drawbacks:

• We would either need separate builds, or to insert these frameworks into the real RC before creating the MARs. This would make the MARs we serve to Beta users a little different than the ones that Release users get. Most notably, the complete MAR would no longer be the exact same one that is shipped to Release users. The differences shouldn't be functional ones, but would be a notable change.
• We would end up paying for storage cost on many, many more MARs going forward.
• The engineering work would not be trivial (but it is tractable).

Another way we could catch such issues in the future is to enhance our update verify tests to be able to accurately diff ChannelPrefs and UpdateSettings frameworks. Over in https://bugzilla.mozilla.org/show_bug.cgi?id=1883123 I had these tests ignore those files because I couldn't see a feasible way to compare these binary files accurately. This incident suggests we need to go the extra mile here though.

Comment 5

[Stephen A Pohl [:spohl]]

My vote would be to keep this watershed in place for as long as it makes sense (i.e. while there are a reasonable number of users on Beta <124.0).

Comment 6

[[e-mail] (:bhearsum)]

Attachment: update_messages_old.log

This log was dumped when reproducing the issue with a recent Firefox Nightly.

Comment 7

[[e-mail] (:bhearsum)]

Here's the state of the update cache directory; I will attach relevant files from within it as well:

bhearsum-tdjgh6:Firefox Nightly vpn$ pwd
/Users/vpn/Library/Caches/Mozilla/updates/Applications/Firefox Nightly
bhearsum-tdjgh6:Firefox Nightly vpn$ ls -l
total 40
-rw-r--r--@ 1 vpn  staff   1227 Apr 10 09:17 active-update.xml
-rw-r--r--@ 1 vpn  staff      0 Jan 10 09:50 macAttributionDataCache-nightly
drwxr-xr-x@ 6 vpn  staff    192 Mar  1 14:22 updates
-rw-r--r--@ 1 vpn  staff  13075 Mar  1 14:22 updates.xml
bhearsum-tdjgh6:Firefox Nightly vpn$ ls -l updates
total 32
drwxr-xr-x@ 5 vpn  staff    160 Apr 10 09:17 0
-rw-rw-rw-@ 1 vpn  staff  10046 Mar  1 14:21 backup-update.log
drwxr-xr-x@ 2 vpn  staff     64 Apr 10 09:17 downloading
-rw-rw-rw-@ 1 vpn  staff   1424 Mar  1 14:22 last-update.log
bhearsum-tdjgh6:Firefox Nightly vpn$ ls -l updates/0
total 263088
-rw-------@ 1 vpn  staff  121463070 Apr 10 09:17 update.mar
-rw-r--r--@ 1 vpn  staff         16 Apr 10 09:17 update.status
-rw-r--r--@ 1 vpn  staff          8 Apr 10 09:17 update.version
bhearsum-tdjgh6:Firefox Nightly vpn$ ls -l updates/downloading
total 0

Comment 8

[[e-mail] (:bhearsum)]

Attachment: active-update.xml

Comment 9

[[e-mail] (:bhearsum)]

Attachment: update.status

Comment 10

[[e-mail] (:bhearsum)]

Attachment: update.version

Comment 11

[[e-mail] (:bhearsum)]

(I didn't attach files like last-update.log because they are quite old, and presumably irrelevant - let me know if there's any other information wanted.)

Comment 12

[Ryan VanderMeulen [:RyanVM]]

I think we can call this specific issue fixed by way of the watershed rule we added forcing users to update through 125.0b9. We can spin off any follow-up work to a new bug.

Comment 13

[Brindusa Tot, Desktop QA]

Verified the fix on macOS for beta channel updates with the watershed rule and can confirm the followings:

• for .dmg build updates work as expected:
  • for beta build < 125 - the build is updated to 125.0b9 and then another update to RC 125.0
  • for 125 beta builds, updates goes directly to RC 125
• for .pkg build updates work as expected:
  • for beta build < 125 - the build is updated to 125.0b9 and then another updated to RC 125.0
  • for 125 beta builds: Tries to update to 125.0 directly but it fails, asking to download the latest build manually. Second try will download 125.0 again, and will update directly, without going through .pkg prompts for user/password. For this behavior another bug was logged: bug 1890764 and does not seem to be related to watershed rule.

In all cases, channel remains beta.
Verification was performed on macOS 12, 13 and 14.

Marking this as verified for Beta 125.