1890764 - [macOS] Firefox 125 fails to update when running on non-admin accounts

Status: VERIFIED FIXED

(Toolkit :: Application Update, defect, P1)

Description

[Catalin Sasca, Desktop Test Engineering [:csasca]]

Found in

• Beta/RC 125

Affected versions

• macOS

Tested platforms

• Affected platforms: all macOS
• Unaffected platforms: Windows, Ubuntu

Steps to reproduce

1. Install a .pkg beta version from 125 of Firefox (e.g 125.0b1)
2. Start Firefox and visit About Firefox to trigger the update & restart the browser
3. Select Restart Firefox from macOS prompt and input the password
4. Firefox is restarted and prompts to manually download the newest build

Expected result

• Firefox should be updated to latest 125 RC version

Actual result

• Firefox is not updated, and will ask the user to manually download the newest build

Additional notes

• The issue can be seen in the following attachment
• Downloading the update again will result in a successful update

Comment 1

[[e-mail] (:bhearsum)]

I've been able to reproduce this, but for me, the problem in constant: I'm stuck on the same version forever, with Firefox continually offering to update me to latest.

Here's what I've looked at:

• I examined a set of 125.0b7 pkg and dmg packages. Both contain a Firefox.app directly that is identical aside from the CodeResources file. (I'm not sure if that's intentional or not, but I do not think it is relevant to this bug.)
• I've pulled some update logs, I'll add them as attachments
• I've verified that I can reproduce this on Nightly at least as far back as https://archive.mozilla.org/pub/firefox/nightly/2024/02/2024-02-01-09-53-46-mozilla-central/ - which suggests that this is unrelated to both the ChannelPrefs framework and the recent watershed we added for Beta

I think we'll need bytesized to investigate this further.

Comment 2

[Alexandru Trif, Desktop Test Engineering [:atrif]]

Hello! We made a manual regression range and got the following results:
last good build: 2024-02-21
first bad build: 2024-03-01
Pushlog: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=f8dd4015fa59fee3a1a74c00cd2393a9578e49f0&tochange=7a6867a3eabf3020e0dc11d629d16309150434ed
Unfortunately, we cannot verify Nightly builds between 21.02 and 01.03 because the updates are not working. We suspect bug 1882322 as a possible regressor. If more information is needed please let us know. I will let the regressionwindow wanted flag until the regressor is confirmed.

Comment 3

[Robin Steuber (she/her) [:bytesized]]

I am getting the elevation dialog, which it sounded like maybe others were not getting. I'm not totally sure what that is about. But once I accept the elevation dialog I can reproduce the rest of the STR.

After the update fails, however, if I try it again it succeeds. This is making me consider lowering the severity since it seems like users will still get updated, it will just take a little bit longer.

@csasca Does the same thing happen for you? Or does it consistently fail?

---

In the meantime, I'll keep investigating.

Comment 4

[Catalin Sasca, Desktop Test Engineering [:csasca]]

Yes, the same thing happens, after a failed update attempt (the message to manually download nightly is displayed), the second try will re-download the build, skip the user/password prompts and actually update to the latest version. It will consistently fail if reinstalling again an older build and re-doing the STR in Comment 0 (the same behavior as in the attachment).

Comment 5

[Alexandru Trif, Desktop Test Engineering [:atrif]]

We retested the good and bad builds today on other macOS stations using the steps from comment 0 and got the same results as in comment 2.
I will drop the regressionwindow-wanted flag and set flags based on the regression range. Unfortunately, we don't know the issue that caused this regression.

Comment 6

[Nick Alexander :nalexander [he/him]]

I've run this down a little further. First, I can reproduce what QA (Catalin?) reports: a 125.0b1 PKG fails to update the first time to 125.0. If I leave the /Applications/Firefox.app permissions at root:wheel (the default after installation), then I get an last-update-elevated.log like:

2024-04-11 11:30:16-0700: sUsingService=false
2024-04-11 11:30:16-0700: sUpdateSilently=false
2024-04-11 11:30:16-0700: gIsElevated=true
2024-04-11 11:30:16-0700: Writing status to file: applying
2024-04-11 11:30:16-0700: PATCH DIRECTORY /Users/nalexander/Library/Caches/Mozilla/updates/Applications/Firefox/updates/0
2024-04-11 11:30:16-0700: INSTALLATION DIRECTORY /Applications/Firefox.app
2024-04-11 11:30:16-0700: WORKING DIRECTORY /Applications/Firefox.app
2024-04-11 11:30:16-0700: failed: 38
2024-04-11 11:30:16-0700: Writing status to file: failed: 38

2024-04-11 11:30:16-0700: calling QuitProgressUI
2024-04-11 11:30:16-0700: Running LaunchCallbackAndPostProcessApps
2024-04-11 11:30:16-0700: Not elevated. Skipping LaunchMacPostProcess and LaunchCallbackApp

If instead I manually change the permissions after installing from the PKG, like sudo chown -R nalexander:admin /Applications/Firefox.app, then I can update as expected!

Comment 7

[Nick Alexander :nalexander [he/him]]

By serving updates myself and changing permissions manually, I can recreate this error 38 (which is UPDATE_SETTINGS_FILE_CHANNEL, see https://searchfox.org/mozilla-central/rev/98a54cbdc5964e778af0e6b325f9e7831f00c05b/toolkit/mozapps/update/common/updatererrors.h#69) continually.

I think we must be seeing some code-loading interaction between the helper application, the disk permissions, and the frameworks. The good news is that PKG installers who already have the helper application blessed should have the correct permissions and therefore be able to continue.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

Attachment: Bug 1890764 - Backout macOS framework changes from Firefox 125.

Backed out changeset 937a51dbe70c (bug 1883775)
Backed out changeset d88ec0d14b41 (bug 1883123)
Backed out changeset 49cc33ed1599 (bug 1883123)
Backed out changeset b6dff3a3d41b (bug 1882228)
Backed out changeset 62677f25e495 (bug 1881662)
Backed out changeset ac4d7c08648c (bug 1881635)
Backed out changeset 3157eece201c (bug 1881360)
Backed out changeset c676fc5d89bc (bug 1848414)
Backed out changeset 5c6748ef5865 (bug 1848414)
Backed out changeset 672c3f18cb8b (bug 1848414)
Backed out changeset 2f4044201b9a (bug 1848414)
Backed out changeset ff5f23856542 (bug 1848414)
Backed out changeset f81e80cf062f (bug 1848414)
Backed out changeset d711a830a93e (bug 1848414)
Backed out changeset 62ab286ed401 (bug 1848414)
Backed out changeset 24143ec374ff (bug 1854868)
Backed out changeset 9c11a87d38fc (bug 1854868)
Backed out changeset d7341669eb0b (bug 1854868)
Backed out changeset b9753f244f18 (bug 1854868)
Backed out changeset 1e094e3ca992 (bug 1799332)
Backed out changeset 62d28922bd23 (bug 1799332)
Backed out changeset c7aa9b74be01 (bug 1799332)
Backed out changeset d8dd28c68304 (bug 1799332)
Backed out changeset a50eb05b0b0b (bug 1799332)
Backed out changeset 21e9f01c0021 (bug 1882322)

Comment 9

[Ryan VanderMeulen [:RyanVM]]

Try push for the above: https://treeherder.mozilla.org/jobs?repo=try&revision=0022c7ac051a4946c4af7e2cba5564f9c63ef446 (just folded the commits together prior to submitting to Phab)

Comment 10

[Phabricator Automation]

release Uplift Approval Request

• User impact if declined: macOS users who are not Administrators will not be able to update Firefox
• Code covered by automated testing: yes
• Fix verified in Nightly: no
• Needs manual QE test: yes
• Steps to reproduce for manual QE testing: Before these patches: Install Firefox from a DMG or from a PKG as a macOS OS-level user that is not a member of the admin group. Serve updates using a local update server. Enter the username and password of an Administrator account when prompted. Witness failing updates. Check the last-update-elevated.log and witness error 38. After these patches, elevated updates should succeed.
• Risk associated with taking this patch: High
• Explanation of risk level: We propose to revert a lot of code with very little testing, automated or manual.
• String changes made/needed: No
• Is Android affected?: no

Comment 11

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9396810 [details]
Bug 1890764 - Backout macOS framework changes from Firefox 125.

Approved for 125.0.1

Comment 12

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-release/rev/624e9a169b06

Comment 13

[Brindusa Tot, Desktop QA]

Verified bug on RC 125.0.1 on macOS 12 x aarch64, macOS 13 x64, macOS 14 x aarch64 - on channels: release-localtest, release-cdntest and release.
Verification following scenarios with admin and standard system account:

MacOS Specific Update Testing (Case 1)

1. While logged into an Administrator account, clear cached update data by deleting ~/Library/Caches/Mozilla/updates
2. While logged in as that Administrator, install a previous Release version of Firefox via the .pkg installer.
3. Verify that /Applications/Firefox.app has owner root:wheel.
4. While logged into a non-Administrator account, attempt to update to 125.0.1. You will need to provide Administrator credentials.
5. After the update is complete, check the version.
6. Check that /Applications/Firefox.app has owner root:admin.

MacOS Specific Update Testing (Case 2)

1. While logged into a non-Administrator account, clear cached update data by deleting ~/Library/Caches/Mozilla/updates
2. As that same user, install a previous Release version of Firefox via the .pkg installer.
3. Verify that /Applications/Firefox.app has owner root:wheel.
4. While logged into an Administrator account, attempt to update to 125.0.1.
5. After the update is complete, check the version.
6. Check that /Applications/Firefox.app has owner root:admin.

Note: During bug verification, we observed that Known bug 1881536 is reproducible on release candidate.

Comment 14

[Stephen A Pohl [:spohl]]

Attachment: Bug 1890764: Ensure that elevated updates on macOS can read the MAR channel IDs. r=mstange,bytesized,nalexander

Comment 15

[Stephen A Pohl [:spohl]]

Comment on attachment 9397587 [details]
Bug 1890764: Ensure that elevated updates on macOS can read the MAR channel IDs. r=mstange,bytesized,nalexander

Beta/Release Uplift Approval Request

• User impact if declined: Users will not be able to update Firefox on macOS if they require elevation.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Please run elevated update tests, particularly the tests that exercise the install of Firefox by one admin user and an update by another admin user, AND the install by an admin user and an update by a standard user.
• List of other uplifts needed: None
• Risk to taking this patch: Medium
• Why is the change risky/not risky? (and alternatives if risky): This is a well-understood and narrow change to allow elevated updates to properly acquire the MAR channel IDs. The risk of taking this is similar to backing out all the macOS Framework changes, which had to occur for release 125.0.1.
• String changes made/needed: none
• Is Android affected?: No

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/70d3eb042ec3
Ensure that elevated updates on macOS can read the MAR channel IDs. r=bytesized,nalexander,mstange,application-update-reviewers

Comment 17

[Aron Cseh]

https://hg.mozilla.org/mozilla-central/rev/70d3eb042ec3

Comment 18

[Donal Meehan [:dmeehan]]

Comment on attachment 9397587 [details]
Bug 1890764: Ensure that elevated updates on macOS can read the MAR channel IDs. r=mstange,bytesized,nalexander

Approved for 126.0b6

Comment 19

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/793d76172075

Comment 20

[Donal Meehan [:dmeehan]]

Some additional info from :spohl:
Fyi, for elevated updates to start working again, we will need two builds before this will start working again. It will not be possible to run elevated updates from a "broken" beta build to a "fixed" beta build.

Comment 21

[Catalin Sasca, Desktop Test Engineering [:csasca]]

We verified all the scenarios provided in Comment 15 with both Nightly 125.0a1 (2024-02-20, unaffected build) and Nightly 127.0a1 (2024-04-25-09-41-16, fixed build) updating to Nightly 127.0a1 (2024-04-25-21-29-29 / 2024-04-26-09-19-52 fixed builds). Tests were performed on on macOS 14.4.1 Intel, macOS 13.6.3 ARM and macOS 12.6.3 ARM with .pkg and .dmg builds. We will verify on Beta as well when Beta 7 is available.

Comment 22

[Catalin Sasca, Desktop Test Engineering [:csasca]]

We verified the beta fix using the same scenarios as above with Firefox 126.0b6 updating to Firefox 126.0b7 and also some exploratory on different update scenarios - see more details here. Tests were performed on macOS 14.4.1 Intel, macOS 13.6.3 ARM and macOS 12.6.3 ARM with .pkg / dmg builds.
We couldn't verify the fix from an unaffected build to the latest (126.0b7), as there is the 125.0b9 Watershed in place and will get the users first to an affected build unfortunately.