1882322 - macOS: Cannot update nightly anymore: "Update failed"

Status: RESOLVED FIXED

(Toolkit :: Application Update, defect, P1)

Description

[Johan Lorenzo [:jlorenzo]]

Attachment: AUS logs on macOS Nightly 20240226165659

Could be reproduced on macOS and Linux[1]. My macOS buildID is 20240226165659. Here are the logs attached. It seems it can't apply neither the partial nor the complete file. Nothing else stands out to me in the logs. I'll try to see if I can get debug ones.

[1] edit 15 minutes after filing this bug: Linux was not impacted I misread a Slack conversation.

Comment 1

[Johan Lorenzo [:jlorenzo]]

Error code 38 keeps showed up multiple times in the logs:

AUS:SVC readStatusFile - status: failed: 38, path: /Users/johanlorenzo/Library/Caches/Mozilla/updates/Applications/Firefox Nightly/updates/0/update.status
[...]
AUS:SVC UpdateManager:refreshUpdateStatus - status = failed: 38

It points to this error variable[1]

#define UPDATE_SETTINGS_FILE_CHANNEL 38

:bytesized, would you have some insights on this?

[1] https://searchfox.org/mozilla-central/rev/098f910d0593b12a42089dd8f40dcd19d1121430/toolkit/mozapps/update/common/updatererrors.h#69

Comment 2

[Johan Lorenzo [:jlorenzo]]

Could be a regression from bug 1848414.

Comment 3

[Stephen A Pohl [:spohl]]

What version of macOS is this on? Do you see any errors in Console.app? Possibly by filtering for org.mozilla.updater and/or firefox?

Comment 4

[Robin Steuber (she/her) [:bytesized]]

Looking into it.

Comment 5

[Robin Steuber (she/her) [:bytesized]]

@jlorenzo Could you also provide the update URL that Firefox is using? The log that you provided starts after the update check. Thanks!

Edit: No longer needed. See Comment 7.

Comment 6

[Robin Steuber (she/her) [:bytesized]]

I also need updates.xml, which should be located at ~/Library/Caches/Mozilla/updates/Applications/<app name>/updates.xml.

Edit: No longer needed. See Comment 7.

Comment 7

[Robin Steuber (she/her) [:bytesized]]

Ah, I've figured out how to reliably reproduce the problem. I no longer need the things I requested in Comments 5 and 6.

---

Steps to Reproduce:

1. Download and install build 20240220212334, here.
2. Add a policies file so that updates can be served locally (more information on serving updates locally here).
3. Inspection of the installation directory shows neither of the frameworks are present and both of the configuration files are, as expected for this build.
4. Download the complete MAR for build 20240222103314, here, serve it, have Firefox download and stage it, and restart Firefox to install it.
5. Inspection of the installation directory shows both of the frameworks are present, channel-prefs.js is still present (unexpectedly), and update-settings.ini has been removed (expectedly).
6. Download the complete MAR for build 20240223210323, here, serve it, have Firefox download and stage it, and restart Firefox to install it.

Expected Results:
7. Inspection of the installation directory shows both of the frameworks are present and neither of the configuration files are.

Actual Results:
7. Inspection of the installation directory shows that channel-prefs.js is present and both frameworks technically present but are missing their binaries and Info.plist files.

---

Some preliminary investigation:

After step (3), the update log indicates that it attempted to remove the directories that channel-prefs.js is in, but not channel-prefs.js itself, for some reason. Since the directories were not empty, they were not removed successfully. This isn't contributing towards the problem at hand, but will presumably break the signature if we don't address it.

After step (5), a filtered update log looks like this (I included the updater binary mostly for reference, since that was updated properly)

> grep 'ChannelPrefs|UpdateSettings|updater.app/Contents/MacOS' ~/Library/Caches/Mozilla/updates/Applications/Firefox\ Nightly/updates/backup-update.log 
2024-02-27 15:25:49-0800: PREPARE REMOVEFILE Contents/MacOS/updater.app/Contents/MacOS/org.mozilla.updater
2024-02-27 15:25:49-0800: PREPARE REMOVEFILE Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/UpdateSettings
2024-02-27 15:25:49-0800: PREPARE REMOVEFILE Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/Resources/Info.plist
2024-02-27 15:25:49-0800: PREPARE REMOVEFILE Contents/Frameworks/ChannelPrefs.framework/Resources/Info.plist
2024-02-27 15:25:49-0800: PREPARE REMOVEFILE Contents/Frameworks/ChannelPrefs.framework/ChannelPrefs
2024-02-27 15:25:49-0800: PREPARE REMOVEDIR Contents/MacOS/updater.app/Contents/MacOS/
2024-02-27 15:25:49-0800: PREPARE REMOVEDIR Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/Resources/
2024-02-27 15:25:49-0800: PREPARE REMOVEDIR Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/
2024-02-27 15:25:49-0800: PREPARE REMOVEDIR Contents/Frameworks/ChannelPrefs.framework/Resources/
2024-02-27 15:25:49-0800: PREPARE REMOVEDIR Contents/Frameworks/ChannelPrefs.framework/
2024-02-27 15:25:49-0800: PREPARE ADD Contents/MacOS/updater.app/Contents/MacOS/org.mozilla.updater
2024-02-27 15:25:49-0800: EXECUTE REMOVEFILE Contents/MacOS/updater.app/Contents/MacOS/org.mozilla.updater
2024-02-27 15:25:49-0800: EXECUTE REMOVEFILE Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/UpdateSettings
2024-02-27 15:25:49-0800: EXECUTE REMOVEFILE Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/Resources/Info.plist
2024-02-27 15:25:49-0800: EXECUTE REMOVEFILE Contents/Frameworks/ChannelPrefs.framework/Resources/Info.plist
2024-02-27 15:25:49-0800: EXECUTE REMOVEFILE Contents/Frameworks/ChannelPrefs.framework/ChannelPrefs
2024-02-27 15:25:49-0800: EXECUTE REMOVEDIR Contents/MacOS/updater.app/Contents/MacOS/
2024-02-27 15:25:49-0800: EXECUTE REMOVEDIR Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/Resources/
2024-02-27 15:25:49-0800: EXECUTE REMOVEDIR Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/
2024-02-27 15:25:49-0800: EXECUTE REMOVEDIR Contents/Frameworks/ChannelPrefs.framework/Resources/
2024-02-27 15:25:49-0800: EXECUTE REMOVEDIR Contents/Frameworks/ChannelPrefs.framework/
2024-02-27 15:25:51-0800: EXECUTE ADD Contents/MacOS/updater.app/Contents/MacOS/org.mozilla.updater
2024-02-27 15:25:58-0800: FINISH REMOVEFILE Contents/MacOS/updater.app/Contents/MacOS/org.mozilla.updater
2024-02-27 15:25:58-0800: FINISH REMOVEFILE Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/UpdateSettings
2024-02-27 15:25:58-0800: FINISH REMOVEFILE Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/Resources/Info.plist
2024-02-27 15:25:58-0800: FINISH REMOVEFILE Contents/Frameworks/ChannelPrefs.framework/Resources/Info.plist
2024-02-27 15:25:58-0800: FINISH REMOVEFILE Contents/Frameworks/ChannelPrefs.framework/ChannelPrefs
2024-02-27 15:25:58-0800: FINISH REMOVEDIR Contents/MacOS/updater.app/Contents/MacOS/
2024-02-27 15:25:58-0800: non-fatal error removing directory: Contents/MacOS/updater.app/Contents/MacOS/, rv: 0, err: 66
2024-02-27 15:25:58-0800: FINISH REMOVEDIR Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/Resources/
2024-02-27 15:25:58-0800: FINISH REMOVEDIR Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/
2024-02-27 15:25:58-0800: non-fatal error removing directory: Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/, rv: 0, err: 66
2024-02-27 15:25:58-0800: FINISH REMOVEDIR Contents/Frameworks/ChannelPrefs.framework/Resources/
2024-02-27 15:25:58-0800: FINISH REMOVEDIR Contents/Frameworks/ChannelPrefs.framework/
2024-02-27 15:25:58-0800: non-fatal error removing directory: Contents/Frameworks/ChannelPrefs.framework/, rv: 0, err: 66
2024-02-27 15:25:58-0800: FINISH ADD Contents/MacOS/updater.app/Contents/MacOS/org.mozilla.updater

Clearly something is wrong here. For some reason, we never get PREPARE ADD or EXECUTE ADD lines for the missing framework files. Interestingly, there are no lines for the CodeResources files that are the one remaining bit of the frameworks after encountering the problem. I'm not yet sure why.

If I extract the 20240220212334 (pre-framework) complete MAR, these lines (among others) are present:

add-if-not "Contents/Resources/update-settings.ini" "Contents/Resources/update-settings.ini"
add-if-not "Contents/Resources/defaults/pref/channel-prefs.js" "Contents/Resources/defaults/pref/channel-prefs.js"

If I extract the 20240223210323 complete MAR (the one that seems to have cause the problem, in my case), these lines are present:

add-if-not "Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/_CodeSignature/CodeResources" "Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/_CodeSignature/CodeResources"
add-if-not "Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/UpdateSettings" "Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/UpdateSettings"
add-if-not "Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/Resources/Info.plist" "Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/Resources/Info.plist"
add-if-not "Contents/Frameworks/ChannelPrefs.framework/_CodeSignature/CodeResources" "Contents/Frameworks/ChannelPrefs.framework/_CodeSignature/CodeResources"
add-if-not "Contents/Frameworks/ChannelPrefs.framework/Resources/Info.plist" "Contents/Frameworks/ChannelPrefs.framework/Resources/Info.plist"
add-if-not "Contents/Frameworks/ChannelPrefs.framework/ChannelPrefs" "Contents/Frameworks/ChannelPrefs.framework/ChannelPrefs"

I don't see anything that would suggest that there is a problem with the manifest.

I'll continue investigating.

Comment 8

[Robin Steuber (she/her) [:bytesized]]

Well, I may be closing in on the issue. Maybe.

There is a file called precomplete that, as I understand it, acts as a sort of supplementary update manifest file. The copy in this installation appears to contain entries calling for the removal of the mysteriously missing files:

remove "Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/UpdateSettings"
remove "Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/Resources/Info.plist"
...
remove "Contents/Frameworks/ChannelPrefs.framework/Resources/Info.plist"
remove "Contents/Frameworks/ChannelPrefs.framework/ChannelPrefs"

And it is missing the CodeResources files in those same directories.

Unfortunately, there is a problem. I know almost nothing about the precomplete file. I don't even know who to ask, other than people that no longer work here. So I think I need to dig into what the heck this thing is, how it works, and where it comes from. Hopefully that will allow me to figure out how to fix this issue.

Comment 9

[Julien Cristau [:jcristau]]

AIUI precomplete is generated by https://searchfox.org/mozilla-central/source/config/createprecomplete.py as part of mach package (https://searchfox.org/mozilla-central/source/toolkit/mozapps/installer/packager.py#291), adding remove instructions for files that are part of the package, so that when applying a complete mar the updater starts by removing those files before adding the new ones from the mar manifest. precomplete is generated pre-signing (since it's itself part of the signed bundle), so it doesn't know about CodeResources. Hope this helps.

Comment 10

[BugBot [:suhaib / :marco/ :calixte]]

The bug is marked as blocking firefox125 (nightly). However, the bug still isn't assigned.

:Amir, could you please find an assignee for this tracked bug? If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit BugBot documentation.

Comment 11

[Robin Steuber (she/her) [:bytesized]]

Attachment: Bug 1882322 - Prevent macOS channel frameworks from being in precomplete file r=nalexander!

Comment 12

[Robin Steuber (she/her) [:bytesized]]

Alright, I believe I have figured out the next steps that ought to be taken here:

The unexpectedly-present channel-prefs.js issue seems to have been identified and spun off into Bug 1882619. But the worst damage that problem should be able to do is to break the app bundle's signature, which doesn't typically cause problems and can be fixed via update. So I'm not worrying too much about that. It'll merge whenever it merges. Once it does, I'll make sure that things look as they should and the signatures validate.

I believe that the patch I posted to this bug should address the missing framework issue. I plan to land the patch as soon as possible.

Once it has landed, there are a couple things to do before we turn macOS Nightly updates back on. One of them is that I would like to try to run a bit more manual testing by downloading the generated MARs and making sure that applying them works as expected. This should be quick. The remaining things have to do with trying to break as few Nightly installations as possible.

Unfortunately, anyone who is already missing their UpdateSettings framework binary is simply not going to be able to update automatically. The relevant code just will not install an update if it can't verify that it is for the correct channel which it cannot do without that framework. Those users will see "Update Failed" notifications that will prompt them to update manually by reinstalling. However, there are other users that are in a bad state but can be "rescued".

The mechanism of this bug is that the frameworks are removed due to erroneous removal instructions in the precomplete file. When applying an update from a "complete" MAR, the precomplete file is used to find and remove almost all existing installation files. This should not include files that define the current channel, but currently it does. Meaning that the inability to update isn't encountered until after one update writes those files into precomplete and then another update, specifically from a complete MAR, reads that precomplete file that tells it to delete the frameworks. Thus, we want to be very careful about how we update users that may have a bad precomplete file. They should be able to update successfully as long as we are careful to serve them a partial MAR to update them to a good version.

I've talked to :jcristau and :bhearsum about this, and it should be possible to do this via some carefully crafted Balrog rules and likely some manually generated partial MARs. This is the other thing that needs to be done between this patch landing and macOS Nightly updates being re-enabled.

---

To summarize, this is my plan:

1. Merge the attached patch.
2. Run some more quick manual testing on the resulting MARs.
3. Manually generate the necessary partial MARs.
4. Add necessary Balrog rules.
5. Turn macOS Nightly updates back on.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8ba058675fbc
Prevent macOS channel frameworks from being in precomplete file r=jcristau

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/21e9f01c0021
Prevent macOS channel frameworks from being in precomplete file r=jcristau

Comment 15

[Cristina Horotan [:chorotan]]

https://hg.mozilla.org/mozilla-central/rev/21e9f01c0021

Comment 16

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/8ba058675fbc