[meta] Fix .moz-backup usage in non-staged updates
Categories
(Toolkit :: Application Update, defect)
Tracking
()
People
(Reporter: yannis, Assigned: yannis)
References
(Depends on 3 open bugs)
Details
(Keywords: meta)
This bug describes how, during a failed non-staged update to version N+1, the updater can end up restoring older versions for a subset of files. These files can end up in a version that is neither N nor N+1. This issue can lead to mixed-version install directories, which can have various consequences depending on which files are in which version at the end of the update:
(a) if only one of firefox.exe and xul.dll is restored to an old version, starting Firefox will result in the Error Platform Version message (see bug 1868951), the user will be unable to use Firefox unless they reinstall, and we will never know about it because there is still no telemetry for this error to this day (see bug 1917359, bug 1917317 comment 7); from 157, 153.4.0esr and 140.17.0esr on, restoring firefox.exe alone results in the startup crash described in comment 2 instead;
(b) if firefox.exe and xul.dll (and, from 157, 153.4.0esr and 140.17.0esr on, mozglue.dll) are all restored to the same old version from an older major version, starting Firefox will refuse to load the user's profile, because the profile was stamped at a newer major version, so the user will lose access to their profile data and we will receive a downgrade ping (if the old version has the same major version, Firefox silently runs the old version instead, because the profile downgrade check only compares major versions);
(c) in other cases, this can result in startup crashes like it did with mozilla::ErrorLoadingSheet for the 153 incident, and we will receive crashes.
I detail the scenario of the bug in comment 1. This is a long-standing bug, but specific circumstances make me more concerned about release 157, as I explain in comment 2; however the telemetry signals we can look at didn't show anything particularly worrying in nightly and beta as far as I can tell. Then I propose some solutions in comment 3.
| Assignee | ||
Comment 1•1 month ago
|
||
Context first. Non-staged updates rename the old version of a listed file by appending a .moz-backup suffix to their filename. These .moz-backup files are normally cleaned up during Finish(), but the updater can occasionally leave stale .moz-backup files in the installation directory, for example:
(a) in Firefox 155 and earlier, if the N+1 version of the file was currently open for execution during rollback (e.g. because Firefox was running), the rollback code would leave version N+1 in place (leading to mixed-version install directories) and in this case would also leave version N behind with the .moz-backup suffix;
(b) the updater may crash or get killed while running;
(c) deletion of a .moz-backup file may fail at the time it is attempted (including the fallback rename to tobedeleted/; e.g. if an antivirus is scanning the file).
Note that (a) is partially mitigated by bug 2058860 (156+) and bug 2058197 (partial MARs from 153.0.3 on).
Now that we know a user can get a .moz-backup file left behind, the future of this file will depend on what kind of updates happen in the future and whether they succeed:
- a staged update will leave such files in place, without cleaning nor modifying them;
- same for a non-staged update whose manifest does not list the corresponding base file;
- however, a successful non-staged update whose manifest lists the corresponding base file under patch or add will clean up the
.moz-backupfile.
The big problem comes when a non-staged update whose manifest lists the corresponding path fails. Namely, if we fail after the Prepare phase, but before creating the new .moz-backup for the corresponding path(s), then the rollback will overwrite the users' version N of the file with the old .moz-backup. This occurs because the updater does not check the provenance of the .moz-backup file and whether it was created by the current run of the updater.
So the full chain scenario is:
- some earlier non-staged update(s) failed and left some
.moz-backupfiles behind; - some staged updates may or may not have been applied since then and kept these files around;
- no non-staged update succeeded since then to clean them up;
- we do a non-staged update, and that update fails at a point before the creation of a subset of these
.moz-backupfiles.
In this scenario, this subset of .moz-backup files will be restored to the older versions of the files left by the previous failed updates.
| Assignee | ||
Comment 2•1 month ago
|
||
The following elements make this bug particularly worrying for release 157:
- the initial requirement (a) is what the reproduction of the 153 incident shows (see bug 2058197 comment 0): the new
firefox.exewas launched mid-update and was running during rollback, sofirefox.exeand/orxul.dlland/or other binaries such asmozglue.dllrollback failed, leaving them at version 153.0; given how the rollback code worked before bug 2058860, this also leftfirefox.exe.moz-backupand/ormozglue.dll.moz-backupand/orxul.dll.moz-backupbehind in version N (e.g. 152.0.6), so these people may have the leftovers still around; - bug 2060275 (156+) delays the creation of
.moz-backupfiles until after the DRAFT phase, so if using a 156+ updater any failure during the draft phase would roll back without any new.moz-backupcreated yet, therefore restoring all.moz-backuppaths listed by the manifest, even under a simple CRC failure (e.g. resulting from the previous failed update); for complete updates this is mostly a new failure mode: older updaters first back up all files listed inprecomplete, overwriting their stale backups, before writing anything, so they only restored stale backups if that backup step itself failed partway, which is a much less likely scenario; - bug 2057112 (157+, also uplifted to 153.4.0esr and 140.17.0esr) makes a version mismatch with a previous version between
mozglue.dllandfirefox.exefail very early during startup (DllBlocklist_Initialize) because the patch changes the layout and size ofModuleLoadInfo; this crashes so early that background tasks would also fail, so we would be unlikely to be able to download a future update that would restore the installation directory.
The following elements may save us:
- assuming that most users experience mostly experience the same kind of updates (staged vs non-staged), we can reasonably hope that the 153 incident population has applied a successful non-staged update since then, resulting in a cleanup of the most dangerous
.moz-backupfiles in their installation directories; xul.dll,omni.ja,browser/omni.jaandfirefox.exebeing last in partial update MARs in 153.0.3+ (bug 2058594) means non-staged partial updates are now less likely to leave these.moz-backupfiles behind upon failure compared to earlier versions (they are created last); unfortunatelymozglue.dllis not part of that set.
Signals to study for 157 release:
- crashes with proto signature contains DllBlocklist_Initialize (or mismatched version crashes in general);
- evolution of the downgrade ping.
| Assignee | ||
Comment 3•1 month ago
|
||
Proposed fixes (will file independent bugs, first one is the only urgent one):
- only restore a
.moz-backupfile if it was created during the current run (strongly recommended for a 157.0.1 uplift; since the installed updater applies the update, this only protects updates from 157.0.1 onwards); - stop preserving
.moz-backupfiles when applying a staged update; - clean up stale
.moz-backupfiles listed in the manifest during thePREPAREphase; - add telemetry counting
.moz-backupfiles in users' install directories, so we can confirm that they correctly disappear progressively; - apply the MAR reordering changes to complete updates too (to shorten the window during which the new
firefox.execan be launched mid-update there too, although the addition of the DRAFT phase already shortened it a lot); - add
mozglue.dllto the set of DLLs that get put to the end of MAR manifests since mismatched versions withfirefox.execan cause critical startup crashes.
Comment 4•1 month ago
|
||
| frontend-triage | ||
backup_restore gates the rollback rename only on X.moz-backup existing (line 1197), so a stale backup left by an earlier failed non-staged update is indistinguishable from one this run created, and no action records whether its Execute ever reached backup_create. Because ActionList::Finish calls Finish(status) on every action and DoUpdate falls straight through to it when DRAFT fails at #5722-5727, the rollback renames those leftovers over the user's version N.
Fix: Record backup ownership per action, with a flag set only on a successful backup_create in AddFile::Execute, PatchFile::Execute and RemoveFile::Execute, and take the restore branch of backup_finish only when it is set; in the same change give backups the stale-leftover sweep drafts already get, by discarding X.moz-backup from each action's Draft() the way draft_discard is called at AddFile::Draft and PatchFile::Draft, so a leftover cannot survive into a later run at all. There is no stale-backup counterpart to marSuccessPartialStaleDraft.js, and one seeding a .moz-backup before a DRAFT-phase failure would pin this directly on top of marFailurePartial.js.
Suggested severity: S2. An affected install ends up mixed-version and is unusable or loses profile access with no in-product recovery, since the DllBlocklist_Initialize crash path is early enough to stop a later update from healing it, though reaching it needs a specific chain of two failed non-staged updates.
If you'd like to provide feedback on this comment, please use the 👍 or 👎 reaction.
If you want to categorize your feedback you can add one of the following tags: ai-triage-wrong-file, ai-triage-wrong-cause, ai-triage-hallucination, ai-triage-out-of-scope, ai-triage-wrong-fix, ai-triage-shallow-fix.
Needinfo hackbot@mozilla.tld to have a patch generated for this bug, include any questions or directions as needed in your comment.
Updated•1 month ago
|
| Assignee | ||
Comment 6•1 month ago
•
|
||
(In reply to Ryan VanderMeulen [:RyanVM] from comment #5)
What's left to do here?
Adding details below.
(In reply to Yannis Juglaret [:yannis] from comment #3)
- only restore a
.moz-backupfile if it was created during the current run (strongly recommended for a 157.0.1 uplift; since the installed updater applies the update, this only protects updates from 157.0.1 onwards);
^ This is the critical issue. Patch is in bug 2076332, part of 158 at the moment. Will ask an ESR uplift (together with the patch from bug 2060275 that it fixes).
- apply the MAR reordering changes to complete updates too (to shorten the window during which the new
firefox.execan be launched mid-update there too, although the addition of the DRAFT phase already shortened it a lot);- add
mozglue.dllto the set of DLLs that get put to the end of MAR manifests since mismatched versions withfirefox.execan cause critical startup crashes.
^ These ones are nice to have and I'll work on them next.
- stop preserving
.moz-backupfiles when applying a staged update;- clean up stale
.moz-backupfiles listed in the manifest during thePREPAREphase;- add telemetry counting
.moz-backupfiles in users' install directories, so we can confirm that they correctly disappear progressively;
^ These ones are only for sanity, not critical.
Updated•1 month ago
|
Comment 7•1 month ago
|
||
This doesn't seem like it's worth carrying release-specific tracking flags on since it's basically a metabug for ongoing work.
Comment 8•20 days ago
|
||
Marking as meta bug and clearing severity rating.
Description
•