Firefox on macOS keeps restarting and downloading an update again when it is left with no windows open and the update can't be installed
Categories
(Toolkit :: Application Update, defect)
Tracking
()
People
(Reporter: spohl, Assigned: spohl)
References
(Regression)
Details
(Keywords: regression, Whiteboard: ai-triage-shallow-fix)
Attachments
(1 external file)
When all windows are closed on macOS and an update is pending, Firefox restarts silently after 5 minutes to install it (bug 1720742). If the install then fails because the updater can't write to the app bundle, Firefox appears to keep repeating this. It retries the same update, falls back to downloading the complete update, and downloads it again after the next update check (every 6 hours). It restarts each time an update is ready and no windows are open. Release telemetry between Oct 1 and Nov 15 shows roughly 10K Mac clients reporting several of these restarts in a day without the version changing, some more than 20. Nearly all of them report a failed update at the next startup, mostly WRITE_ERROR_EXTRACT (70) and WRITE_ERROR_PATCH_FILE (64). getElevationRequired() seems to pass for them, since only a few are in the pending-elevation state.
We should bound this the way we bound other update attempts. Once an update fails to install with a write error, Firefox should stop restarting silently for it and shouldn't download the same update again, but retry at most about every 12 hours. That interval should be measured in wall-clock time rather than with a timer, since timers on macOS don't count time asleep (see bug 2089250 and bug 1204823). The new limit should be controllable by a pref. It would also help to find out why the updater can't write to the bundle when getElevationRequired() thinks it can, which likely needs an update.log from an affected machine.
Comment 1•2 days ago
|
||
Set release status flags based on info from the regressing bug 1720742
Comment 2•2 days ago
|
||
| frontend-triage | ||
RestartOnLastWindowClosed restarts whenever AUS.currentState is pending (UpdateService.sys.mjs), and its timer is armed at startup because a silent restart opens no windows, so every write error that handleUpdateFailure resets to pending (#1603-1624) and the complete patch that handleFallbackToCompleteUpdate downloads (#1860) each cause another silent restart. Once the complete patch also fails, #selectUpdate (#3793) keeps no record of the failed version, so the next check downloads it again and the cycle repeats.
Fix: When #asyncInit processes a write error, store Date.now() and the update's appVersion in prefs. While that same version failed less than a new interval pref ago (default 12 hours, capped with Math.min as onStateAccessFailure does), skip the silent restart in #maybeRestartBrowser, the re-download in #selectUpdate following the macOS elevation-version gate at #3873-3950, and the direct fallback download, which bypasses #selectUpdate. Cover the version gate in updateFailureFallBack.js and the restart gate in test_no_window_update_restart.py, the only test that exercises this class.
Suggested severity: S2. About 10K release Mac clients restart silently many times a day and repeatedly re-download a full update that never installs, and the only way to stop it from inside the product is an about:config pref.
If you'd like to provide feedback on this comment, please use the 👍 or 👎 reaction.
If you want to categorize your feedback you can add one of the following tags: ai-triage-{wrong-file, wrong-cause, hallucination, out-of-scope, not-a-bug, wrong-fix, shallow-fix, wrong-severity}.
Needinfo hackbot@mozilla.tld to have a patch generated for this bug, include any questions or directions as needed in your comment.
Updated•2 days ago
|
| Assignee | ||
Comment 3•2 days ago
|
||
(In reply to Hackbot [:hackbot] from comment #2)
RestartOnLastWindowClosedrestarts wheneverAUS.currentStateis pending (UpdateService.sys.mjs), and its timer is armed at startup because a silent restart opens no windows, so every write error thathandleUpdateFailureresets to pending (#1603-1624) and the complete patch thathandleFallbackToCompleteUpdatedownloads (#1860) each cause another silent restart. Once the complete patch also fails,#selectUpdate(#3793) keeps no record of the failed version, so the next check downloads it again and the cycle repeats.Fix: When
#asyncInitprocesses a write error, storeDate.now()and the update'sappVersionin prefs. While that same version failed less than a new interval pref ago (default 12 hours, capped withMath.minasonStateAccessFailuredoes), skip the silent restart in#maybeRestartBrowser, the re-download in#selectUpdatefollowing the macOS elevation-version gate at #3873-3950, and the direct fallback download, which bypasses#selectUpdate. Cover the version gate in updateFailureFallBack.js and the restart gate in test_no_window_update_restart.py, the only test that exercises this class.Suggested severity: S2. About 10K release Mac clients restart silently many times a day and repeatedly re-download a full update that never installs, and the only way to stop it from inside the product is an about:config pref.
If you'd like to provide feedback on this comment, please use the 👍 or 👎 reaction.
If you want to categorize your feedback you can add one of the following tags: ai-triage-{wrong-file, wrong-cause, hallucination, out-of-scope, not-a-bug, wrong-fix, shallow-fix, wrong-severity}.Needinfo hackbot@mozilla.tld to have a patch generated for this bug, include any questions or directions as needed in your comment.
I'm not sure if I'm supposed to provide Hackbot-feedback here or elsewhere, but I thought I'd record this just in case:
The analysis of the restart loop matches what I found, and skipping the direct fallback download in handleFallbackToCompleteUpdate is a good catch. A few problems with the proposed fix, though:
- Putting the check in
#selectUpdatewould also affect manual update checks.AppUpdater.sys.mjscallsselectUpdate()for the About dialog, so a user who explicitly clicks "Check for updates" would be blocked too, and it could look like no update is available. The check belongs on the background path only (_selectAndInstallUpdate). - Keying on
appVersionwould also hold back a respin of the same version, which is a different build that might install fine. The build ID seems like the better key, and it still covers the partial and complete patches of the same update. - It doesn't get at why this happens in the first place.
getElevationRequired()says the app bundle is writable, but the updater then fails withWRITE_ERROR_EXTRACTandWRITE_ERROR_PATCH_FILE. Limiting the retries should stop the loop, but these clients still never update, so the mismatch between that check and the updater needs its own investigation.
| Assignee | ||
Comment 4•2 days ago
|
||
When the updater can't write to the app bundle, Firefox on macOS with no windows open restarts silently every few minutes to retry, and downloads the update again after each update check. After a write error, wait app.update.writeError.retryInterval seconds (12 hours) before restarting silently for that update or downloading it again in the background. The time is compared against the wall clock, since timers on macOS don't advance while the Mac is asleep.
| Assignee | ||
Comment 5•1 day ago
|
||
(Trying this via n-i...)
Hackbot, see comment 3.
Comment 6•1 day ago
|
||
| bug-fix | ||
Thanks, the feedback in comment 3 is correct, and the patch in comment 4 already handles points 1 and 2. I checked both against the source:
AppUpdater.sys.mjs:337callsaus.selectUpdate()for the About dialog, and that goes through#selectUpdate. A gate there would hide updates from manual checks too, so it belongs in_selectAndInstallUpdate(UpdateService.sys.mjs:3971).- Keying on build ID is better than
appVersion. It still covers both the partial and the complete patch of the same update, and it doesn't block a respin.
On point 3, this is an untested guess: getElevationRequired() relies only on nsIFile.isWritable(), which is access(W_OK), run on the bundle and its entries (areDirectoryEntriesWriteable). That checks POSIX permissions and ACLs. It doesn't catch failures that only show up at open()/rename() time, such as macOS App Management/TCC denials or a bundle on a read-only or translocated volume. Those would pass the check and then fail with WRITE_ERROR_*. An update.log (and last-update.log) from an affected machine would confirm or rule this out. I'd suggest a separate follow-up bug for it, so it doesn't hold up the retry limit here.
If you'd like to provide feedback on this comment, please use the 👍 or 👎 reaction.
| Assignee | ||
Comment 7•1 day ago
•
|
||
In comment 6, Hackbot guessed that the write errors come from macOS App Management or TCC denials, or from a read-only or translocated volume, which getElevationRequired() wouldn't catch. I looked into this a bit more, and I don't think permissions are the problem here. It looks like these clients are running out of disk space while the update installs.
Both error codes are raised after the file in question was opened successfully:
WRITE_ERROR_EXTRACT(70) comes from a failedfwriteinArchiveReader::ExtractItemToStream, aftercreat()succeeded. A failedcreat()returnsWRITE_ERROR(7) instead.WRITE_ERROR_PATCH_FILE(64) comes from a failedfwriteinbspatch.cpp(or zucchini's patch write error), again on a file that's already open. A failed open would beWRITE_ERROR_OPEN_PATCH_FILE(63), which only 4 of the 491 sampled clients reported.
A write failing after a successful open points to a full disk, a quota or an I/O error rather than permissions or access checks, which would also explain why getElevationRequired() passes. As far as I can tell, a read-only or translocated volume wouldn't pass that check either: access(W_OK) fails with EROFS there, nsLocalFile::IsWritable then returns an error, and getElevationRequired() treats that as "elevation required".
Telemetry seems to back this up. browser.backup.prof_d_disk_space reports the available space on the profile's volume, which is usually but not always the same disk as /Applications. Mac release clients, 5% sample, Oct 1 to Nov 15:
| Group | Clients | Under 1 GB free | Under 2 GB free | Median free |
|---|---|---|---|---|
| No automatic restart | 300,769 | 0.2% | 0.5% | 143 GB |
| Normal automatic restart | 288,484 | 0.2% | 0.4% | 128 GB |
| Restart loop | 491 | 37.1% | 58.4% | 1.5 GB |
That would also explain why the download succeeds while the install fails: the downloaded MAR itself takes up space, and installing needs room for the extracted files plus backups. Firefox appears to delete the previous download before it fetches the update again, so the downloads shouldn't pile up. Each attempt needs that space again, though, so it's likely to keep failing the same way. Limiting retries should stop the restarts and the repeated downloads, but it won't free up any space.
This is a strong correlation rather than proof, and about a quarter of the loop clients have 5 GB or more free, so there may be a second cause. An update.log showing ENOSPC from an affected machine would confirm it. I think checking free space before downloading and installing, and telling the user when low disk space is what's blocking the update, would be best handled in a follow-up bug.
Updated•8 hours ago
|
Description
•