Open Bug 2093841 Opened 2 days ago Updated 8 hours ago

Firefox on macOS keeps restarting and downloading an update again when it is left with no windows open and the update can't be installed

Categories

(Toolkit :: Application Update, defect)

Unspecified
macOS
defect

Tracking

()

ASSIGNED
Tracking Status
firefox-esr153 --- affected
firefox160 --- wontfix
firefox161 --- affected
firefox162 --- affected

People

(Reporter: spohl, Assigned: spohl)

References

(Regression)

Details

(Keywords: regression, Whiteboard: ai-triage-shallow-fix)

Attachments

(1 external file)

When all windows are closed on macOS and an update is pending, Firefox restarts silently after 5 minutes to install it (bug 1720742). If the install then fails because the updater can't write to the app bundle, Firefox appears to keep repeating this. It retries the same update, falls back to downloading the complete update, and downloads it again after the next update check (every 6 hours). It restarts each time an update is ready and no windows are open. Release telemetry between Oct 1 and Nov 15 shows roughly 10K Mac clients reporting several of these restarts in a day without the version changing, some more than 20. Nearly all of them report a failed update at the next startup, mostly WRITE_ERROR_EXTRACT (70) and WRITE_ERROR_PATCH_FILE (64). getElevationRequired() seems to pass for them, since only a few are in the pending-elevation state.

We should bound this the way we bound other update attempts. Once an update fails to install with a write error, Firefox should stop restarting silently for it and shouldn't download the same update again, but retry at most about every 12 hours. That interval should be measured in wall-clock time rather than with a timer, since timers on macOS don't count time asleep (see bug 2089250 and bug 1204823). The new limit should be controllable by a pref. It would also help to find out why the updater can't write to the bundle when getElevationRequired() thinks it can, which likely needs an update.log from an affected machine.

Set release status flags based on info from the regressing bug 1720742

RestartOnLastWindowClosed restarts whenever AUS.currentState is pending (UpdateService.sys.mjs), and its timer is armed at startup because a silent restart opens no windows, so every write error that handleUpdateFailure resets to pending (#1603-1624) and the complete patch that handleFallbackToCompleteUpdate downloads (#1860) each cause another silent restart. Once the complete patch also fails, #selectUpdate (#3793) keeps no record of the failed version, so the next check downloads it again and the cycle repeats.

Fix: When #asyncInit processes a write error, store Date.now() and the update's appVersion in prefs. While that same version failed less than a new interval pref ago (default 12 hours, capped with Math.min as onStateAccessFailure does), skip the silent restart in #maybeRestartBrowser, the re-download in #selectUpdate following the macOS elevation-version gate at #3873-3950, and the direct fallback download, which bypasses #selectUpdate. Cover the version gate in updateFailureFallBack.js and the restart gate in test_no_window_update_restart.py, the only test that exercises this class.

Suggested severity: S2. About 10K release Mac clients restart silently many times a day and repeatedly re-download a full update that never installs, and the only way to stop it from inside the product is an about:config pref.


If you'd like to provide feedback on this comment, please use the 👍 or 👎 reaction.
If you want to categorize your feedback you can add one of the following tags: ai-triage-{wrong-file, wrong-cause, hallucination, out-of-scope, not-a-bug, wrong-fix, shallow-fix, wrong-severity}.

Needinfo hackbot@mozilla.tld to have a patch generated for this bug, include any questions or directions as needed in your comment.

(In reply to Hackbot [:hackbot] from comment #2)

RestartOnLastWindowClosed restarts whenever AUS.currentState is pending (UpdateService.sys.mjs), and its timer is armed at startup because a silent restart opens no windows, so every write error that handleUpdateFailure resets to pending (#1603-1624) and the complete patch that handleFallbackToCompleteUpdate downloads (#1860) each cause another silent restart. Once the complete patch also fails, #selectUpdate (#3793) keeps no record of the failed version, so the next check downloads it again and the cycle repeats.

Fix: When #asyncInit processes a write error, store Date.now() and the update's appVersion in prefs. While that same version failed less than a new interval pref ago (default 12 hours, capped with Math.min as onStateAccessFailure does), skip the silent restart in #maybeRestartBrowser, the re-download in #selectUpdate following the macOS elevation-version gate at #3873-3950, and the direct fallback download, which bypasses #selectUpdate. Cover the version gate in updateFailureFallBack.js and the restart gate in test_no_window_update_restart.py, the only test that exercises this class.

Suggested severity: S2. About 10K release Mac clients restart silently many times a day and repeatedly re-download a full update that never installs, and the only way to stop it from inside the product is an about:config pref.


If you'd like to provide feedback on this comment, please use the 👍 or 👎 reaction.
If you want to categorize your feedback you can add one of the following tags: ai-triage-{wrong-file, wrong-cause, hallucination, out-of-scope, not-a-bug, wrong-fix, shallow-fix, wrong-severity}.

Needinfo hackbot@mozilla.tld to have a patch generated for this bug, include any questions or directions as needed in your comment.

I'm not sure if I'm supposed to provide Hackbot-feedback here or elsewhere, but I thought I'd record this just in case:

The analysis of the restart loop matches what I found, and skipping the direct fallback download in handleFallbackToCompleteUpdate is a good catch. A few problems with the proposed fix, though:

  1. Putting the check in #selectUpdate would also affect manual update checks. AppUpdater.sys.mjs calls selectUpdate() for the About dialog, so a user who explicitly clicks "Check for updates" would be blocked too, and it could look like no update is available. The check belongs on the background path only (_selectAndInstallUpdate).
  2. Keying on appVersion would also hold back a respin of the same version, which is a different build that might install fine. The build ID seems like the better key, and it still covers the partial and complete patches of the same update.
  3. It doesn't get at why this happens in the first place. getElevationRequired() says the app bundle is writable, but the updater then fails with WRITE_ERROR_EXTRACT and WRITE_ERROR_PATCH_FILE. Limiting the retries should stop the loop, but these clients still never update, so the mismatch between that check and the updater needs its own investigation.
Whiteboard: ai-triage-shallow-fix

When the updater can't write to the app bundle, Firefox on macOS with no windows open restarts silently every few minutes to retry, and downloads the update again after each update check. After a write error, wait app.update.writeError.retryInterval seconds (12 hours) before restarting silently for that update or downloading it again in the background. The time is compared against the wall clock, since timers on macOS don't advance while the Mac is asleep.

(Trying this via n-i...)

Hackbot, see comment 3.

Flags: needinfo?(hackbot)

Thanks, the feedback in comment 3 is correct, and the patch in comment 4 already handles points 1 and 2. I checked both against the source:

  1. AppUpdater.sys.mjs:337 calls aus.selectUpdate() for the About dialog, and that goes through #selectUpdate. A gate there would hide updates from manual checks too, so it belongs in _selectAndInstallUpdate (UpdateService.sys.mjs:3971).
  2. Keying on build ID is better than appVersion. It still covers both the partial and the complete patch of the same update, and it doesn't block a respin.

On point 3, this is an untested guess: getElevationRequired() relies only on nsIFile.isWritable(), which is access(W_OK), run on the bundle and its entries (areDirectoryEntriesWriteable). That checks POSIX permissions and ACLs. It doesn't catch failures that only show up at open()/rename() time, such as macOS App Management/TCC denials or a bundle on a read-only or translocated volume. Those would pass the check and then fail with WRITE_ERROR_*. An update.log (and last-update.log) from an affected machine would confirm or rule this out. I'd suggest a separate follow-up bug for it, so it doesn't hold up the retry limit here.


If you'd like to provide feedback on this comment, please use the 👍 or 👎 reaction.

Flags: needinfo?(hackbot)

In comment 6, Hackbot guessed that the write errors come from macOS App Management or TCC denials, or from a read-only or translocated volume, which getElevationRequired() wouldn't catch. I looked into this a bit more, and I don't think permissions are the problem here. It looks like these clients are running out of disk space while the update installs.

Both error codes are raised after the file in question was opened successfully:

  1. WRITE_ERROR_EXTRACT (70) comes from a failed fwrite in ArchiveReader::ExtractItemToStream, after creat() succeeded. A failed creat() returns WRITE_ERROR (7) instead.
  2. WRITE_ERROR_PATCH_FILE (64) comes from a failed fwrite in bspatch.cpp (or zucchini's patch write error), again on a file that's already open. A failed open would be WRITE_ERROR_OPEN_PATCH_FILE (63), which only 4 of the 491 sampled clients reported.

A write failing after a successful open points to a full disk, a quota or an I/O error rather than permissions or access checks, which would also explain why getElevationRequired() passes. As far as I can tell, a read-only or translocated volume wouldn't pass that check either: access(W_OK) fails with EROFS there, nsLocalFile::IsWritable then returns an error, and getElevationRequired() treats that as "elevation required".

Telemetry seems to back this up. browser.backup.prof_d_disk_space reports the available space on the profile's volume, which is usually but not always the same disk as /Applications. Mac release clients, 5% sample, Oct 1 to Nov 15:

Group Clients Under 1 GB free Under 2 GB free Median free
No automatic restart 300,769 0.2% 0.5% 143 GB
Normal automatic restart 288,484 0.2% 0.4% 128 GB
Restart loop 491 37.1% 58.4% 1.5 GB

That would also explain why the download succeeds while the install fails: the downloaded MAR itself takes up space, and installing needs room for the extracted files plus backups. Firefox appears to delete the previous download before it fetches the update again, so the downloads shouldn't pile up. Each attempt needs that space again, though, so it's likely to keep failing the same way. Limiting retries should stop the restarts and the repeated downloads, but it won't free up any space.

This is a strong correlation rather than proof, and about a quarter of the loop clients have 5 GB or more free, so there may be a second cause. An update.log showing ENOSPC from an affected machine would confirm it. I think checking free space before downloading and installing, and telling the user when low disk space is what's blocking the update, would be best handled in a follow-up bug.

Attachment #9684657 - Attachment description: Bug 2093841: Limit how often macOS retries an update that failed to install with a write error. r?#application-update-reviewers!,yjuglaret! → Bug 2093841: Limit how often macOS retries an update that failed to install. r?#application-update-reviewers!,yjuglaret!
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: