Closed Bug 1596778 Opened 10 months ago Closed 10 months ago

[Windows] Firefox fails to update to the latest available beta version when the installation directory is opened in Windows explorer.exe

Categories

(Toolkit :: Application Update, defect)

71 Branch
Desktop
Windows
defect
Not set
major

Tracking

()

VERIFIED FIXED
mozilla72
Tracking Status
firefox-esr68 72+ fixed
firefox70 --- wontfix
firefox71 + disabled
firefox72 + verified

People

(Reporter: atrif, Assigned: robert.strong.bugs)

References

(Blocks 1 open bug, Regression)

Details

(Keywords: regression)

Attachments

(4 files)

Affected versions

  • This issue is reproducible with builds >= 71.0b3

Affected platforms

  • Windows 10 64bit
  • Windows 7 x64

Preconditions

  • Install or upgrade Firefox 71.0b5 to default location.

Steps to reproduce

  1. Launch Firefox.
  2. Try to update Firefox (hamburger menu Help -> about Firefox -> Restart to update).

Expected result

  • Firefox is updated to the latest version.

Actual result

  • Firefox fails to update to the latest version. If not reinstall Firefox 71.0b5 over the updated firefox in default location.

Regression Range

  • Tested 71.0b3, 71.0b4, 71.0b5, 71.0b6 .exe -> all fails

Notes

  • This seems to be reproducible with builds starting from Firefox 71.0b3.
  • Also, it seems that Firefox needs to be installed in the default location. If the updates succeed exit Firefox and reinstall Firefox 71.0b5 over the installed Firefox and try to update again.
  • Attached a screen recording with the issue.
  • Attached a logfile from browser console after the failed update.
  • I can't reproduce with 72.0a1.
  • Also please note that the issue may be intermittent.

Robert, that seems like a serious issue, could you investigate please? Thanks

Flags: needinfo?(robert.strong.bugs)
Severity: normal → major

[rstrong is out today, so I'll take this for now]

Typically when something like this happens, the first thing I look at is whether something is preventing Firefox from shutting down properly, and the updater is failing to overwrite it with the new version for that reason (on Windows you can't overwrite executable files that have running processes).

The console log doesn't show anything wrong, so I'm going to ask you for another log file, which is the one created by the updater binary when it tries to overwrite your installation with the patched copy. To collect that, please do the following:

  1. Make sure that "Enable browser chrome and add-on debugging toolboxes" is checked in the dev tools settings (or that the pref, devtools.chrome.enabled, is true).
  2. After a restart where an update has failed, open the browser console and run var fileLocator = Cc["@mozilla.org/file/directory_service;1"].getService(Ci.nsIProperties); var dir = fileLocator.get("UpdRootD", Ci.nsIFile); dir.reveal(); to get a File Explorer window at the correct install-specific directory.
  3. From there open the updates directory, and attach the file last-update.log. I think the relevant errors will have been logged there. You should also find backup-update.log there, and attaching it as well wouldn't hurt.

Thanks!

Flags: needinfo?(robert.strong.bugs) → needinfo?(alexandru.trif)
Attached file update_log.zip

Hello!
I think I managed to find what's triggering this issue and I don't know if this is the expected behavior.
I can reproduce this every time by following the next STR:

  1. Install Firefox 71.0b5 to default location (C:\Program Files\Mozilla Firefox).
  2. Open the folder where Firefox is installed (e.g C:\Program Files\Mozilla Firefox) and keep it open.
  3. Update Firefox (Hamburger menu -> Help -> About Firefox -> Restart to update).
    AR: Firefox fails to update.
    On an isolated network, it seems that at step 2 it needs another subfolder opened to fail (e.g: open C:\Program Files\Mozilla Firefox\defaults) and keep it open. I can't seem to repro this by using the .zip

Also attaching the required files. Thank you!

Flags: needinfo?(alexandru.trif)

It seems that keeping the Firefox installation folder and a subfolder opened while updating results in a failed update. Is this expected? Thank's!

Flags: needinfo?(mhowell)

Adding some new information and modifying the flags.
It seems that updating from 70.0b14 (20191010142853) to the latest beta 71.0b10 (20191114160003) while following the STR from comment 3 works as expected. Also worth noting that after the restart to update button is pushed there is a "Firefox Update" window that appears as seen in the screenshot that is not displayed after updating from 71.0b3 (20191021164841). Following the STR from comment 3 on 71.0b3 (20191021164841) for updating to the latest beta results in a failed update. Also adding Firefox 72.0a1 (20191117170005) as affected.
I can’t reproduce this while updating from Firefox 70.0 (20191016161957) to Firefox 70.0.1 (20191030021342).

Has Regression Range: --- → no
Has STR: --- → yes
Keywords: regression

Thanks! That's the log output that I expected to see if my initial theory about Firefox shutting down was right, but that wasn't correct; it seems you're right about the File Explorer being related.

What it looks like is happening is that if either an Explorer window is open on a subdirectory of the Firefox installation, or such a directory is listed in the Quick Access menu, Explorer opens a locking handle on that directory which prevents anything from moving or deleting its parent. This is what's preventing the update from running; we can't rename the directory with the old version so that we can then move the new version into place. Forcing Explorer's directory handles closed using Process Explorer causes the update to succeed.

I'm not sure there's anything we can do about this; we can't forcibly close Explorer's handles because that could destabilize Explorer in unpredictable ways. It's probably very uncommon for anyone to have open Explorer windows on subdirectories of a Firefox installation or to have any in Quick Access, so hopefully this problem isn't affecting too many people at least.

This doesn't answer the question of why you weren't able to reproduce with certain Firefox versions, particular current nightlies; I can reproduce the problem with those myself, and I can't think of anything that should be different about them.

I'll leave this bug alone for Robert to decide what to do with, or maybe they can think of a way to handle it that I haven't.

Flags: needinfo?(mhowell)

Given the specific conditions required to have this bug happen, I am ranking the severity for 71 from blocking to tracking+.

I think I see what is going on with this bug and will investigate further.

Assignee: nobody → robert.strong.bugs
Status: NEW → ASSIGNED
Summary: [Windows] Firefox fails to update to the latest available beta version → [Windows] Firefox fails to update to the latest available beta version when the installation directory is opened in Windows explorer.exe

Hello Robert!
Do you think that a regression range would be useful here? It would require manual bisection, which will take quite a bit of time considering the nature of the issue- I'd like to make sure that spending a lot of time on it would actually bring value to your investigation.

Flags: needinfo?(robert.strong.bugs)

Thanks for the offer Alexandru but it isn't necessary. This was regressed by bug 1510494.

Flags: needinfo?(robert.strong.bugs)
Regressed by: CVE-2019-17009

Filed bug 1598449 to add a test for this bug.

I manually verified that this patch fixes this bug using oak and that the typical update paths are working properly.

Comment on attachment 9110462 [details]
Bug 1596778 - fallback to the normal update path when a replace request fails. r=mhowell

Beta/Release Uplift Approval Request

  • User impact if declined: Updates will fail when the install directory is in use.

  • Is this code covered by automated tests?: No

  • Has the fix been verified in Nightly?: No

  • Needs manual test from QE?: Yes

  • If yes, steps to reproduce: From comment #3

    Install Firefox 71.0b5 to default location (C:\Program Files\Mozilla Firefox).
    Open the folder where Firefox is installed (e.g C:\Program Files\Mozilla Firefox) and keep it open.
    Update Firefox (Hamburger menu -> Help -> About Firefox -> Restart to update).
    AR: Firefox fails to update.
    On an isolated network, it seems that at step 2 it needs another subfolder opened to fail (e.g: open C:\Program Files\Mozilla Firefox\defaults) and keep it open. I can't seem to repro this by using the .zip

  • List of other uplifts needed: None

  • Risk to taking this patch: Low

  • Why is the change risky/not risky? (and alternatives if risky): The different update paths have been verified using the oak project branch and the patch isn't complicated.

  • String changes made/needed: None

Attachment #9110462 - Flags: approval-mozilla-beta?
Flags: qe-verify+
Pushed by rstrong@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9f52e0351809
fallback to the normal update path when a replace request fails. r=mhowell

We have shipped our last beta for 71, since there is a patch in the bug, I am setting the status as fix-optional for 71 in case there is a safe uplift possible in a potential dot release as a ride-along.

Comment on attachment 9110462 [details]
Bug 1596778 - fallback to the normal update path when a replace request fails. r=mhowell

The uplift in bug 1510494 will fix this issue.

Attachment #9110462 - Flags: approval-mozilla-beta? → approval-mozilla-beta-
Status: ASSIGNED → RESOLVED
Closed: 10 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla72
QA Whiteboard: [qa-triaged]

Tested while updating from Firefox 72.0a1 (20191123214900) containing this fix to the latest Nightly (72.0a1- 20191124230652) on Windows 10x64 and the update is successfully executed while having the installation directory/subfolders opened in explorer.exe.
Also, just want to make sure about one thing here: when trying to update from an affected version of Firefox (e.g Firefox 72.0a1 (20191121214422)) with the installation directory opened, Firefox fails to update. Is this expected? Thanks!

Flags: needinfo?(robert.strong.bugs)

(In reply to Alexandru Trif, QA [:atrif] from comment #19)

Tested while updating from Firefox 72.0a1 (20191123214900) containing this fix to the latest Nightly (72.0a1- 20191124230652) on Windows 10x64 and the update is successfully executed while having the installation directory/subfolders opened in explorer.exe.
Also, just want to make sure about one thing here: when trying to update from an affected version of Firefox (e.g Firefox 72.0a1 (20191121214422)) with the installation directory opened, Firefox fails to update. Is this expected? Thanks!

That is correct... versions with this bug will still fail to update when the installation directory is opened in explorer.

Flags: needinfo?(robert.strong.bugs)

This will affect ESR once we uplift other work, but right now, it doesn't.

The regressing case was disabled in 71 (in an extra check-in for bug 1510494). We're not leaving users affected by this bug as implied by "wontfix". The bug is latent, but disabled by a pref (until the real fix landed in Fx72)

Removing the qe+ flag.

Flags: qe-verify+

Rechecked when updating from Firefox 72.0a1 (20191125095200) to Firefox 72.0a1 (20191125214644) when the installation folder is opened in explore.exe. Sorry for the spam.

Status: RESOLVED → VERIFIED
Attached patch ESR PatchSplinter Review

This patch also re-enables update staging

Attachment #9111672 - Flags: review+

Comment on attachment 9111672 [details] [diff] [review]
ESR Patch

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration: Fixes a regression with update staging introduced by bug 1510494
  • User impact if declined: Update staging will remain disabled.
  • Fix Landed on Version: 71
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): This patch has baked for quite some time without any bug reports.
  • String or UUID changes made by this patch: None
Attachment #9111672 - Flags: approval-mozilla-esr68?

Changing flag from disabled to affected so staging can be re-enabled on ESR

Duplicate of this bug: 1591141
Comment on attachment 9111672 [details] [diff] [review]
ESR Patch

Fixes a regression from bug 1510494 and re-enables update staging (previously disabled for 68.3esr as a temporary workaround). Approved for 68.4esr.
Attachment #9111672 - Flags: approval-mozilla-esr68? → approval-mozilla-esr68+
You need to log in before you can comment on or make changes to this bug.