Open Bug 1931088 Opened 21 days ago Updated 5 days ago

Update may fail if the installation directory permissions differ from those of the install files

Categories

(Toolkit :: Application Update, defect, P3)

Product:
Toolkit
Component:
Application Update
Version:
Firefox 132
Type:
defect

Tracking

()

Status:
UNCONFIRMED

People

(Reporter: ceplaw, Unassigned, NeedInfo)

References

Details

(Whiteboard: [fidedi-ope])

Steps to reproduce:

Attempted to update from Help | About Firefox, beginning with updating from 131.0.3 to 132.0

Actual results:

Update failed, entered an endless loop: In Help | About Firefox, the displayed version number remained the previous version that safely installed, with the "Restart to Update" button still present.

Problem persisted from 131.0.3 to 132.0.0, from 132.0.0 to 132.0.1, and again in 132.0.1 (can't see the new version number)! Each time, it required using the full Firefox Installer (setup_firefox.exe) to get the update... which risks damage to the profile.

Expected results:

Update should have applied, and Help | About Firefox should reflect the new version number and NOT display the "Restart to Update" button.

Clarification, missed fully describing a step: Each time attempting to update, I clicked the "Restart to Update" button; Firefox shut down and restarted without doing the update. Same result when manually shutting down Firefox, and when manually shutting down Firefox, rebooting the machine, and restarting Firefox.

The Bugbug bot thinks this bug should belong to the 'Toolkit::Application Update' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Component: Untriaged → Application Update
Product: Firefox → Toolkit

This seems potentially serious. needinfo-ing triage owner to try to get a quick response.

Flags: needinfo?(mpohle)

Hi CEP! We are taking this seriously and working to resolve it as soon as possible, but we require more details so we can fix the issue effectively.

To help analyze and resolve this issue, please provide the last-update.log file immediately after a failed update.

You can find that by

  • visiting about:support or Help-> More Troubleshooting Information (as explained here)
  • There a table, under General and next to "Updates-Folder" is a button to open the folder.
  • Clicking it opens your Explorer there and in that folder there is a subfolder called "updates".
  • Go there and that is where you can find the last-update.log file, which I hope contains more information for us to work with.

Also under about:support you can find the Update chronic, which you could check for error messages from past update attempts.

Thanks for filing this bug and your support!

Flags: needinfo?(mpohle) → needinfo?(ceplaw)

(In reply to Max from comment #4)

Hi CEP! We are taking this seriously and working to resolve it as soon as possible, but we require more details so we can fix the issue effectively.

To help analyze and resolve this issue, please provide the last-update.log file immediately after a failed update.

You can find that by

  • visiting about:support or Help-> More Troubleshooting Information (as explained here)
  • There a table, under General and next to "Updates-Folder" is a button to open the folder.
  • Clicking it opens your Explorer there and in that folder there is a subfolder called "updates".
  • Go there and that is where you can find the last-update.log file, which I hope contains more information for us to work with.

Also under about:support you can find the Update chronic, which you could check for error messages from past update attempts.

Thanks for filing this bug and your support!

Will do after the next update attempt -- I went ahead and manually updated (using the Firefox installer from Explorer) before I saw this or got a response notification, so it's now showing as 132.0.2 with no update available.

Flags: needinfo?(ceplaw)

One further bit of information, to rule out a potential issue that has arisen with other (non-Mozilla) software in the past: The machine is running Windows 11 Home (24H2 is not yet applied). About two months ago, I was forced to do a repair on it, but there was a successful Firefox update (from 131.0.2 to 131.0.3) after I did that repair on Windows. I think we can probably therefore rule out any "write permissions" problems, especially since the offline installer works and installs to the same folder — unless, that is, there's been a change in the way the internal-update system interacts with file-write permissions in version 132 that is not reflected in the offline installer.

Looking at the update.log file for an (apparently?) successful update, from the offline installer... it says it wasn't successful.

{begin quotation}
2024-11-13 15:55:26-0800: sUsingService=false
2024-11-13 15:55:26-0800: sUpdateSilently=false
2024-11-13 15:55:26-0800: gUseSecureOutputPath=false
2024-11-13 15:55:26-0800: useService=false
2024-11-13 15:55:26-0800: gIsElevated=false
2024-11-13 15:55:26-0800: Writing status to file: applying
2024-11-13 15:55:26-0800: PATCH DIRECTORY C:\ProgramData\Mozilla-1de4eec8-1241-4177-a864-e594e8d1fb38\updates\308046B0AF4A39CB\updates\0
2024-11-13 15:55:26-0800: INSTALLATION DIRECTORY C:\Program Files\Mozilla Firefox
2024-11-13 15:55:26-0800: WORKING DIRECTORY C:\Program Files\Mozilla Firefox
2024-11-13 15:55:26-0800: Checking whether elevation is needed
2024-11-13 15:55:26-0800: Successfully opened lock file
2024-11-13 15:55:26-0800: Going to update via this updater instance.
2024-11-13 15:55:26-0800: NS_main: callback app file open attempt 1 failed. File: C:\Program Files\Mozilla Firefox\firefox.exe. Last error: 5
2024-11-13 15:55:26-0800: NS_main: callback app file open attempt 2 failed. File: C:\Program Files\Mozilla Firefox\firefox.exe. Last error: 5
2024-11-13 15:55:26-0800: NS_main: callback app file open attempt 3 failed. File: C:\Program Files\Mozilla Firefox\firefox.exe. Last error: 5
2024-11-13 15:55:27-0800: NS_main: callback app file open attempt 4 failed. File: C:\Program Files\Mozilla Firefox\firefox.exe. Last error: 5
2024-11-13 15:55:27-0800: NS_main: callback app file open attempt 5 failed. File: C:\Program Files\Mozilla Firefox\firefox.exe. Last error: 5
2024-11-13 15:55:27-0800: NS_main: callback app file open attempt 6 failed. File: C:\Program Files\Mozilla Firefox\firefox.exe. Last error: 5
2024-11-13 15:55:27-0800: NS_main: callback app file open attempt 7 failed. File: C:\Program Files\Mozilla Firefox\firefox.exe. Last error: 5
2024-11-13 15:55:27-0800: NS_main: callback app file open attempt 8 failed. File: C:\Program Files\Mozilla Firefox\firefox.exe. Last error: 5
2024-11-13 15:55:27-0800: NS_main: callback app file open attempt 9 failed. File: C:\Program Files\Mozilla Firefox\firefox.exe. Last error: 5
2024-11-13 15:55:27-0800: NS_main: callback app file open attempt 10 failed. File: C:\Program Files\Mozilla Firefox\firefox.exe. Last error: 5
2024-11-13 15:55:27-0800: NS_main: callback app file in use, failed to exclusively open executable file: C:\Program Files\Mozilla Firefox\firefox.exe
2024-11-13 15:55:27-0800: Writing status to file: failed: 35
{end quotation}

However, all Mozilla programs had been manually closed down (using the top-right-corner close button), and the machine had been sitting for about two minutes while I answered a phone call. (I did not go into Process Explorer and manually kill processes, though). And even though "writing status to file failed," Help | About shows that the update was successful (that is, the version number increased).

So I'm even more confused.

And to reconfirm, everything is being done from an administrator Windows account (the same one that has done all Windows installs, and that has done all Firefox installs for that matter), so "ownership" shouldn't be an issue.

(In reply to CEP from comment #7)

Looking at the update.log file for an (apparently?) successful update, from the offline installer... it says it wasn't successful.

The installer doesn't write an update.log. That must be from the updater.

2024-11-13 15:55:27-0800: NS_main: callback app file open attempt 10 failed. File: C:\Program Files\Mozilla Firefox\firefox.exe. Last error: 5
2024-11-13 15:55:27-0800: NS_main: callback app file in use, failed to exclusively open executable file: C:\Program Files\Mozilla Firefox\firefox.exe
2024-11-13 15:55:27-0800: Writing status to file: failed: 35

Hmm. Error 5 = ERROR_ACCESS_DENIED. Error 35 = WRITE_ERROR_ACCESS_DENIED.

(In reply to CEP from comment #6)

I think we can probably therefore rule out any "write permissions" problems

I am not so sure. Can we see the permissions of your installation directory? Run this command, substituting your install directory: icacls "<installdir>". For example, I would run icacls "C:\Program Files\Firefox Nightly". Attach the output of this command to this bug, please.

Flags: needinfo?(ceplaw)

Besides checking the permissions on the directory from Bug 1931088 Comment 9, I would be interested to learn:

  • Do you or have you had other versions of Firefox installed (e.g. beta, nightly or installed through the MS Store)?

  • In your Task Scheduler under Task Scheduler Library -> Mozilla: How many entries do you see there named "Firefox Background Update 01234567890ABCDEF" and what do they say in the column "Last Run Result"?

  • Do you use any non-default security software like virus scanners, which may be reading and locking files when they get accessed? (if you have an outstanding windows update, it could be that this software does not display notifications, so that the updater times out, because the notification is not displayed and access to the files not granted).

(In reply to Max from comment #10)

Besides checking the permissions on the directory from Bug 1931088 Comment 9, I would be interested to learn:

  • Do you or have you had other versions of Firefox installed (e.g. beta, nightly or installed through the MS Store)?

  • In your Task Scheduler under Task Scheduler Library -> Mozilla: How many entries do you see there named "Firefox Background Update 01234567890ABCDEF" and what do they say in the column "Last Run Result"?

  • Do you use any non-default security software like virus scanners, which may be reading and locking files when they get accessed? (if you have an outstanding windows update, it could be that this software does not display notifications, so that the updater times out, because the notification is not displayed and access to the files not granted).

Answering this one first:

No other versions of Firefox on this machine (don't do beta, or nightly, or install from the MS Store on this machine).

There is exactly one task "Firefox Default Browser Agent 308046B0AF4A39CB" in Task Scheduler, 0x0 "(The operation completed successfully)". There is no "Firefox Background Update" task, because I have selected "notify only." (I never allow unattended-and-uncommanded updates. For anything.)

No non-default security software on this machine.

Flags: needinfo?(ceplaw)

(In reply to Robin Steuber (she/her) [:bytesized] from comment #9)

{begin quotation}
[SYSTEMID]:(OI)(CI)(RX)
[SYSTEMID]:(RX)
[SYSTEMID]:(OI)(CI)(IO)(RX,GR,GE)
BUILTIN\Administrators:(F)
NT SERVICE\TrustedInstaller:(I)(F)
NT SERVICE\TrustedInstaller:(I)(CI)(IO)(F)
NT AUTHORITY\SYSTEM:(I)(F)
NT AUTHORITY\SYSTEM:(I)(OI)(CI)(IO)(F)
BUILTIN\Administrators:(I)(F)
BUILTIN\Administrators:(I)(OI)(CI)(IO)(F)
BUILTIN\Users:(I)(RX)
BUILTIN\Users:(I)(OI)(CI)(IO)(GR,GE)
[USERNAME REDACTED]:(I)(F)
CREATOR OWNER:(I)(OI)(CI)(IO)(F)
APPLICATION PACKAGE AUTHORITY\ALL APPLICATION PACKAGES:(I)(RX)
APPLICATION PACKAGE AUTHORITY\ALL APPLICATION PACKAGES:(I)(OI)(CI)(IO)(GR,GE)
APPLICATION PACKAGE AUTHORITY\ALL RESTRICTED APPLICATION PACKAGES:(I)(RX)
APPLICATION PACKAGE AUTHORITY\ALL RESTRICTED APPLICATION PACKAGES:(I)(OI)(CI)(IO)(GR,GE)
{end quotation}

(redacting the system ID and username as noted) And as confirmation, this is the same user, same username, and same (full) administrator privileges. The command was run as an administrator.

I don't see any missing permissions here, unless (somehow?) the user identification associated with Firefox's update system has gone nuts... and it's not the OS repair, there was at least one successful update (131.0.2) after the OS repair.

(In reply to CEP from comment #7)

Looking at the update.log file for an (apparently?) successful update, from the offline installer... it says it wasn't successful.

The installer doesn't write an update.log. That must be from the updater.

2024-11-13 15:55:27-0800: NS_main: callback app file open attempt 10 failed. File: C:\Program Files\Mozilla Firefox\firefox.exe. Last error: 5
2024-11-13 15:55:27-0800: NS_main: callback app file in use, failed to exclusively open executable file: C:\Program Files\Mozilla Firefox\firefox.exe
2024-11-13 15:55:27-0800: Writing status to file: failed: 35

Hmm. Error 5 = ERROR_ACCESS_DENIED. Error 35 = WRITE_ERROR_ACCESS_DENIED.

(In reply to CEP from comment #6)

I think we can probably therefore rule out any "write permissions" problems

I am not so sure. Can we see the permissions of your installation directory? Run this command, substituting your install directory: icacls "<installdir>". For example, I would run icacls "C:\Program Files\Firefox Nightly". Attach the output of this command to this bug, please.

(In reply to CEP from comment #11)

(In reply to Max from comment #10)

Besides checking the permissions on the directory from Bug 1931088 Comment 9, I would be interested to learn:

  • Do you or have you had other versions of Firefox installed (e.g. beta, nightly or installed through the MS Store)?

  • In your Task Scheduler under Task Scheduler Library -> Mozilla: How many entries do you see there named "Firefox Background Update 01234567890ABCDEF" and what do they say in the column "Last Run Result"?

  • Do you use any non-default security software like virus scanners, which may be reading and locking files when they get accessed? (if you have an outstanding windows update, it could be that this software does not display notifications, so that the updater times out, because the notification is not displayed and access to the files not granted).

Answering this one first:

No other versions of Firefox on this machine (don't do beta, or nightly, or install from the MS Store on this machine).

There is exactly one task "Firefox Default Browser Agent 308046B0AF4A39CB" in Task Scheduler, 0x0 "(The operation completed successfully)". There is no "Firefox Background Update" task, because I have selected "notify only." (I never allow unattended-and-uncommanded updates. For anything.)

No non-default security software on this machine.

Further clarification: This problem arose well before there were any non-installed Windows Updates, with attempted first-day-of-release update from 131.0.3 to 132.0.0. I thus discount as the main cause anything related to "pending but uninstalled updates" (indeed, the update hasn't even been downloaded).

Hi CEP! Thanks for providing all the information. While I am trying to make sense of it, I hope that @bytesized can take another look on it.

Flags: needinfo?(bytesized)

(In reply to CEP from comment #12)

[USERNAME REDACTED]:(I)(F)

Fairly certain that's your problem right there. Something gave your user control of the Firefox installation directory, but not in such a way that any of the files within that directory inherit those permissions. I believe that means we can successfully create our lock file in the directory, making it look like we have the necessary permissions without needing to elevate them. But then we try to lock firefox.exe and fail because its permissions are more restrictive.

It seems like maybe we should fall back to elevating our permissions when we fail this way. But I'm going to set the severity to S3 because it's fairly questionable whether or not this is a configuration we really support.

Severity: -- → S3
Flags: needinfo?(bytesized)
Summary: Windows 132.0 and Later Updates Fail → Update may fail if the installation directory permissions differ from those of the install files
Whiteboard: [fidedi-ope]
Priority: -- → P3

I have assigned P3, because manual intervention is required for this issue to become one.

With that said: We have discussed possible solutions. One idea was to improve the logging to make the underlaying issue easier to identify. But we came to conclude that we could also attempt to automatically fix it by catching this error case (e.g. "updateFailedDueToWriteError") and elevate the process in that case (and if it is not already). That would simplify a patch, because we would not have to care about the permissions of individual files in order to fix it.

(sorry, been away ill for a couple days)

I have no idea what might have led to the "non-inheritance" problem -- it certainly wasn't intentional! I didn't command any software installations taking place (even if later uninstalled) between the last successful update (to 131.0.3) and the first unsuccessful one (to 132.0.0), and my log file doesn't show one occurring in the background, either. That leaves only a program "deciding" that it needed to do this on its own, and doing it in a way that disrupts the internal-to-Firefox updater but not the offline updater.

What further puzzles me a bit is that there is only one user account on this machine (and it has only ever had one user), with full administrative privileges, and that user account did all installations — specifically including all Mozilla software. The guest account is both protected by a nondefault login and fully disabled, as it has been since the machine was purchased a couple years ago; I checked, and the Windows repair appears to have left that completely alone. So under this setup, there shouldn't be any ownership issue at all. I can't imagine that any of my extensions (all from the Mozilla "store") would muck with permissions for the main program, either.

Further diagnostic attempt pointing at I don't know what (and it's bizarre/complicated enough you should read through instead of doing an Ah Ha! in the middle; mine led me astray):

This machine starts (and always has started) Firefox at startup. Prior to that Windows repair, it was always done with a copy of a shortcut link pasted into the "Program Data" startup hierarchy, along with two other programs including Tbird. That should start the program for "all users," not just the current user. And it did until the first monthly update after the Windows repair (October 2024), when Firefox no longer started at startup (but Tbird continued to). This was also the same rough time period that the internal updater problems began

I then used the Startup Programs manager in Task Manager to add Firefox again, and Firefox started at startup, but the internal updater did not work (this thread/bug/whatever it really is, starting to look like "badly documented interface weirdism"). Neither did the internal updater work after manually restarting Firefox using the same desktop link as had been copied into the Program Data hierarchy. The offline installer, however, did (although after running the offline installer, the internal installer still failed).

So, while in administrator mode, I deleted the desktop icon; made a new desktop icon; then manually copied that new icon into the Program Data hierarchy, while disabling via Task Manager. Firefox started correctly on startup this time (in "all users" rather than "current user (who happens to be the only user and an administrator"); but because there isn't a pending update to Firefox (yet), I can't test if this also reenabled the internal installer.

According to the API documentation I've been able to find, the place of an automatic startup should not matter because it's silent as to permissions and inherited permissions from a shortcut link. That it seems to nonetheless is concerning. That it doesn't seem to matter to Tbird is equally concerning. I'm very old school, in that "the program isn't done and ready for release until the documentation is done and ready for release," but it appears that somewhere that's not entirely accurate here... and I'm not pointing any fingers, as it could very well be one of those unimagined side effects.

So I can't test this until 132.0.4, or later, is out (I don't do nightly builds, and that wouldn't be a fair test anyway because I have no track record for nightly build updating). It seems dubiously promising — since the internal installer failed after manual restarts — but we'll have to see.

(I should also note that all of the shortcuts were in "Run as Administrator" mode)

(and that's a typo for next version number, I'm at 132.0.2 waiting for 132.0.3)

...and no luck. Got the "download new version" update while running "as administrator", tried to use the Help | About button to update, and it failed (still shows "ready to update" and 132.0.2 after clicking the button and waiting for the restart).

Hi Nicholas. This bug is currently not displayed on our triage center, even though it should. Could you have a look on it?

Flags: needinfo?(nrishel)
You need to log in before you can comment on or make changes to this bug.