Closed Bug 1682810 Opened 3 years ago Closed 3 years ago

always getting UAC prompt to elevate permissions if user account if maintenanceservice_installer.exe gone missing

Categories

(Toolkit :: Application Update, defect)

Product:
Toolkit
Component:
Application Update
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
87 Branch
Tracking Flags:
Tracking Status
firefox87 --- fixed

People

(Reporter: aryx, Assigned: bytesized)

Details

Attachments

(2 files)

Bug 1682810 - When staging fails, leave the update in the same state regardless of how it failed r=nalexander!
48 bytes, text/x-phabricator-request
Details | Review
Bug 1682810 - Fix tests to expect the correct pending status r=#application-update-reviewers!
48 bytes, text/x-phabricator-request
Details | Review

Two days ago maintenanceservice_installer.exe got quarantine by Antivirus during the Nightly update (bug 1682286). Restoring the file didn't put it into a folder on the desktop.

Since this event, Nightly updates always prompts for the elevated account permissions to update Nightly.

Should the state of the Maintenance Service get detected and a full Nightly update be scheduled for the next update to get the maintenance service reinstalled.

Edge case, I know.

Hmm. I wonder if something else also happened around the same time that file was quarantined, because we don't actually use maintenanceservice_installer.exe during normal updates, and I don't think we would notice if it was gone; it's only used as a fallback if it looks like the service has never been installed before on that machine. Normally we manually copy over the new maintenanceservice.exe and update the service registration with the OS. And the next update should restore the file itself anyway; the partial update would fail to apply because of the missing file and we'd transparently fall back to a complete update. Can you check if the Mozilla Maintenance Service is still listed in the Services app, and if C:\Program Files (x86)\Mozilla Maintenance Service\maintenanceservice.exe exists? The service itself being removed somehow is all I can think of that would cause behavior like this.

Flags: needinfo?(aryx.bugmail)

C:\Program Files (x86)\Mozilla Maintenance Service\maintenanceservice.exe exists (version 86.0.0.7655). Windows 'Programs and Features' claims 84.0 is installed.

There is a second Nightly installation which uses a different locale. It updates as expected and contains a META-INF folder which is missing for the install with the UAC prompt. Both installs are 32-bit.

Flags: needinfo?(aryx.bugmail)

Well, having a META-INF directory is weird, because those directories are part of the JAR format, so that sounds like that build got unpacked or something. I don't think it's related to this issue though.

So if the service itself is fine and it works for a different installation, what I'm left with is that the registration for that one copy must be what's still broken; the installer creates a registry key in HKEY_LOCAL_MACHINE\SOFTWARE\Mozilla\MaintenanceService for each installation, and as a security measure the service looks for that key and refuses to run if it's not there. The name of the key that should be there is the MD5 hash of the install directory in UTF-16; here is a CyberChef recipe that will calculate that for you. If there isn't anything there, then running an installer would be the easiest way to generate one. If there is, then I don't know what's going on and would need to ask for logs.

Flags: needinfo?(aryx.bugmail)

Thanks for guiding how to provide the information. This Nightly install has been reinstalled but I still cannot find the hash of the install location (C:\Program Files (x86)\Browser\Firefox\unstable\central) in the registry. Also tried with \ appended and both UTF-16LE and BE and with backslashes in front of the spaces. I also looked at the path of a different install and couldn't find the a matching child of HKEY_LOCAL_MACHINE\SOFTWARE\Mozilla\MaintenanceService. Is this an escaping issue or a user error?

Flags: needinfo?(aryx.bugmail)

I apologize, the error is mine; I should have included a "convert to lowercase" step. The expected hash for that path is ee9b0fa02e1500dedaee385404422c69. If the installer isn't creating that key, then I'm really not sure what's going on, because it never skips that unless it's not running with administrator privileges (and so wouldn't have permission to write it).

Okay, thanks. If that doesn't fix the problem, then I'm not sure what's up, so we would need to see the maintenance service log from the failed run, because I'm assuming it's being started and refusing to work for some reason (those logs are in C:\Program Files (x86)\Mozilla Maintenance Service\logs), plus the last pair of updater logs (which are in the updates subdirectory of the directory that's listed in about:support as the update folder). Thanks!

Flags: needinfo?(aryx.bugmail)

The issue persists.

Differences in the Maintenance Service logs:

Good:

Starting update process as the service in session 0.
Starting service with cmdline: "C:\Program Files (x86)\Mozilla Maintenance Service\update\updater.exe" C:\ProgramData\Mozilla\updates\103DF809AB5A0977\updates\0 "c:\Program Files (x86)\Browser\Firefox\unstable\central-de" "c:\Program Files (x86)\Browser\Firefox\unstable\central-de\updated" 15704/replace C:\Mozilla\L10n\Applications\Firefox\Profiles "c:\Program Files (x86)\Browser\Firefox\unstable\central-de\firefox.exe"
Process was started... waiting on result.
Process finished with return code 0.
updater.exe was launched and run successfully!
The file "C:\Program Files (x86)\Mozilla Maintenance Service\maintenanceservice_tmp.exe" is signed and the signature was verified.
Service command software-update complete.
service command MozillaMaintenance complete with result: Success.

Bad:

Starting update process as the service in session 0.
Starting service with cmdline: "C:\Program Files (x86)\Mozilla Maintenance Service\update\updater.exe" C:\ProgramData\Mozilla\updates\3EA28945215FD6AC\updates\0 "C:\Program Files (x86)\Browser\Firefox\unstable\central" "C:\Program Files (x86)\Browser\Firefox\unstable\central\updated" -1
Process was started... waiting on result.
Process finished with return code 1.
*** Warning: Error running update process. Updating update.status  (0)***
Service command software-update complete.
service command MozillaMaintenance complete with result: Failure.

Install log

Good:

Performing a replace request
PATCH DIRECTORY C:\ProgramData\Mozilla\updates\103DF809AB5A0977\updates\0
INSTALLATION DIRECTORY c:\Program Files (x86)\Browser\Firefox\unstable\central-de
WORKING DIRECTORY c:\Program Files (x86)\Browser\Firefox\unstable\central-de\updated
Begin moving destDir (c:\Program Files (x86)\Browser\Firefox\unstable\central-de) to tmpDir (c:\Program Files (x86)\Browser\Firefox\unstable\central-de.bak)
rename_file: proceeding to rename the directory
rename_file: failed to rename file - src: c:\Program Files (x86)\Browser\Firefox\unstable\central-de, dst:c:\Program Files (x86)\Browser\Firefox\unstable\central-de.bak, err: 13
PerformReplaceRequest: destDir rename attempt 1 failed. File: c:\Program Files (x86)\Browser\Firefox\unstable\central-de. Last error: 32, err: 7
rename_file: proceeding to rename the directory
rename_file: failed to rename file - src: c:\Program Files (x86)\Browser\Firefox\unstable\central-de, dst:c:\Program Files (x86)\Browser\Firefox\unstable\central-de.bak, err: 13
PerformReplaceRequest: destDir rename attempt 2 failed. File: c:\Program Files (x86)\Browser\Firefox\unstable\central-de. Last error: 32, err: 7
rename_file: proceeding to rename the directory
rename_file: failed to rename file - src: c:\Program Files (x86)\Browser\Firefox\unstable\central-de, dst:c:\Program Files (x86)\Browser\Firefox\unstable\central-de.bak, err: 13
PerformReplaceRequest: destDir rename attempt 3 failed. File: c:\Program Files (x86)\Browser\Firefox\unstable\central-de. Last error: 32, err: 7
rename_file: proceeding to rename the directory
rename_file: failed to rename file - src: c:\Program Files (x86)\Browser\Firefox\unstable\central-de, dst:c:\Program Files (x86)\Browser\Firefox\unstable\central-de.bak, err: 13
PerformReplaceRequest: destDir rename attempt 4 failed. File: c:\Program Files (x86)\Browser\Firefox\unstable\central-de. Last error: 32, err: 7
rename_file: proceeding to rename the directory
rename_file: failed to rename file - src: c:\Program Files (x86)\Browser\Firefox\unstable\central-de, dst:c:\Program Files (x86)\Browser\Firefox\unstable\central-de.bak, err: 13
PerformReplaceRequest: destDir rename attempt 5 failed. File: c:\Program Files (x86)\Browser\Firefox\unstable\central-de. Last error: 32, err: 7
rename_file: proceeding to rename the directory
rename_file: failed to rename file - src: c:\Program Files (x86)\Browser\Firefox\unstable\central-de, dst:c:\Program Files (x86)\Browser\Firefox\unstable\central-de.bak, err: 13
PerformReplaceRequest: destDir rename attempt 6 failed. File: c:\Program Files (x86)\Browser\Firefox\unstable\central-de. Last error: 32, err: 7
rename_file: proceeding to rename the directory
rename_file: failed to rename file - src: c:\Program Files (x86)\Browser\Firefox\unstable\central-de, dst:c:\Program Files (x86)\Browser\Firefox\unstable\central-de.bak, err: 13
PerformReplaceRequest: destDir rename attempt 7 failed. File: c:\Program Files (x86)\Browser\Firefox\unstable\central-de. Last error: 32, err: 7
rename_file: proceeding to rename the directory
rename_file: failed to rename file - src: c:\Program Files (x86)\Browser\Firefox\unstable\central-de, dst:c:\Program Files (x86)\Browser\Firefox\unstable\central-de.bak, err: 13
PerformReplaceRequest: destDir rename attempt 8 failed. File: c:\Program Files (x86)\Browser\Firefox\unstable\central-de. Last error: 32, err: 7
rename_file: proceeding to rename the directory
Begin moving newDir (c:\Program Files (x86)\Browser\Firefox\unstable\central-de.bak/updated) to destDir (c:\Program Files (x86)\Browser\Firefox\unstable\central-de)
rename_file: proceeding to rename the directory
Now, remove the tmpDir
ensure_remove: failed to remove file: c:\Program Files (x86)\Browser\Firefox\unstable\central-de.bak/updater.exe, rv: -1, err: 13
ensure_remove_recursive: unable to remove directory: c:\Program Files (x86)\Browser\Firefox\unstable\central-de.bak, rv: -1, err: 41
Removing tmpDir failed, err: -1
remove_recursive_on_reboot: file will be removed on OS reboot: c:\Program Files (x86)\Browser\Firefox\unstable\central-de\tobedeleted\rep9b0ecfcf-be42-467b-b0c0-490ceca307aa
succeeded
calling QuitProgressUI

Bad:

PATCH DIRECTORY C:\ProgramData\Mozilla\updates\3EA28945215FD6AC\updates\0
INSTALLATION DIRECTORY C:\Program Files (x86)\Browser\Firefox\unstable\central
WORKING DIRECTORY C:\Program Files (x86)\Browser\Firefox\unstable\central
Could not disable token privilege value: SeAssignPrimaryTokenPrivilege. (1300)
Could not disable token privilege value: SeAuditPrivilege. (1300)
Disabled unneeded token privilege: SeBackupPrivilege.
Disabled unneeded token privilege: SeCreateGlobalPrivilege.
Disabled unneeded token privilege: SeCreatePagefilePrivilege.
Could not disable token privilege value: SeCreatePermanentPrivilege. (1300)
Disabled unneeded token privilege: SeCreateSymbolicLinkPrivilege.
Could not disable token privilege value: SeCreateTokenPrivilege. (1300)
Disabled unneeded token privilege: SeDebugPrivilege.
Could not disable token privilege value: SeEnableDelegationPrivilege. (1300)
Disabled unneeded token privilege: SeImpersonatePrivilege.
Disabled unneeded token privilege: SeIncreaseBasePriorityPrivilege.
Disabled unneeded token privilege: SeIncreaseQuotaPrivilege.
Disabled unneeded token privilege: SeIncreaseWorkingSetPrivilege.
Disabled unneeded token privilege: SeLoadDriverPrivilege.
Disabled unneeded token privilege: SeLockMemoryPrivilege.
Could not disable token privilege value: SeMachineAccountPrivilege. (1300)
Disabled unneeded token privilege: SeManageVolumePrivilege.
Disabled unneeded token privilege: SeProfileSingleProcessPrivilege.
Could not disable token privilege value: SeRelabelPrivilege. (1300)
Disabled unneeded token privilege: SeRemoteShutdownPrivilege.
Disabled unneeded token privilege: SeRestorePrivilege.
Disabled unneeded token privilege: SeSecurityPrivilege.
Disabled unneeded token privilege: SeShutdownPrivilege.
Could not disable token privilege value: SeSyncAgentPrivilege. (1300)
Disabled unneeded token privilege: SeSystemEnvironmentPrivilege.
Disabled unneeded token privilege: SeSystemProfilePrivilege.
Disabled unneeded token privilege: SeSystemtimePrivilege.
Disabled unneeded token privilege: SeTakeOwnershipPrivilege.
Could not disable token privilege value: SeTcbPrivilege. (1300)
Disabled unneeded token privilege: SeTimeZonePrivilege.
Could not disable token privilege value: SeTrustedCredManAccessPrivilege. (1300)
Disabled unneeded token privilege: SeUndockPrivilege.
Could not disable token privilege value: SeUnsolicitedInputPrivilege. (1313)
UPDATE TYPE partial
PREPARE PATCH xul.dll.sig
...
EXECUTE PATCH AccessibleHandler.dll
FINISH PATCH xul.dll.sig
FINISH PATCH xul.dll
FINISH PATCH updater.exe
ensure_remove: failed to remove file: C:\Program Files (x86)\Browser\Firefox\unstable\central/updater.exe.moz-backup, rv: -1, err: 13
backup_discard: unable to remove: updater.exe.moz-backup
backup_discard: file renamed and will be removed on OS reboot: updater.exe
FINISH PATCH uninstall/helper.exe
...
FINISH PATCH AccessibleHandler.dll
succeeded
calling QuitProgressUI
NS_main: unable to remove directory: tobedeleted, err: 41
NS_main: directory will be removed on OS reboot: tobedeleted
Flags: needinfo?(aryx.bugmail)

Okay, thank you. So that's different from what I thought; the service isn't refusing to run, it's launching the updater normally but then the updater is failing. But I don't know why that's happening, so that means I need to see the other updater log, the one called backup-update.log; the updater is running a second time after that failure, and that second instance (which is the one that raises the UAC prompt) is writing its own log into last-update.log and moving the one with the actual failure log into the backup slot. Can you post that file as well when you get a chance? Thanks again, sorry for the back and forth.

Flags: needinfo?(aryx.bugmail)

Full backup-update.log content:

Performing a staged update
PATCH DIRECTORY C:\ProgramData\Mozilla\updates\3EA28945215FD6AC\updates\0
INSTALLATION DIRECTORY C:\Program Files (x86)\Browser\Firefox\unstable\central
WORKING DIRECTORY C:\Program Files (x86)\Browser\Firefox\unstable\central\updated
ensure_copy: failed to copy the file C:\Program Files (x86)\Browser\Firefox\unstable\central/maintenanceservice_installer.exe over to C:\Program Files (x86)\Browser\Firefox\unstable\central\updated/maintenanceservice_installer.exe, lasterr: 5
failed: 61
calling QuitProgressUI

No need to be sorry, I am very thankful this bug sees action so quickly.

Flags: needinfo?(aryx.bugmail)

So that error is saying that we are trying to copy maintenanceservice_installer.exe into the updated directory, but we are getting an ERROR_ACCESS_DENIED.

One of the first steps of staging an update is to copy the contents of the installation directory to the updated directory so that we can apply the update to that directory without interfering with running instances of Firefox. I'm guessing that the ERROR_ACCESS_DENIED is a result of interference from antivirus.

I don't really understand why we are getting this error code or how to address it directly. But I am seeing something that we might want to address that I believe would fix this. This is a log from a staging attempt rather than an installation attempt, which is not what I expected to see. Looking at the code involved, it looks like if staging fails then we do not attempt to use the Mozilla Maintenance Service during installation. Since installation without staging would not make this copy, I believe it would probably succeed. IIRC a failure when updating with the maintenance service will still cause a fallback to not using the service. So there seems to be no reason why we shouldn't use the service in this case.

Severity: -- → S3
Assignee: nobody → ksteuber

I am changing the way that we handle staging failures here: https://searchfox.org/mozilla-central/rev/851bbbd9d9a38c2785a24c13b6412751be8d3253/toolkit/mozapps/update/UpdateService.jsm#4322
to more closely match the way we handle staging failures here: https://searchfox.org/mozilla-central/rev/851bbbd9d9a38c2785a24c13b6412751be8d3253/toolkit/mozapps/update/UpdateService.jsm#6015

This change will allow the Mozilla Maintenance Service to be used when the partial patch fails to stage. The updater already falls back if the Maintenance Service fails, so if there is indeed a problem with the Service a UAC will still be shown.

Pushed by ksteuber@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/d26c67a77da4
When staging fails, leave the update in the same state regardless of how it failed r=nalexander

Backed out changeset d26c67a77da4 (bug 1682810) for XPCshell failures in toolkit/mozapps/update/tests/unit_service_updater/marAppApplyDirLockedStageFailureSvc_win.js. CLOSED TREE

Log:
https://treeherder.mozilla.org/logviewer?job_id=328772043&repo=autoland&lineNumber=6915

Push with failures:
https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&revision=d26c67a77da462dac9a3918f48fc57ef55eb6969

Backout:
https://hg.mozilla.org/integration/autoland/rev/43be5eca15bafb20258ae89c0752f5300126b8ef

Flags: needinfo?(ksteuber)

I'll take a look.

Flags: needinfo?(ksteuber)
Pushed by ksteuber@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6819a1a82853
When staging fails, leave the update in the same state regardless of how it failed r=nalexander
https://hg.mozilla.org/integration/autoland/rev/814285c2ebc2
Fix tests to expect the correct pending status r=application-update-reviewers,nalexander

https://hg.mozilla.org/mozilla-central/rev/6819a1a82853
https://hg.mozilla.org/mozilla-central/rev/814285c2ebc2

Status: UNCONFIRMED → RESOLVED
Closed: 3 years ago
status-firefox87: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 87 Branch
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: