1675816 - Updating Fx 83.0b8 fails

Status: RESOLVED INCOMPLETE

(Toolkit :: Application Update, defect)

Description

[Florian Bender]

This is now the second time this is happening for a 83b release, both with the Beta and the Developer Edition. The update is downloaded but fails to install. With the previous attempt (I think it was from b2 or b3), I managed to "regain" auto-update (until today) by downloading and executing the installer on the existing program folder. Note that both installations are in non-standard folders (because I have a corporate-distributed ESR in the default installation folder).

If that issue propagates to the 83 release, you risk having many people stuck on an old version of Fx because they don't know (or don't feel confident) following the "manual re-install" approach.

Log from (almost) newest attempt to update Firefox:

AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanStageUpdates - able to stage updates using the service
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanStageUpdates - able to stage updates using the service
AUS:SVC Checker: checkForUpdates, force: true
AUS:SVC UpdateService.canCheckForUpdates - able to check for updates
AUS:SVC Checker:getUpdateURL - update URL: https://aus5.mozilla.org/update/6/Firefox/83.0/20201103183834/WINNT_x86_64-msvc-x64/de/aurora/Windows_NT%2010.0.0.0.17763.1518%20(x64)/ISET:SSE4_2,MEM:16242/default/default/update.xml?force=1
AUS:SVC Checker:checkForUpdates - sending request to: https://aus5.mozilla.org/update/6/Firefox/83.0/20201103183834/WINNT_x86_64-msvc-x64/de/aurora/Windows_NT%2010.0.0.0.17763.1518%20(x64)/ISET:SSE4_2,MEM:16242/default/default/update.xml?force=1
AUS:SVC Checker:onLoad - request completed downloading document
AUS:SVC Checker:onLoad - Getting sslStatus failed.
AUS:SVC Checker:onLoad - number of updates available: 1
AUS:SVC getCanApplyUpdates - testing write access C:\ProgramData\Mozilla\updates\CA9422711AE1A81C\update.test
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanApplyUpdates - bypass the write checks because the Windows Maintenance Service can be used
AUS:SVC Creating Downloader
AUS:SVC UpdateService:_downloadUpdate
AUS:SVC readStringFromFile - file doesn't exist: C:\ProgramData\Mozilla\updates\CA9422711AE1A81C\updates\0\update.status
AUS:SVC readStatusFile - status: null, path: C:\ProgramData\Mozilla\updates\CA9422711AE1A81C\updates\0\update.status
AUS:SVC getCanUseBits - BITS can be used to download updates
AUS:SVC Downloader:_canUseBits - Patch is able to use BITS download
AUS:SVC Downloader:downloadUpdate - Starting BITS download with url: https://download.mozilla.org/?product=devedition-83.0b9-partial-83.0b8&os=win64&lang=de, updateDir: C:\ProgramData\Mozilla\updates\CA9422711AE1A81C\updates\0, filename: update.mar
AUS:SVC Downloader:downloadUpdate - BITS download running. BITS ID: {D203129C-3697-498A-AA29-B11FEE1AAAC0}
AUS:SVC Downloader:onProgress - progress: 524288/6186036
AUS:SVC Downloader:onProgress - progress: 2883584/6186036
AUS:SVC Downloader:onProgress - progress: 5505024/6186036
AUS:SVC Downloader:onProgress - progress: 6186036/6186036
AUS:SVC Downloader:onStopRequest - downloader: BITS, status: 0
AUS:SVC Downloader:onStopRequest - status: 0, current fail: 0, max fail: 10, retryTimeout: 2000
AUS:SVC Downloader:_verifyDownload called
AUS:SVC Downloader:_verifyDownload downloaded size == expected size.
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanStageUpdates - able to stage updates using the service
AUS:SVC Downloader:onStopRequest - setting state to: pending-service
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanStageUpdates - able to stage updates using the service
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanStageUpdates - able to stage updates using the service
AUS:SVC Downloader:onStopRequest - attempting to stage update: Firefox Developer Edition 83.0 Beta 9
AUS:SVC readStatusFile - status: failed: 57, path: C:\ProgramData\Mozilla\updates\CA9422711AE1A81C\updates\0\update.status
AUS:SVC readStringFromFile - file doesn't exist: C:\ProgramData\Mozilla\updates\CA9422711AE1A81C\updates\0\bt.result
AUS:SVC readBinaryTransparencyResult - result: null, path: C:\ProgramData\Mozilla\updates\CA9422711AE1A81C\updates\0\bt.result
AUS:SVC handleFallbackToCompleteUpdate - install of partial patch failed, downloading complete patch
AUS:SVC Creating Downloader
AUS:SVC UpdateService:_downloadUpdate
AUS:SVC readStringFromFile - file doesn't exist: C:\ProgramData\Mozilla\updates\CA9422711AE1A81C\updates\0\update.status
AUS:SVC readStatusFile - status: null, path: C:\ProgramData\Mozilla\updates\CA9422711AE1A81C\updates\0\update.status
AUS:SVC Downloader:_selectPatch - found existing patch with state: null
AUS:SVC getCanUseBits - BITS can be used to download updates
AUS:SVC Downloader:_canUseBits - Patch is able to use BITS download
AUS:SVC Downloader:downloadUpdate - Starting BITS download with url: https://download.mozilla.org/?product=devedition-83.0b9-complete&os=win64&lang=de, updateDir: C:\ProgramData\Mozilla\updates\CA9422711AE1A81C\updates\0, filename: update.mar
AUS:SVC promiseLangPacksUpdated - waiting for language pack updates to stage.
AUS:SVC UpdateManager:refreshUpdateStatus - Notifying observers that the update was staged. topic: update-staged, status: downloading
AUS:SVC Downloader:downloadUpdate - BITS download running. BITS ID: {F1E0C1A3-7C2E-421A-B93F-3FB7A25A6F10}
AUS:SVC Downloader:onProgress - progress: 2097152/61453116
AUS:SVC Downloader:onProgress - progress: 4718592/61453116
AUS:SVC Downloader:onProgress - progress: 7340032/61453116
AUS:SVC Downloader:onProgress - progress: 9699328/61453116
AUS:SVC Downloader:onProgress - progress: 12320768/61453116
AUS:SVC Downloader:onProgress - progress: 14680064/61453116
AUS:SVC Downloader:onProgress - progress: 17301504/61453116
AUS:SVC Downloader:onProgress - progress: 19398656/61453116
AUS:SVC Downloader:onProgress - progress: 21757952/61453116
AUS:SVC Downloader:onProgress - progress: 24379392/61453116
AUS:SVC Downloader:onProgress - progress: 27000832/61453116
AUS:SVC Downloader:onProgress - progress: 29622272/61453116
AUS:SVC Downloader:onProgress - progress: 31981568/61453116
AUS:SVC Downloader:onProgress - progress: 34340864/61453116
AUS:SVC Downloader:onProgress - progress: 36700160/61453116
AUS:SVC Downloader:onProgress - progress: 39321600/61453116
AUS:SVC Downloader:onProgress - progress: 41943040/61453116
AUS:SVC Downloader:onProgress - progress: 44302336/61453116
AUS:SVC Downloader:onProgress - progress: 46661632/61453116
AUS:SVC Downloader:onProgress - progress: 49283072/61453116
AUS:SVC Downloader:onProgress - progress: 51904512/61453116
AUS:SVC Downloader:onProgress - progress: 53739520/61453116
AUS:SVC Downloader:onProgress - progress: 56098816/61453116
AUS:SVC Downloader:onProgress - progress: 58458112/61453116
AUS:SVC Downloader:onProgress - progress: 61079552/61453116
AUS:SVC Downloader:onProgress - progress: 61453116/61453116
AUS:SVC Downloader:onStopRequest - downloader: BITS, status: 0
AUS:SVC Downloader:onStopRequest - status: 0, current fail: 0, max fail: 10, retryTimeout: 2000
AUS:SVC Downloader:_verifyDownload called
AUS:SVC Downloader:_verifyDownload downloaded size == expected size.
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanStageUpdates - able to stage updates using the service
AUS:SVC Downloader:onStopRequest - setting state to: pending-service
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanStageUpdates - able to stage updates using the service
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanStageUpdates - able to stage updates using the service
AUS:SVC Downloader:onStopRequest - attempting to stage update: Firefox Developer Edition 83.0 Beta 9
AUS:SVC readStatusFile - status: failed: 57, path: C:\ProgramData\Mozilla\updates\CA9422711AE1A81C\updates\0\update.status
AUS:SVC readStringFromFile - file doesn't exist: C:\ProgramData\Mozilla\updates\CA9422711AE1A81C\updates\0\bt.result
AUS:SVC readBinaryTransparencyResult - result: null, path: C:\ProgramData\Mozilla\updates\CA9422711AE1A81C\updates\0\bt.result
AUS:SVC handleFallbackToCompleteUpdate - install of complete or only one patch offered failed. Notifying observers. topic: update-error, status: unknown, update.patchCount: 2, oldType: complete
AUS:SVC promiseLangPacksUpdated - waiting for language pack updates to stage.
AUS:SVC UpdateManager:refreshUpdateStatus - Notifying observers that the update was staged. topic: update-staged, status: failed
AUS:SVC UpdateManager:_writeUpdatesToXMLFile - no updates to write. removing file: C:\ProgramData\Mozilla\updates\CA9422711AE1A81C\active-update.xml

Comment 1

[Robin Steuber (she/her) [:bytesized]]

Hi there! Thank you for taking the time to report this.

I see an error code 57 in there, which is SERVICE_INSTALL_DIR_REG_ERROR. I'll give some explanation of what this means, but you can skip down to the last paragraph if you aren't interested.

This all has to do with the Mozilla Maintenance Service. The Maintenance Service is installed in such a way that it can run with the elevated permissions necessary to install updates to Firefox. This allows updates to be installed without the user having to give permission every time (i.e. without showing a UAC prompt). However, this is slightly complicated by the fact that the updater itself is not part of the Maintenance Service, so the Maintenance Service has to run the updater from the Firefox directory with the necessary elevated permissions. But we need to be very careful that the updater we run is indeed our updater. We don't want to be tricked into running something malicious with elevated permissions.

When the Firefox installer runs, if it has the permissions to install the Maintenance Service and the user didn't tell it not to use the Maintenance Service, it will write out a registry key that is named using a hash of the Firefox installation directory. Within that key, some certificate information is written. This serves two purposes: It allows Firefox installations to opt-in to using the Maintenance Service (so that installations that are not configured to use the Maintenance Service will not be able to), and it allows the Maintenance Service to verify the certificate used to sign the updater to make sure that it is in fact an updater and not something malicious.

The error that you are getting indicates that this registry key is missing. Since it sounds like automatic updates were previously working for this installation, this suggests that either something happened to the registry key, or the path to the installation changed (causing the path hash to change, causing the Maintenance Service to look in the wrong location for the key).

I don't know of anything in recent versions of Firefox that could have caused such a problem. Do you know of anything on your system that might have caused the location of the Firefox installation directory to change? Is Firefox installed on a removable or remote drive that doesn't have a consistent path? Or maybe one of the directories that contains the installation got renamed?

Comment 2

[Florian Bender]

Thanks a lot, I feel we can work this out. Given your explanation, and if we assume the problem is not caused by a code change, my working theory is that the corp distribution process of the ESR (via Microsoft SCCM) may muck with the maintenance service and/or registry. That‘s at least the only thing I can think of that changed on the system: New ESR update distributed and installed (which sometimes leads to my Beta to being uninstalled, but I immediately re-install it afterwards). I‘m not 100% certain on the timeline, though, so I‘d need a few more hints where to look (registry keys, logs, etc) for next time it happens.

All my installations are in the Windows program files folder (i.e. nothing unusual) with just the beta in a non-default folder, which used to be called „Mozilla Firefox Beta“ but after the last „automatic“ uninstall, I‘ve now re-installed it at „Firefox Beta“ (in the hope to trick the corp distribution system to not uninstall upon next ESR update). DEV is in the (iirc) default „Firefox Developer Edition“. Auto Updates worked multiple times before.

Comment 3

[BugBot [:suhaib / :marco]]

The severity field is not set for this bug.
:nalexander, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 4

[Nick Alexander :nalexander [he/him] (PTO August 22-September 12)]

Florian: sorry to not respond to your comment. Given that you haven't commented further can you confirm that this issue has not happened again?

Kirk: can you answer Florian's question about "a few more hints" where to look to try to understand what's happening? Based on my quick read I'd look in about:support to find the update directory, which, IIRC, contains the update hash, and then I'd look in the registry under HKLM\SOFTWARE\Mozilla\MaintenanceService\$HASH and expect to find keys 0 and 1 that are populated. But I see now that the update hash and the hash in the registry aren't the same, so perhaps Kirk can also suggest how to find the correct hash from within the product.

Thanks, all!

Comment 5

[Nick Alexander :nalexander [he/him] (PTO August 22-September 12)]

S4 'cuz this is not confirmed yet, let alone widely seen.

Comment 6

[Robin Steuber (she/her) [:bytesized]]

(In reply to Nick Alexander :nalexander [he/him] from comment #4)

Kirk: can you answer Florian's question about "a few more hints" where to look to try to understand what's happening? Based on my quick read I'd look in about:support to find the update directory, which, IIRC, contains the update hash, and then I'd look in the registry under HKLM\SOFTWARE\Mozilla\MaintenanceService\$HASH and expect to find keys 0 and 1 that are populated. But I see now that the update hash and the hash in the registry aren't the same, so perhaps Kirk can also suggest how to find the correct hash from within the product.

Yeah, the update hash is a City Hash, but the one used by the maintenance service key is an MD5 Hash. Specifically, an MD5 Hash of the lowercase installation directory, not including the trailing slash, encoded as a wide string. So, for example, my Firefox binary is located at C:\Program Files\Firefox Nightly\firefox.exe, so I would take the MD5 hash of a wide string representing: c:\program files\firefox nightly.

This Utility can do the wide string conversion and the MD5 calculation. If I paste c:\program files\firefox nightly into the input box, I get 35562fadc262dec332219264bffef2fb as the output. And, as expected, if I open regedit.exe from the Run window (Win+R to open the Run window), I can see the HKEY_LOCAL_MACHINE\SOFTWARE\Mozilla\MaintenanceService\35562fadc262dec332219264bffef2fb key, which contains some sub-keys with certificate information.

If this key is initially present, but is missing when you encounter this problem, that would explain why the problem is happening.

Comment 7

[Al]

@Florian Bender : I saw your link on my thread, I also distribute Firefox with SCCM, and install it in the default folder.
Even after pushing an update from 68 to 78, the problem is still there.

Comment 8

[[e-mail] (:bhearsum)]

Closing this due to inactivity. If you're still experiencing the bug, please feel free to re-open with the information requested from comment #6.