1600080 - Firefox can't auto-update to the latest version because maintenance service seems to not run properly

Status: UNCONFIRMED

(Toolkit :: Application Update, defect, P3)

Description

[Thomas]

Attachment: maintenanceservice.log

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.45 Safari/537.36 Edg/79.0.309.30

Steps to reproduce:

When executing Firefox while pending for an update, Firefox updater is asking for admin rights (Windows UAC pop up) to update Firefox.
I tried to delete configuration files we previously deployed (policies.json and autoconfig.js), but the problem still occurs.
I didn't find how to reproduce the problem on others computers yet.

Actual results:

I'm currently running Firefox 68.1. The upgrade from 68.0 to 68.1 works properly without admins right. However, it doesn't work from 68.1 to current version.
Here are log entries about update services :
AUS:SVC Creating UpdateService
AUS:SVC Logging current UpdateService status:
AUS:SVC UpdateService.canCheckForUpdates - able to check for updates
AUS:SVC getCanApplyUpdates - testing write access C:\ProgramData\Mozilla\updates\308046B0AF4A39CB\update.test
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanApplyUpdates - bypass the write checks because the Windows Maintenance Service can be used
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanStageUpdates - able to stage updates using the service
AUS:SVC Elevation required: false
AUS:SVC Update being handled by other instance: false
AUS:SVC Downloading: false
AUS:SVC End of UpdateService status
AUS:SVC readStatusFile - status: failed: 9, path: C:\ProgramData\Mozilla\updates\308046B0AF4A39CB\updates\0\update.status
AUS:SVC getCanApplyUpdates - testing write access C:\ProgramData\Mozilla\updates\308046B0AF4A39CB\update.test
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanApplyUpdates - bypass the write checks because the Windows Maintenance Service can be used
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanStageUpdates - able to stage updates using the service
AUS:SVC getCanUseBits - Not using BITS because of proxy usage
AUS:SVC isServiceInstalled - returning true
AUS:SVC UpdateService.canCheckForUpdates - able to check for updates
AUS:SVC getCanApplyUpdates - testing write access C:\ProgramData\Mozilla\updates\308046B0AF4A39CB\update.test
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanApplyUpdates - bypass the write checks because the Windows Maintenance Service can be used
AUS:SVC readStatusFile - status: failed: 9, path: C:\ProgramData\Mozilla\updates\308046B0AF4A39CB\updates\0\update.status 2
AUS:SVC readStringFromFile - file doesn't exist: C:\ProgramData\Mozilla\updates\308046B0AF4A39CB\updates\0\bt.result
AUS:SVC readBinaryTransparencyResult - result: null, path: C:\ProgramData\Mozilla\updates\308046B0AF4A39CB\updates\0\bt.result
AUS:SVC handleUpdateFailure - notifying observers of error. topic: update-error, status: elevation-attempts-exceeded
AUS:SVC Checker:getUpdateURL - update URL: https://aus5.mozilla.org/update/6/Firefox/68.1.0/20190826132627/WINNT_x86_64-msvc-x64/fr/esr/Windows_NT%2010.0.0.0.18362.476%20(x64)/ISET:SSE4_2,MEM:8076/default/default/update.xml
AUS:SVC UpdateService.canCheckForUpdates - able to check for updates
AUS:SVC Checker: checkForUpdates, force: false
AUS:SVC Creating UpdateService
AUS:SVC Logging current UpdateService status:
AUS:SVC UpdateService.canCheckForUpdates - able to check for updates
AUS:SVC getCanApplyUpdates - testing write access C:\ProgramData\Mozilla\updates\308046B0AF4A39CB\update.test
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanApplyUpdates - bypass the write checks because the Windows Maintenance Service can be used
AUS:SVC isServiceInstalled - returning true
AUS:SVC shouldUseService - returning true
AUS:SVC getCanStageUpdates - able to stage updates using the service
AUS:SVC Elevation required: false
AUS:SVC Update being handled by other instance: false
AUS:SVC Downloading: false
AUS:SVC End of UpdateService status
AUS:SVC UpdateService.canCheckForUpdates - able to check for updates
AUS:SVC Checker:getUpdateURL - update URL: https://aus5.mozilla.org/update/6/Firefox/68.1.0/20190826132627/WINNT_x86_64-msvc-x64/fr/esr/Windows_NT%2010.0.0.0.18362.476%20(x64)/ISET:SSE4_2,MEM:8076/default/default/update.xml
AUS:SVC Checker:checkForUpdates - sending request to: https://aus5.mozilla.org/update/6/Firefox/68.1.0/20190826132627/WINNT_x86_64-msvc-x64/fr/esr/Windows_NT%2010.0.0.0.18362.476%20(x64)/ISET:SSE4_2,MEM:8076/default/default/update.xml
AUS:SVC Checker:onLoad - request completed downloading document
AUS:SVC Checker:onLoad - Getting sslStatus failed.
AUS:SVC Checker:onLoad - number of updates available: 1

Expected results:

I think the problem is from maintenance service. It tries to run but looks to get a warning that was not here before (See attachment)
Do you know how to repair it?

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The priority flag is not set for this bug.
:robert.strong.bugs, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 2

[Robert Strong (they/them - no direct email)]

The maintenance service log shows that firefox.exe is launching the maintenance service without the minimum required arguments.

Do you have or have you had multiple installs of Firefox on this system?
Is this error happening on any other systems in your environment?
Could you try reinstalling 68.0 to see if you can reproduce it?

Comment 3

[Thomas]

There was no multiple installations of Firefox on this system at any time.

This has happened on another system before, but currently it only happens on this system. It was previously fixed by reinstalling Firefox.
Reinstalling Firefox on the system also fixed the problem. How can we identify a broken maintenance service on a system, and how to prevent this problem to appears?

Comment 4

[Robert Strong (they/them - no direct email)]

Since there aren't other bug reports this is likely due to something unique to your systems. Perhaps an administrative task. Can you provide a list of the changes made to your systems?

Comment 5

[Thomas]

There was no major changes on the system except Windows 10 build update from 1809 to 1903. Could that have caused the issue?

Comment 6

[Robert Strong (they/them - no direct email)]

There haven't been any reports of update 1903 causing this so it is doubtful. I also ran update 1903 after it came out without any issues.

It might shed some light if you, on an affected system, could uninstall the "Mozilla Maintenance Service" in "Add or remove programs" and then install it again using maintenanceservice_installer.exe in the installation directory. If it fixes it then it might point to this due to permissions getting reset.

Comment 7

[Robert Strong (they/them - no direct email)]

Clearing priority so this will get attention

Comment 8

[Thomas]

Thanks for your feedback.

As I don't have any system with the bug since I reinstalled firefox on the affected computer, I can't test to reinstall Maintenance Service right now.
I will test it the next time a system will have this problem.

Comment 9

[Thomas]

I have a new computer with the issue. Firefox was installed yesterday on the system via a deployment package in 68.0 version (ESR).
Since Firefox doesn't update to the current version, I tried to uninstall maintenance service and reinstall it, but it doesn't fix the issue. Then I uninstalled Firefox and redeployed with the same deployment package and now it works...

Any ideas?

Comment 10

[Robert Strong (they/them - no direct email)]

I don't have any ideas as to why uninstalling the maintenance service and then re-installing didn't fix it and redeploying did fix it.

Comment 11

[Rachel Tublitz [:rachel]]

Can you also attach the maintenanceservice.log file for the new system with the problem, as well as include any relevant log entries? It might help us to see if there are any differences between these and the ones you found on the other machine with the problem.

Comment 12

[Thomas]

Attachment: logs.zip

Here you can find maintenanceservice.log and maintenanceservice-install.log

Comment 13

[Lucas]

Hi, i work with Thomas and we still have this issue with UAC on few computers.
Currently on Windows 10 build 1909 and Firefox ESR 68.0 and maintenance service 68.0. The update from 68.0 to 68.9 does not work on some computer.

I see two things :

• On the computer where the update works, maintenanceservice-install.log is like this :
  « Upgrading service if installed...
  User access was set successfully on the service.
  The Mozilla Maintenance service path is correct.
  The service description was updated successfully.
  Sending stop request...
  … »
  On the computer where it does not work :
  « Installing service...
  User access was set successfully on the service.
  The service description was updated successfully.
  The service was installed successfully »
  Nothing about the Mozilla Maintenance service path, is this normal ?

• In addition, sometimes the owner of directories C:\Program Files\Mozilla Firefox and C:\Program Files (x86)\Mozilla Maintenance Service is SYSTEM and sometimes hostname\administrators can this be a problem ?

Regards

Comment 14

[Lucas]

Attachment: 2020-07-01 15_46_39-Window.png - UAC details

Comment 15

[Thomas]

Comment on attachment 9160666 [details]
2020-07-01 15_46_39-Window.png - UAC details

Can you please delete this attachment?
Regards

Comment 16

[David Lawrence [:dkl]]

The content of attachment 9160666 [details] has been deleted for the following reason:

Deleted at request of bug reporter

Comment 17

[Rachel Tublitz [:rachel]]

Hi Lucas,

I'm sorry you're running into this issue! Since we just had point release for 68.x ESR on June 30th, would you mind double-checking that you're still unable to update now that the latest version is 68.10 vs 68.9 as you noted in your details here? It would be good to rule 68.x < 68.10 out as the culprit; if you're still seeing the issue let us know and we'll keep digging.

Also, if you are still seeing the problem, can you confirm specifically what deployment package you mentioned above? Does downloading and installing 68.0 from the 68.0 installer for your system yield the same issues? If so, can you link us to the exact installer you're using?

I just tested by installing against this specific version: https://ftp.mozilla.org/pub/firefox/releases/68.0esr/win64/en-US/Firefox%20Setup%2068.0esr.exe of the installer on Windows version 1909 and was able to manually initiate an upgrade (via Help > About Firefox) to 68.10.0esr (64 bit).

Comment 18

[Lucas]

Attachment: UAC that appears when I launch Firefox

Hi Rachel,
the problem is the same today to go to version 68.10 as you can see in attachement.
I use the 68.0 msi package
regards

Comment 19

[Lucas]

Attachment: esr 68.0 msi details

Comment 20

[Lucas]

Comment on attachment 9161197 [details]
esr 68.0 msi details

You can see the package details

Comment 21

[Molly Howell (she/her) (no longer active)]

Rachel is out this week, so I'll try to take over here for now. I'm afraid I don't have much though.

First, thanks for the info. That installer package looks to be in order.

(In reply to Lucas from comment #13)

On the computer where the update works, maintenanceservice-install.log is like this :
« Upgrading service if installed...
User access was set successfully on the service.
The Mozilla Maintenance service path is correct.
The service description was updated successfully.
Sending stop request...
… »
On the computer where it does not work :
« Installing service...
User access was set successfully on the service.
The service description was updated successfully.
The service was installed successfully »
Nothing about the Mozilla Maintenance service path, is this normal ?

That's a good catch, thanks for pointing that out. I have to admit to being pretty confused about this. Looking through the code, I don't see any way for neither that message nor its opposite (the one about the path being incorrect) to show up in an install log. Unless the version of the service that generated that log is from before the function that prints the message was added, but that was in Firefox 19, so that seems unlikely.

(In reply to Lucas from comment #13)

• In addition, sometimes the owner of directories C:\Program Files\Mozilla Firefox and C:\Program Files (x86)\Mozilla Maintenance Service is SYSTEM and sometimes hostname\administrators can this be a problem ?

That's also really odd. On Windows we don't set any permissions or ownership on either the Firefox installation or the maintenance service's files, we just allow them to be inherited from Program Files (because whatever is there is what we're going to want anyway). But also, the maintenance service runs as SYSTEM, so it ought to have the access it needs in either case. So this might be a clue in some way, but I don't think it's the cause.

The next thing I think we need to see would be the updater's log files, since the maintenance service ones don't really look suspicious. Right after a failed update attempt (or any time before the next attempt), can you get the files last-update.log and backup-update.log from C:\ProgramData\Mozilla\updates\308046B0AF4A39CB\updates and attach those? Thanks again.

Comment 22

[Lucas]

Attachment: backup-update.log

Comment 23

[Lucas]

Attachment: last-update.log

Hi Molly,

Thank you for this information.
You can find both files in the attachment.

regards

Comment 24

[Lucas]

Hi
do you have any ideas after looking at the files i sent you
regards

Comment 25

[Rachel Tublitz [:rachel]]

Attachment: last-update__working.log

Adding a last-update.log file here for comparison where updates were working on my win10 machine when updating from 68.0.esr (fr/win64) - https://ftp.mozilla.org/pub/firefox/releases/68.0esr/win64/fr/Firefox%20Setup%2068.0esr.msi to 68.11.0esr. Interestingly, we do have a difference between the files provided and what I'm seeing. Ie, mine start with Performing a staged update and continue after the WORKING DIRECTORY line, whereas the problem updates seem to die between WORKING DIRECTORY and UPDATE TYPE complete.

Comment 26

[Rachel Tublitz [:rachel]]

In trying to understand what might be different about your system than mine, perhaps there's a lingering profile? Lucas, can you also try refreshing Firefox to clear profile data on the machine just to rule that out as a possible issue?

Comment 27

[Lucas]

Hello Rachel. Actually I use persistents profiles. I tried to delete all the old profiles and create a new one, the problem is the same, the update starts UAC

Comment 28

[Rachel Tublitz [:rachel]]

Thanks, Lucas. To help us continue to debug this, can you also provide a few more details on your system by visiting about:support on one of the problem machines, clicking Copy raw data to clipboard and attaching that here?

Comment 29

[Lucas]

Attachment: about support

Rachel, here is the export attached

Comment 30

[Lucas]

Hi Rachel, do you found something ?
regards

Comment 31

[Rachel Tublitz [:rachel]]

Hi Lucas,

Thanks for including that. I'm not seeing anything notable in what you've attached that might help us pinpoint the problem. Molly, can you think of anything else we can use to help debug what's going on here?

Comment 32

[Molly Howell (she/her) (no longer active)]

I really don't have any ideas, sorry. :(
What it looks like is that the maintenance service is starting, but then not really doing anything, and also not logging about it. I don't know what could cause that scenario that reinstalling wouldn't fix.

Comment 33

[custom.firefox.lady [:tawn]]

Not certain if this is related, but running release Fx on Win7, Fx updates continually fail even from clean profile (unless using 'Run as administrator' or install a fresh copy as suggested, but that needs repeated for every update). Appears to be MMS related since disabling it via Options enables the UAC to popup and successfully install update. This has been occurring for maybe the last half-dozen or so updates.