Open Bug 1828245 Opened 1 year ago Updated 2 months ago

Windows updater requires elevated privileges

Categories

(Toolkit :: Application Update, defect, P3)

Product:
Toolkit
Component:
Application Update
Version:
Firefox 111
Type:
defect

Tracking

()

Status:
UNCONFIRMED

People

(Reporter: bugzilla, Unassigned, NeedInfo)

References

Details

Attachments

(14 files, 6 obsolete files)

Update Error Doorhanger.png
31.31 KB, image/png
Details
about:support troubleshooting output with paranoid redactions
48.06 KB, text/plain
Details
maintenanceservice.log
4.90 KB, text/plain
Details
256-bit Hex ID REDACTED.log in C:\Program Files (x86)\Mozilla Maintenance Service\UpdateLogs
7.38 KB, text/plain
Details
update browser console log
5.09 KB, text/plain
Details
Another UAC elevation prompt
314.59 KB, image/png
Details
maintenanceservice.log
4.90 KB, text/plain
Details
update_messages_old.log
9.73 KB, text/plain
Details
update_messages-9.log
4.96 KB, text/plain
Details
Firefox_Update_and_restart.PNG
7.62 KB, image/png
Details
UAC_elevation_prompt.jpg
5.71 MB, image/jpeg
Details
last-update.log
1.23 KB, text/plain
Details
backup-update.log
1.57 KB, text/plain
Details
update_messages*.log
10.86 KB, application/x-zip-compressed
Details

Steps to reproduce:

As per https://support.mozilla.org/en-US/questions/1410763 we occasionally run into instances where deployments of Firefox rapid release on Windows 10 in our enterprise require elevated privileges for the updater to execute on the workstation. This seems to be happening only for a small minority of users.

Actual results:

In the most recent instance, Firefox 111.0.1 advised the user that they need to update manually, and then requested elevated privileges to do so (users do not have Admin privileges on the workstation). After support staff provided elevated privileges the update to Firefox 112.0 was completed successfully. The Mozilla Maintenance Service appears to be successfully installed on this workstation, albeit reporting an earlier version (106.0.5) than Firefox.

Expected results:

This seems to be a sporadic issue with most deployments updating for users without requiring elevated privileges, and so I'm asking for assistance on how we can troubleshoot this further.

(In reply to oneofthedamons from comment #0)

In the most recent instance, Firefox 111.0.1 advised the user that they need to update manually, and then requested elevated privileges to do so (users do not have Admin privileges on the workstation).

Do you meant that Firefox showed a notification like the one I attached, indicating that automatic update failed? And then elevation was required when the user clicked "Download" and ran the installer? Or that Firefox attempted to install an update normally (which is to say, at Firefox startup) and it needed elevation to do so and thus showed a UAC prompt?

Could you additionally tell me whether these users are on dedicated machines (ex: Windows 10 Pro) or whether they use a shared machine (such as Windows Small Business Server and/or using software like Citrix)? Is Firefox installed into the %PROGRAMFILES% directory?

Flags: needinfo?(bugzilla)
Attached image Firefox update notification (obsolete) —

Do you meant that Firefox showed a notification like the one I attached, indicating that automatic update failed? And then elevation was required when the user clicked "Download" and ran the installer? Or that Firefox attempted to install an update normally (which is to say, at Firefox startup) and it needed elevation to do so and thus showed a UAC prompt?

The second: Firefox attempted to install an update normally (at startup) and it needed elevation. See attached screen grab.

Could you additionally tell me whether these users are on dedicated machines (ex: Windows 10 Pro) or whether they use a shared machine (such as Windows Small Business Server and/or using software like Citrix)?

Dedicated workstations, nothing virtualised.

Is Firefox installed into the %PROGRAMFILES% directory?

Yes.

Just had another ticket with the same issue, this time needing to elevate to update 112.0 -> 112.0.1. I've attached the about:support Troubleshooting Information output for that one. Please let me know what else you need.

Flags: needinfo?(bugzilla)

(In reply to oneofthedamons from comment #4)

The second: Firefox attempted to install an update normally (at startup) and it needed elevation. See attached screen grab.

To be completely clear here, is it true that a UAC prompt is also needed in order to install the update? Because the screen capture that you have sent isn't a request for elevation. That is just the dialog that is shown when there is an update available if you have gone to the "Update" section of about:preferences and selected the "Check for updates but let you choose to install them" setting. If everything else is working as expected, it doesn't indicate that elevation will be needed to update.

When are your users seeing these UAC prompts? When Firefox is launched? Or at some other time?

It would be very helpful to get an update browser console log, but it probably won't be doable to collect one unless there are machines that can reproduce the issue consistently. But it sounds like this is an intermittent issue, is that correct? If possible, you can collect an update browser console log by following these steps when the installation is not at the most current version:

  1. Navigate to about:config.
  2. Set app.update.log to true.
  3. Open the Browser Console either with the hotkey Control+Shift+J (Command+Shift+J on macOS), or via Hamburger Menu->More Tools->Browser Console
  4. In the Filter textbox at the top, enter AUS: to filter out everything except the update messages.
  5. Navigate to the "Update" section of about:preferences. It should automatically check for an update.
  6. Once the update check has completed, copy the messages out of the Browser Console and attach them to this bug.

It would also be helpful to get the logs from the updater itself. These are written to a file, so as long as Firefox hasn't updated again since the problem was encountered, they may still contain helpful information.

  1. Navigate to about:support
  2. Find the "Update Folder" entry and click "Open Folder".
  3. Open the updates directory.
  4. Inside, you should find files named last-update.log and backup-update.log. Attach both of these files to this bug.

When you collect those, please simultaneously collect the maintenance service log from C:\Program Files (x86)\Mozilla Maintenance Service\logs\maintenanceservice.log and attach it to this bug.

Flags: needinfo?(bugzilla)

To be completely clear here, is it true that a UAC prompt is also needed in order to install the update? Because… [snip]

Yes, I'll attach a photo of the elevation prompt when the user sends to me.

When are your users seeing these UAC prompts? When Firefox is launched? Or at some other time?

When Firefox is launched.

It would be very helpful to get an update browser console log, but it probably won't be doable… [snip]

The next time it is reported I will do so. We've averaging about 1 per week at the moment so shouldn't be too long if future performance is a reliable indicator of past performance.

It would also be helpful to get the logs from the updater itself. These are written to a file, so as long as Firefox hasn't updated again since the problem was encountered…

It has been updated since by me supplying elevation, unfortunately (in hindsight upon starting Firefox I should have pressed "Cancel" on the elevation prompt, especially since it was not a security-related bugfix — next time).

When you collect those, please simultaneously collect the maintenance service log…

This one I did think to collect before providing elevation to complete to update. Please see attached. Note this log was taken after the UAC prompt timed out (while the user was waiting for me to attend their workstation), but before I started Firefox again and provided elevation to complete the update.

Flags: needinfo?(bugzilla)
Attached file maintenanceservice.log —

Inside, you should find files named last-update.log and backup-update.log. Attach both of these files to this bug.

Looking at my workstation it looks like the content of last-update.log is replicated into C:\Program Files (x86)\Mozilla Maintenance Service\UpdateLogs which I did think to gather. Assuming my observation is correct, I'll attach that log too.

Please delete previous.

Attachment #9329163 - Attachment is obsolete: true
Attached image UAC elevation prompt (obsolete) —

The Bugbug bot thinks this bug should belong to the 'Toolkit::Application Update' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Component: Untriaged → Application Update
Product: Firefox → Toolkit
Severity: -- → S3
Priority: -- → P3

Sorry for the delay at looking at this, I've been out.

(In reply to oneofthedamons from comment #6)

It would be very helpful to get an update browser console log, but it probably won't be doable… [snip]

The next time it is reported I will do so. We've averaging about 1 per week at the moment so shouldn't be too long if future performance is a reliable indicator of past performance.

This may not really be sufficient to collect the logs that I need. The Browser Console log that I described collecting really needs to be collected when downloading the problem-causing update in order to be useful. Collecting the log after the fact will really only tell us about problems checking for new updates, not problems installing the previous update.

So if you can't cause or predict the issue, it probably won't be doable to collect a useful log here. Which is unfortunate, because this is the log that would be most likely to provide an explanation of what's going wrong here.

Were the logs from Comment 7 and Comment 10 collected directly after an update that improperly showed a UAC prompt?

Are there any other hints that you can give that might help explain things here? Does the problem always happen to the same few machines? Or does it happen to machines seemingly at random? Are there any commonalities between the machines that are having these problems?

What about characteristics of the machines themselves? Are they horrendously slow? Do they tend to have extremely low free hard drive space?

Could you tell me what happens next after a user reports such a problem? Is someone able to accept the UAC? Does Firefox try to update again later? Does it succeed?

Flags: needinfo?(bugzilla)

This may not really be sufficient to collect the logs that I need. The Browser Console log that I described collecting really needs to be collected when downloading the problem-causing update in order to be useful. Collecting the log after the fact will really only tell us about problems checking for new updates, not problems installing the previous update.

Is there a way to set app.update.log to true through policies? We can push this out easily through Group Policy while we troubleshoot.

Were the logs from Comment 7 and Comment 10 collected directly after an update that improperly showed a UAC prompt?
They were collected subsequent to the UAC prompt being incorrectly shown and then timing-out, but prior to the update being applied successfully by providing credentials to elevate.

Are there any other hints that you can give that might help explain things here? Does the problem always happen to the same few machines?
The problem seems to be resolved once an elevated update has been allowed. For example, on 2 workstations where we elevated to allow an update, subsequent update(s) to 112.0.2 have occurred without elevation.

Or does it happen to machines seemingly at random? Are there any commonalities between the machines that are having these problems?

What about characteristics of the machines themselves? Are they horrendously slow? Do they tend to have extremely low free hard drive space?
Nothing comes to mind. There is nothing these 3 workstations have in common that they don't share with the 150+ others. Well, the only thing that does come to mind about our users is their resistance to restarting Windows ever (the majority of our support tickets are things breaking because users resist restarting Windows despite training and reminders). Not sure if this would explain this issue, but thought it worth mentioning.

Could you tell me what happens next after a user reports such a problem? Is someone able to accept the UAC? Does Firefox try to update again later? Does it succeed?
Yes, updates proceed if elevated. If elevation is cancelled, the next time Firefox starts it will attempt to update and UAC will prompt.

Flags: needinfo?(bugzilla)

Sorry the correct quoting syntax evaded me with the above. It's been a long day.

(In reply to oneofthedamons from comment #14)

This may not really be sufficient to collect the logs that I need. The Browser Console log that I described collecting really needs to be collected when downloading the problem-causing update in order to be useful. Collecting the log after the fact will really only tell us about problems checking for new updates, not problems installing the previous update.

Is there a way to set app.update.log to true through policies? We can push this out easily through Group Policy while we troubleshoot.

There is an enterprise policy that lets you set certain preferences, including update related ones. I suspect that this wouldn't actually help directly, since the contents of the Browser Console isn't saved anywhere when Firefox exits, and it sounds like you won't know about the issue until after Firefox has shut down to apply the update.

There is app.update.log.file, which I initially thought might actually help out here. It normally turns itself off at the beginning of each session to prevent people from leaving it on and constantly logging a ton of junk to the disk unnecessarily. But if you use the enterprise policy to lock the pref, it doesn't actually get turned off. Unfortunately, the bigger issue here seems to be that the functionality that the pref controls is badly broken. I tried it out so that I could see if it would help with our situation here, and it is doing some very, very strange and unhelpful things. I filed Bug 1830368 to get this fixed, but that doesn't really help us at the moment.

One additional question: Has this problem only happened updating from 112.0 to 112.0.1? Do you know if it has ever happened updating to or from any other versions?

(In reply to oneofthedamons from comment #15)

Sorry the correct quoting syntax evaded me with the above. It's been a long day.

Gah, yeah. I've been tripped up by that too. You have to leave a blank line between the quoted text and the response or that happens for some reason.

Flags: needinfo?(bugzilla)

Has this problem only happened updating from 112.0 to 112.0.1? Do you know if it has ever happened updating to or from any other versions?

We definitely had an instance of 111.0(.1?) -> 112.0, another for 112.0 (but possibly older) -> 112.0.1 and just yesterday one more for 112.0(.1?) -> 112.0.2, but again could have been older. As I noted previously, given these users restart so seldom it's possible I they've skipped one or more updates.

One question: will setting the BackgroundAppUpdate enterprise policy make any difference? We currently do not define it; I think because at the time we deployed Firefox across our enterprise I wasn't certain what its effect would be.

You have to leave a blank line between the quoted text and the response or that happens for some reason.

I really need to use that Preview tab so it becomes a reflex :)

Flags: needinfo?(bugzilla)

(In reply to oneofthedamons from comment #17)

One question: will setting the BackgroundAppUpdate enterprise policy make any difference? We currently do not define it; I think because at the time we deployed Firefox across our enterprise I wasn't certain what its effect would be.

If Background Update is enabled, Firefox will download and install updates when it isn't running. It follows pretty much exactly the same process as Firefox does when it is running so I wouldn't really expect there to be a difference.

Background Update is enabled by default - sort of. There are a number of factors that it currently cannot handle and cause it to turn itself off such as having langpacks installed, having automatic update disabled, or using a proxy other than the system proxy to connect to the internet.

@nalexander - I seem to remember that when this bug was first filed, you thought that there might be a Background Update angle that should be investigated, but I can't seem to remember what it was. Off the top of my head, I'm not thinking of a way that this could be affected by Background Update. Is there anything related to that that you think would be worth looking into? Thanks!

Flags: needinfo?(nalexander)

Had another occurrence of this yesterday, on a host prompting to update from 114.0.1->114.0.2. On this one I was signed-in as a member of the Administrators group when I started Firefox, but the update would have been downloaded previously I suspect so I'm not sure how useful any of this is. Information requested earlier to follow.

Attached file last-update.log (obsolete) —
Attached file backup-update.log (obsolete) —
Attached file maintenanceservice.log (obsolete) —

This time signed-in as a member of the local Administrators group

It seems like last-update.log is cut off. Off the top of my head, I'm not sure if that's expected if a UAC prompt is cancelled. Regardless though, whether or not to use the Service is generally decided by Firefox rather than by the updater, so most likely there wouldn't be anything valuable in that log regardless.

Since this is a bug that would potentially benefit from being able to use app.update.log.file for debugging, I just thought I'd mention that that functionality has been fixed (Bug 1830368) as of Firefox 118. Version 118 isn't scheduled to hit the Release channel (which I believe you are on) until September 26th. But if this bug remains stalled until then, we may be able to use this functionality to figure out what is going on at that time.

Given the intermittent nature of this bug, I'm afraid that I still don't have any great ideas for how to debug further without this sort of feature. Sorry that I don't have better news.

Just to confirm: the preference that should be set using Enterprise Policy is app.update.log.file to true?

Is there any downside to pushing this preference out before the fix to Bug 1830368 hits Release in September (other than it producing non-useful logs)?

We haven't had any further reports since the most recent cited above; which could mean the issue is no longer manifesting, or could mean it just is not being reported to our Service Desk.

(In reply to oneofthedamons from comment #27)

Just to confirm: the preference that should be set using Enterprise Policy is app.update.log.file to true?

Is there any downside to pushing this preference out before the fix to Bug 1830368 hits Release in September (other than it producing non-useful logs)?

I wouldn't do this. The behavior prior to the bugs being fixed was a little unpredictable. I think it's unlikely but possible that it could create one or two new files on every Firefox launch that the pref is activated and might not ever delete them. So let's try to avoid that.

We haven't had any further reports since the most recent cited above; which could mean the issue is no longer manifesting, or could mean it just is not being reported to our Service Desk.

Curious. Maybe it was some sort of transient issue? Let's see if this is still the case once Release 118 is available and your users have had a chance to update.

As to how you would activate it, I normally wouldn't recommend that it be activated in the way that I'm about to explain. Usually this feature turns itself off on every startup to prevent users from leaving it on and causing Firefox to constantly write a bunch of unnecessary data to the hard drive. But I'm not sure how to troubleshoot this problem further in any other way. So you would want to set app.update.log.file to locked, which should prevent the feature from turning itself off. However, the other thing that we do on every startup is move the file (update_messages.log) to a backup location (update_messages_old.log) and start a new log. So promptly when a user experiences a problem, you should collect both the log and the backup log from the user's profile directory, which will be in C:\Users\<username>\AppData\Roaming\Mozilla\Firefox\Profiles. If you launch Firefox again, you can easily get to the right profile (if there are multiple) by navigating to about:profiles, finding the entry marked "This is the profile in use and it cannot be deleted.", and clicking the "Open Folder" button in the "Root Directory" row. Of course, launching Firefox to do this will cause the log backup that I mentioned, but if the problem just happened, that should be fine; it will just mean that the relevant information will be in update_messages_old.log.

A couple of further observations:

  1. We've had another occurrence of this behaviour, so it seems it remains an issue.
  2. As I'm running the beta train I've set app.update.log.file to true to test, and I'm noting that it's not sticking between reboots. I can't see any other Enterprises Policies that might be enforcing this, aside from app.update.log also being set to true. Any ideas what might be causing this?

(In reply to oneofthedamons from comment #29)

A couple of further observations:

  1. We've had another occurrence of this behaviour, so it seems it remains an issue.
  2. As I'm running the beta train I've set app.update.log.file to true to test, and I'm noting that it's not sticking between reboots. I can't see any other Enterprises Policies that might be enforcing this, aside from app.update.log also being set to true. Any ideas what might be causing this?

That option turns itself off when Firefox re-launches. We aren't really keen on it getting turned on and then hitting the disk for every log message forever. But you should be able to force it to remain on by using an enterprise policy to lock it to true.

Adding the qa-not-actionable tag for now because the issue occurs intermittently for a small number of users and it can't be reproduced easily.

QA Whiteboard: [qa-not-actionable]

I finally have been able to reproduce this, on my own workstation which is a huge help!

Attached are the maintenanceservice.log and the 2x update_messages.log — the date-time stamps of the later are shown below as that doesn't seem to be included in the logging:

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----        23/01/2023  12:13 PM             71 update_messages-1.log
-a----        18/10/2023   3:17 PM           2451 update_messages-2.log
-a----        18/10/2023   3:28 PM           2451 update_messages-3.log
-a----        18/10/2023   3:36 PM           2451 update_messages-4.log
-a----         6/11/2023  10:44 AM           4816 update_messages-5.log
-a----        14/12/2023   2:26 PM           6209 update_messages-6.log
-a----        29/01/2024   8:25 AM           4816 update_messages-7.log
-a----        28/02/2024   9:35 AM           7442 update_messages-8.log
-a----        13/03/2024  10:08 AM           5081 update_messages-9.log
-a----        13/03/2024  10:03 AM           9963 update_messages_old.log

The failure mode is as follows:

  1. Restarted to apply this month's Patch Tuesday Windows Updates (may or may not be relevant)
  2. On starting Firefox the updater asked to elevate
  3. I dismissed the elevation request without elevating
  4. Firefox started and then showed the manual update UI (also attached) to which I have not yet responded

One other observation: I note in the update_messages.log reference to a proxy. We do have an application-level gateway in place which is not a full proxy but does do some MITM on traffic. Happy to provide further details of this if it's relevant.

Attached file maintenanceservice.log —
Attachment #9340607 - Attachment is obsolete: true
Attachment #9340606 - Attachment is obsolete: true
Attached file update_messages-9.log —
Attachment #9340605 - Attachment is obsolete: true
Attachment #9328964 - Attachment is obsolete: true

Further comment, selecting Update and restart prompts for elevation again, I have taken a photo of that this time and attached.

Attachment #9329171 - Attachment is obsolete: true

In case it's relevant, attaching the update logging from the folder specified in the UAC prompt. These were located in C:\ProgramData\Mozilla-1de4eec8-1241-4177-a864-e594e8d1fb38\updates\308046B0AF4A39CB\updates

Attached file last-update.log —

From C:\ProgramData\Mozilla-1de4eec8-1241-4177-a864-e594e8d1fb38\updates\308046B0AF4A39CB\updates

Attached file backup-update.log —

From C:\ProgramData\Mozilla-1de4eec8-1241-4177-a864-e594e8d1fb38\updates\308046B0AF4A39CB\updates

Ugh, well this is a bit annoying. It seems that when update elevation is cancelled, we simply retry the update without even attempting to use the Maintenance Service (here). So once this problem happens the first time, it sort of perpetuates itself. I've filed Bug 1885195 to look into changing this behavior because, at first glance, that doesn't seem like the right thing to do.

So the most recent update_messages logs are telling the updater not to use the Maintenance Service for that reason.

Could you attach the older update_messages? I'm hoping that one of the older ones will have the original cause of using a UAC when we shouldn't be. Maybe consider zipping them or something just to keep the number of attachments on this bug from getting ridiculous.

Flags: needinfo?(bugzilla)
Attached file update_messages*.log —

As requested in comment #42

Flags: needinfo?(bugzilla)

Unfortunately, none of these seem to contain a log of the original problem. As you can sort of see from the dates of the logs from Comment 32, the ordering is maybe not what one would expect. And, to be honest, I was really only expecting to see update_messages_old.log and maybe update_messages.log. But I believe that the module that is being used to log here has sort of a backup/renaming mechanism that I wasn't fully aware of that appears to function by writing <filename>-1 first and then when it gets to <filename>-9, it just overwrites that one every time, I guess to avoid making an unlimited number of these copies. Then the update logging code separately makes the _old backup that gets overwritten each time.

Which is a long way of saying that it looks like we've got a 8 logs from before the problem started on this machine and then 2 logs from after the problem started being self-perpetuated as I described in comment 42.

I'll look into addressing Bug 1885195, which should hopefully decrease the incidence of this problem.
@oneofthedamons - If you are able to get a log of this right when it starts happening or, alternately, if you can get any log of this happening from a Firefox version after Bug 1885195 has been addressed, I can try to find out why it was failing in the first place.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: