Closed Bug 1599590 Opened 5 years ago Closed 3 years ago

Firefox ESR 68.2 is saying updates are available even though they aren't (and updates have been disabled)

Categories

(Toolkit :: Application Update, defect, P3)

Product:
Toolkit
Component:
Application Update
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
85 Branch
Tracking Flags:
Tracking Status
firefox-esr78 85+ fixed
firefox84 --- wontfix
firefox85 --- fixed

People

(Reporter: mkaply, Assigned: bytesized)

References

(Blocks 1 open bug)

Details

Attachments

(2 files)

Browser-Console.txt
3.43 KB, text/plain
Details
Bug 1599590 - Prevent entering the update system if update is disabled by policy r=mhowell!
47 bytes, text/x-phabricator-request
RyanVM
: approval-mozilla-esr78+
Details | Review
Attached file Browser-Console.txt —

In:

https://github.com/mozilla/policy-templates/issues/496

Customer is reporting strangeness around updates.

They have German 68.2 ESR installed, but they are still getting told an update is available but it won't install.

I've verified that they are on the correct channel. Log from the update is attached.

They don't appear to have the maintenance service installed.

The priority flag is not set for this bug.
:robert.strong.bugs, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(robert.strong.bugs)

Could you get the permissions of the updates directory using the following command and add it to this bug? Thanks!

cacls C:\ProgramData\Mozilla /t

Flags: needinfo?(robert.strong.bugs) → needinfo?(mozilla)
Priority: -- → P3

NT AUTHORITY\SYSTEM:(OI)(CI)(ID)F
BUILTIN\Administrators:(OI)(CI)(ID)F
BUILTIN\Users:(OI)(CI)(ID)R

Clearing priority so this will get attention

Priority: P3 → --
Blocks: 1609964

Clearing needinfo but please do reply with the info and someone else will continue with the diagnosis.

Flags: needinfo?(mozilla)

(In reply to 11chrisi from comment #3)

NT AUTHORITY\SYSTEM:(OI)(CI)(ID)F
BUILTIN\Administrators:(OI)(CI)(ID)F
BUILTIN\Users:(OI)(CI)(ID)R

Thanks for checking those permissions. They are definitely not right; BUILTIN\Users should have been given full control (F), and nothing there should show up as inherited ((ID)). We have code that attempts to fix all this, and the log shows that it did run and doesn't show that it failed, but evidently it did fail. But if the affected machines don't have the maintenance service installed, that might be what we expect to happen; I don't think changing any of those without having admin privileges ought to work.

Kirk, do you have ideas about what to do with this? I wonder if deleting the existing C:\ProgramData\Mozilla directory would help?

Flags: needinfo?(ksteuber)

I suspect that deleting the C:\ProgramData\Mozilla would prevent the prompts from being shown, but before you do that, I'd really like to nail down what the underlying problem is.

I think that I have figured out how to reliably reproduce the bug, but my reproduction steps require that the pref app.update.log already be set to true. If it is set to false, I can no longer reproduce the problem. This is sort of a debugging pref that I wouldn't expect most people to have turned on normally. But the bug I have identified seems like a perfect fit given the problem described and the log posted.

Do you know if you manually enabled app.update.log before you started seeing this problem? Or, alternately, could you disable that pref and see if the problem goes away? Thanks!

Flags: needinfo?(ksteuber) → needinfo?(11chrisi)

(In reply to Kirk Steuber (he/him) [:bytesized] from comment #7)

I suspect that deleting the C:\ProgramData\Mozilla would prevent the prompts from being shown, but before you do that, I'd really like to nail down what the underlying problem is.

I think that I have figured out how to reliably reproduce the bug, but my reproduction steps require that the pref app.update.log already be set to true. If it is set to false, I can no longer reproduce the problem. This is sort of a debugging pref that I wouldn't expect most people to have turned on normally. But the bug I have identified seems like a perfect fit given the problem described and the log posted.

Do you know if you manually enabled app.update.log before you started seeing this problem? Or, alternately, could you disable that pref and see if the problem goes away? Thanks!

I didn't enabled the app.update.log. It is default set to false.
As a workaround I set app.update.doorhanger to false to disable the banner message.

We use Microsoft Terminal Server 2012 R2. Maybe next week we will update our environment to Firefox 68.4.2 ESR.I will provide an update, if the issue still exists.

Flags: needinfo?(11chrisi)

Well now I am confused. The attached Browser-Console.txt was clearly made with app.update.log set to true. That pref is what allows that output to be created in the first place.

@mkaply - Did you recreate that output? Where did it come from?

Flags: needinfo?(mozilla)

Or perhaps app.update.log.file was used to make it? I believe that forces logging, even if app.update.log is false.

I definitely asked them to set app.update.log in order to get this log:

https://github.com/mozilla/policy-templates/issues/496#issuecomment-558734405

So app.update.log should be set.

Flags: needinfo?(mozilla)

(In reply to Kirk Steuber (he/him) [:bytesized] from comment #10)

Or perhaps app.update.log.file was used to make it? I believe that forces logging, even if app.update.log is false.

I set app.update.log to true only for the browser console log attached. Before it was set to false. After receiving the logs, I reset this setting to false.

Depends on: 1613245

Well, it looks to me like the underlying issue here is going to be very hard to get at because it seems to be "covered up" by Bug 1613245. I should have a fix for that up soon.

I have some thoughts here. I could walk the reporter through running some more recent, or possibly custom builds and see if we can shed some more light on the situation. But if they are happy with their workaround, that is perfectly understandable.

The workaround, however, will stop working at ESR 78.0, because the pref they are using (app.update.doorhanger) was removed in Bug 1515484. But that ESR should also contain my fix for Bug 1613245. So another potential option is just to use the workaround, wait for the next ESR, and if the workaround stops working and the problem comes back, we can get another log and see if fixing Bug 1613245 allows me to get a bit more insight into what's going on.

For your information: The update message still comes up with Firefox 68.4.2 ESR.

I found out, that our server installation routine reset the permissions of the C:\ProgramData\Mozilla to
NT AUTHORITY\SYSTEM:(OI)(CI)(ID)F
BUILTIN\Administrators:(OI)(CI)(ID)F
BUILTIN\Users:(OI)(CI)(ID)R

I think this is done because of security purposes - a normal user of our terminal server shouldn't write into that area, because it is shared with many others.

So like Molly has described, that are not the correct permissions.

If I install Firefox manually without our installation routine, the permissions look like this:
NT AUTHORITY\SYSTEM:(OI)(CI)F
BUILTIN\Administrators:(OI)(CI)F
BUILTIN\Users:(OI)(CI)F
BUILTIN\Users:(OI)(CI)R

With these permissions the update message don't appaer.

Do you know if anything has changed with this bahaviour? We reset the permissions of this folder since many years.
So what can we do now? Should we set permissions back to the original? Then the first user that opens firefox is set to the owner of some files in C:\ProgramData\Mozilla. Or do you have any other idea?

(In reply to 11chrisi from comment #14)

Do you know if anything has changed with this bahaviour? We reset the permissions of this folder since many years.
So what can we do now? Should we set permissions back to the original? Then the first user that opens firefox is set to the owner of some files in C:\ProgramData\Mozilla. Or do you have any other idea?

We used this directory many versions ago. Then we stopped using it. Now we are using it once again. Unfortunately, when we used it in older versions, we just used the default permissions, which now causes problems.

Now, when we use the directory, we specify that, if it doesn't exist, it be created with permissions that allow all users to access it. When I made this change, I was made aware of our old usage of the directory, which is why I wrote some code that tries very hard to fix the permissions on that directory, if they are wrong. If the user cannot do this directly, Firefox attempts to call into the Mozilla Maintenance Service, which can do this with elevated permissions.

However, there are some situations, particularly when the Mozilla Maintenance Service is unavailable, where the permissions just cannot be fixed. This appears to be the situation you are in currently.

It is very likely that your issue would go away if you fixed the permissions on the directory (or even just delete it and let Firefox recreate it with the correct permissions - though this may cause you to lose download progress of pending updates and the value of your "Update Automatically"/"Ask me to update" setting).

Ideally, however, the situation you are in should not happen (i.e. the doorhanger should never appear when updates are disabled by policy). My ideal solution would be to find the underlying cause of the problem. I enumerated the options that I think are available for doing this in my previous comment, Comment 13.

Priority: -- → P3

The workaround with app.update.doorhanger set to false doesn‘t work anymore.

Are there any updates related to this bug?

Kirk, you were looking into this one before. Is there more you can do now under the newer ESR (going by comment 13)?

Flags: needinfo?(ksteuber)

(In reply to 11chrisi from comment #16)

The workaround with app.update.doorhanger set to false doesn‘t work anymore.

Are there any updates related to this bug?

Yes! I believe you should now have a copy of Firefox in which Bug 1613245 has been fixed. Could you please provide us with a new update log from the browser console? I'd like to see what is going on that causes this problem.

This is how you can obtain such a log:

First, turn on logging by navigating the browser to about:config and setting app.update.log to true. Then, after you experience this problem, open the browser console either via the hotkey (Control+Shift+J) or through the menu (Menu->Web Developer->Browser Console).

We are interested in the AUS:SVC lines, so if you can attach those messages to this bug, I believe I should be able to figure out what's going on.

Flags: needinfo?(ksteuber) → needinfo?(11chrisi)

AUS:SVC Creating UpdateService
AUS:SVC Current UpdateService status: disabledByPolicy
AUS:SVC getCanApplyUpdates - unable to apply updates without write access to the update directory. Exception: [Exception... "Component returned failure code: 0x80520015 (NS_ERROR_FILE_ACCESS_DENIED) [nsIFile.create]" nsresult: "0x80520015 (NS_ERROR_FILE_ACCESS_DENIED)" location: "JS frame :: resource://gre/modules/FileUtils.jsm :: FileUtils_getDir :: line 76" data: no]
AUS:SVC Attempting to fix update directory permissions
AUS:SVC isServiceInstalled - returning false
AUS:SVC shouldUseService - returning false
AUS:SVC isServiceInstalled - returning false
AUS:SVC shouldUseService - returning false
AUS:SVC gCanStageUpdatesSession - testing write access C:\Program Files\Mozilla Firefox\update.test
AUS:SVC gCanStageUpdatesSession - unable to stage updates. Exception: [Exception... "Component returned failure code: 0x80520015 (NS_ERROR_FILE_ACCESS_DENIED) [nsIFile.create]" nsresult: "0x80520015 (NS_ERROR_FILE_ACCESS_DENIED)" location: "JS frame :: resource://gre/modules/UpdateService.jsm :: testWriteAccess :: line 291" data: no]
AUS:SVC getCanUseBits - BITS can be used to download updates
AUS:SVC isServiceInstalled - returning false
UTM:SVC TimerManager:notify - notified @mozilla.org/updates/update-service;1
AUS:SVC Checker:getUpdateURL - update URL: https://aus5.mozilla.org/update/6/Firefox/78.4.0/20201013163257/WINNT_x86_64-msvc-x64/de/esr/Windows_NT%206.3.0.0.9600%20(x64)/ISET:SSE4_2,MEM:130942/default/default/update.xml
UpdateUtils.getAppUpdateAutoEnabled - Unable to read app update configuration file. Exception: [Exception... "Component returned failure code: 0x80520015 (NS_ERROR_FILE_ACCESS_DENIED) [nsIFile.create]" nsresult: "0x80520015 (NS_ERROR_FILE_ACCESS_DENIED)" location: "JS frame :: resource://gre/modules/FileUtils.jsm :: FileUtils_getDir :: line 76" data: no]
AUS:SVC Checker: checkForUpdates, force: false
AUS:SVC Creating UpdateService
AUS:SVC Current UpdateService status: disabledByPolicy
AUS:SVC Checker: checkForUpdates, disabled by policy

Flags: needinfo?(11chrisi)

Hmm. That output is a bit unexpected to me. I'll do some digging and see if I can figure out what's going on.

Ok, I think I might know what the problem is but I need you to check one thing so I can be sure.

At first, I couldn't figure out how anything update-related was starting in the first place when update is disabled by policy. But there is actually one way that it seems you can still get in: Post Update Processing. Post Update Processing is started based on the update status when Firefox launches. I believe it was originally assumed that if you have update turned off, there will be no update status. But if there is an update status hanging around that we don't have the permissions to delete, we could persistently hit exactly this problem.

Here's what I'd like you to check:

  • Navigate to about:support
  • Find the row that reads "Update Folder" (it's the 5th row down on my machine).
  • Click the "Open Folder" button in that row. It should open a directory that looks like C:\ProgramData\Mozilla\updates\<hash>
  • I want to know if C:\ProgramData\Mozilla\updates\<hash>\updates\0\update.status exists, and what it contains.
Assignee: nobody → ksteuber
Flags: needinfo?(11chrisi)

The Folder C:\ProgramData\Mozilla is empty.

Flags: needinfo?(11chrisi)

Ok, I think I've determined a very likely cause for why this is happening. I don't really have any steps for you to verify, but I can reproduce the issue locally, and it matches the reproduction steps and symptoms that you have been describing very closely. I should have a fix ready shortly.

Pushed by ksteuber@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6fb41e11811f
Prevent entering the update system if update is disabled by policy r=mhowell

https://hg.mozilla.org/mozilla-central/rev/6fb41e11811f

Status: NEW → RESOLVED
Closed: 3 years ago
status-firefox85: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 85 Branch

Comment on attachment 9192470 [details]
Bug 1599590 - Prevent entering the update system if update is disabled by policy r=mhowell!

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration: This bug is most likely to affect centrally managed Windows machines, which is one of the main targets for ESR. I believe it can be triggered by seemingly benign actions.
  • User impact if declined: Some Firefox deployments may tell the user that application update is failing and they need to reinstall manually despite update being disabled by Enterprise Policy. Depending on exactly how they trigger the bug, it could affect every managed computer.
  • Fix Landed on Version: 85
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): This is a low risk change. It simply returns early from 2 update functions if Update is disabled to prevent any other update code (potentially with side effects) from executing.
  • String or UUID changes made by this patch: None
Attachment #9192470 - Flags: approval-mozilla-esr78?

Comment on attachment 9192470 [details]
Bug 1599590 - Prevent entering the update system if update is disabled by policy r=mhowell!

Approved for 78.7esr.

Attachment #9192470 - Flags: approval-mozilla-esr78? → approval-mozilla-esr78+
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: