1761880 - update

Status: RESOLVED INCOMPLETE

(Toolkit :: Application Update, defect)

Description

[kenji]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0

Steps to reproduce:

Let nightly update automatically using the background service

Actual results:

Update fails, prompting to download the installer for the newer version

Expected results:

As it happens on another pc, it should seamlessly update at next application initialization.

Comment 1

[kenji]

SPECS PC with WORKING updates:

laptop acer predator g3-572

windows 10 2004

intel 7700hq/hd630 + gtx 1060 + 32GB RAM

SPECS PC with BROKEN updates:

desktop DIY

windows 10 1809 LTSC

ryzen 2600 + R9380 + 16 GB RAM

I'm using both pcs in a daily basis. In both, I have an admin account just for installing things, and a non-elevated 100% local account, which is the one I use normally.

The thing is, I use Firefox, and have the same setting on both pcs, for having the updates managed by a background service, and auto-updates enabled. On my laptop, sometimes I see the prompt for updating firefox after restarting the application, and it works flawlessly, it doesn't even asks for admin password.

On my desktop, twice or thrice, updates work that way, but after sometimes, it may ask for admin password after restarting firefox. Everything would be ok if it was only that, but sometimes it just prompts me telling it couldn't download nor apply the update. I can log at admin, and even after that it says it cannot update. Then, my only option is downloading firefox installer, uninstalling it and installing the new build.

Anyone can point me a direction for fixing this, so updates can just work on both machines?

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Toolkit::Application Update' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 3

[kenji]

In about:support, update logs have a message like "incremental updates failed to apply", clicking on details leads to mozilla website.

Comment 4

[Robin Steuber (she/her) [:bytesized]]

Could you attach your update logs by following these steps?

1. Navigate to about:support
2. Find the "Update Folder" entry and click "Open Folder".
3. Open the updates directory.
4. Inside, you should find files named last-update.log and backup-update.log. Attach both of these files to this bug.

Comment 5

[kenji]

Attachment: backup-update.log

Comment 6

[kenji]

Attachment: last-update.log

Here are both files.

Comment 7

[Robin Steuber (she/her) [:bytesized]]

Huh. These logs are very odd and I'm not quite sure what to do with them.

The more recent one (last-update.log) looks like either it was still in the process of attempting to stage an update when the log was collected, or else I guess maybe the updater crashed.

The older log (backup-update.log) appears to have failed here:

ensure_copy_recursive: path doesn't exist: C:\Program Files\Firefox Nightly/firefox.exe.moz-backup, rv: -1, err: 2

This is an extremely odd message. .moz-backup files are only produced by the Firefox updater. The fact that ensure_copy_recursive tried to copy that file but then couldn't find it suggests that it was probably removed while ensure_copy_recursive was running. This implies that 2 Firefox updaters were running at the same time, both trying to do slightly different things (one trying to stage an update, one trying to apply an un-staged update).

At first I thought that maybe this could happen if Firefox was started twice in rapid succession. We have some checks in place to ensure that Firefox isn't started while an update is occurring, but it might be possible for it to happen anyways if the timing was exactly right. Except that there doesn't seem to be any reason why we would immediately start staging in this situation.

Maybe it would help if you could tell me a bit more about your configuration.

1. It looks like you are using Firefox Nightly, is that correct?
2. Are you using multiple Firefox profiles on this computer?
3. When you open Firefox, do you often have an extra Firefox window open slightly after? Perhaps this could indicate that Firefox opened twice and one instance updated while the other instance attempted to run normally?
4. How do you normally launch Firefox?

Maybe we could also try to get a "cleaner" update log by doing something like this:

1. Open Firefox.
2. Let Firefox just sit for maybe 5 minutes, just in case the updater is busy doing something weird.
3. Open about:preferences and find the "Update" section.
4. It should automatically check for updates. If you have the "Check for updates but let you choose to install them" option selected, accept the update prompt to let it download the update.
5. Wait for the update to fail.
6. Collect the update logs using the steps in Comment 4.

I'm hoping that this will give us something a bit more useful so that we can figure out what's going on.

Comment 8

[kenji]

Attachment: updates.zip

Comment 9

[kenji]

No problem:

1 - Yup, using nightly here;
2 - Profiles on nightly were left as-is, only the default profile running. I have 3 user accounts on OS, but only the one I use in a daily basis is synced, with the other machine which doesn't has this issue;
3 - No, there isn't any other windows popping or flashing;
4 - There's a shortcut on the taskbar, just clicking it.

I followed your instructions, there are 3 folders with the files at these steps:
1 - I started firefox but refused the UAC prompt from the firefox update service. Firefox shows a prompt saying that there is an available update for download, or dismiss.
2 - I closed firefox, then started it again, now accepting the prompt from the update service. It doesn't show any prompting regarding updates after that, but entering about:preferences, there is a button "restart firefox for update".
3 - After clicking that button, firefox closed and restarted. It is set for returning to the last open tab, and there is the button "restart for update" again, and now it is becames a loop if I click it again.

Comment 10

[Robin Steuber (she/her) [:bytesized]]

Well. This is a tricky one. I've got some other things that I need to work on, but I think that Ben should have some time to look into this further next week.

---

So, the first backup-update.log had this bit that I called out before:

ensure_copy_recursive: path doesn't exist: C:\Program Files\Firefox Nightly/firefox.exe.moz-backup, rv: -1, err: 2

There really isn't any reason why we would be calling that function with that path as an argument if the file didn't exist. But the error makes it pretty clear that the file doesn't exist (errno 2 = ENOENT). So presumably that file existed and was then removed, I guess.

A later backup log (update.zip/2-accepting UAC prompt/last-update.log) shows something very similar happening in a different way

rename_file: failed to rename file - src: C:\Program Files\Firefox Nightly/firefox.exe, dst:C:\Program Files\Firefox Nightly/firefox.exe.moz-backup, err: 17

errno 17 = EEXIST, which seems to mean that firefox.exe.moz-backup already exists. But then later

backup_restore: backup file doesn't exist: firefox.exe.moz-backup

That file now doesn't exist. I guess? Though, as I write this, it occurs to me that it doesn't seem like we should actually be attempting to restore a backup that we never successfully made. Odd.

I can't really think of any plausible reason for .moz-backup files to be move around except if another updater was trying to update Firefox at the same time that this one was.

That would be a pretty weird thing to happen for a couple of reasons. When we start updating, we lock the Firefox binary. This should prevent another copy of Firefox from launching, which should prevent the update process from being started a second time. We also set the update status to "applying" which, even if Firefox was launched, should cause it to just run rather than starting the update process. There is technically a race condition where both instances could do these things at pretty much the exact same time, avoiding the lock. But threading the needle like that not just once but repeatedly seems so unlikely.

So I'm not really sure where to go with this next. I'm hoping that Ben has some clever ideas when he looks at this.

Comment 11

[kenji]

Is there anything on the profile files that could prevent that update? Something that persists even after uninstalling nightly, along with customizations and saved data? Because I already uninstalled it several times, and the error persists. I have alpenglow and a custom background for blank tabs, nothing else I remember.

Comment 12

[Robin Steuber (she/her) [:bytesized]]

We clear out the update related files on uninstall. The profiles typically remain, but I don't know of anything in the profiles that would be capable of causing a problem like this.

Comment 13

[kenji]

Attachment: after manual reinstall.zip

I uninstalling it completely yesterday. Also tried to make anything with this OS. Cleaning disks, sfc, chkdsk, updating...

So I copied the files on the zipped folder at two occasions: after the fresh install, and then after first restarting firefox when being prompted to update. After a fresh install, the update as it should, seamless, without UAC prompt, and working, works two or three times. If it works more than that I will report.

Comment 14

[kenji]

Attachment: follow up and issue loop.zip

Last update for now. At this restart UAC didn't appear, but nightly prompts a message that there is an update available for download, since the update failed.

Comment 15

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:bhearsum, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 16

[[e-mail] (:bhearsum)]

Thank you so much for the information you've provided so far.

(In reply to kenji from comment #9)

2 - Profiles on nightly were left as-is, only the default profile running. I have 3 user accounts on OS, but only the one I use in a daily basis is synced, with the other machine which doesn't has this issue;

It certainly looks like something external to the usual update process is interfering in some way here -- either another copy of the updater like Kirk mentioned, or something else entirely.

Can you confirm exactly what you're syncing between the two machines (the profile is both in local and roaming appdata, but I'm also wondering if you're syncing c:\Program Files\Mozilla Firefox as well), and when the sync is happening? Is it possible the syncing is happening while Firefox is running on one or the other machine?

I'm also wondering if you have any Antivirus software installed (other than Windows Defender) -- it's not uncommon for it to cause issues with our updater.

Comment 17

[kenji]

Can you confirm exactly what you're syncing between the two machines (the profile is both in local and roaming appdata, but I'm also wondering if you're syncing c:\Program Files\Mozilla Firefox as well), and when the sync is happening? Is it possible the syncing is happening while Firefox is running on one or the other machine?

I'm using the sync with firefox account, all seven boxes are marked, nothing else.

I didn't told about AV, I'm using malwarebytes, the same on both machines. On the pc with the problem, there was one box ticked regarding "specialized algorithms", which wasn't marked on the other machine, so I disabled it on both.

But yesterday, even before reading that, I noticed the update is working in intermittent fashion. When there is a prompt asking for restarting nightly for updating, I'm closing the program and leaving it for some minutes before starting it again.

I'm uploading three folders, one with files when nighlty first prompt for update, then after it failed, and after starting it again a third time, update was successful. At least it is now updating with a 50/50 chance... and triggering UAC.

Comment 18

[kenji]

Attachment: 3 folders.zip

Comment 19

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:bhearsum, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 20

[[e-mail] (:bhearsum)]

Thank you for continuing to provide such good information!

(In reply to kenji from comment #17)

Can you confirm exactly what you're syncing between the two machines (the profile is both in local and roaming appdata, but I'm also wondering if you're syncing c:\Program Files\Mozilla Firefox as well), and when the sync is happening? Is it possible the syncing is happening while Firefox is running on one or the other machine?

I'm using the sync with firefox account, all seven boxes are marked, nothing else.

Oh I see -- I thought you were talking about syncing profile files directly with some external software or such. Firefox Sync definitely should not be causing any issues like this.

I didn't told about AV, I'm using malwarebytes, the same on both machines. On the pc with the problem, there was one box ticked regarding "specialized algorithms", which wasn't marked on the other machine, so I disabled it on both.

If you're comfortable doing so, I'd also be curious if disabling Malwarebytes entirely fixes the issue.

But yesterday, even before reading that, I noticed the update is working in intermittent fashion. When there is a prompt asking for restarting nightly for updating, I'm closing the program and leaving it for some minutes before starting it again.

I'm uploading three folders, one with files when nighlty first prompt for update, then after it failed, and after starting it again a third time, update was successful. At least it is now updating with a 50/50 chance... and triggering UAC.

Thanks for these. Unfortunately, they don't show us why the update is getting interrupted or failing, just that it is :(.

Comment 21

[[e-mail] (:bhearsum)]

(In reply to [e-mail] (:bhearsum) from comment #20)

I didn't told about AV, I'm using malwarebytes, the same on both machines. On the pc with the problem, there was one box ticked regarding "specialized algorithms", which wasn't marked on the other machine, so I disabled it on both.

If you're comfortable doing so, I'd also be curious if disabling Malwarebytes entirely fixes the issue.

Mike, do you know if we have any known issues with Malwarebytes interfering with updates?

Comment 22

[Mike Kaply [:mkaply]]

I'm not aware of any issues with malwarebytes and they actively encourage folks to update Firefox using our update mechanism, so I can't think of anything they would do to interfere with it.

Comment 23

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:bhearsum, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 24

[[e-mail] (:bhearsum)]

I had a reread of this bug today and I agree with Kirk's earlier assessment that it seems like multiple update processes are running at the same time. One thing that might be able to help us confirm or deny that is to run Process Monitor to try to capture what's actually running when the failed update happens. It might be enough to just start Process Monitor before starting Firefox -- but it would probably be ideal just to constantly run it until you hit a failure. If you're able to hit it with Process Monitor hopefully the trace will yield something useful. I realize this is quite a hassle so please don't feel obligated - we're just running out of other angles to investigate this :(.

Comment 25

[Nick Rishel [:nrishel]]

Closing this issue for now as the needinfo request has gone unanswered for two weeks.

@kenji please reopen the report if you can get back to us with the requested information.