Open Bug 1480452 Opened 6 years ago Updated 10 days ago

Second instance of Firefox will apply update downloaded by still running first

Categories

(Toolkit :: Application Update, defect, P3)

Product:
Toolkit
Component:
Application Update
Platform:
Unspecified
Windows
Type:
defect

Tracking

()

People

(Reporter: agashlin, Unassigned)

References

(Depends on 1 open bug, Blocks 1 open bug)

Details

(Whiteboard: [iu_tracking]) 

Steps to reproduce:
1. Install an out-of-date nightly
2. Start an instance
2a. Let it download an update
3. Start another instance with -no-remote

The second instance will apply the update, which could be a problem. Seems to be (at least directly) because of bug 715746 allowing the updater to proceed without being able to get exclusive access to the callback.

Two ways I see around this, maybe a combination of both for general safety:
- Try to get the update mutex, if it fails don't launch the updater (maybe even don't look at update.status at all)
- The updater could retry a few times if it can't get the callback exclusive
Another scenario:
3. Switch to another user
4. Start Firefox
5. Switch back
6. Restart for update

---

I forgot about the existing retry loop [1] in my first suggestion above, but I don't see (yet) why it should continue in the face of the sharing violation.

[1] https://searchfox.org/mozilla-central/source/toolkit/mozapps/update/updater/updater.cpp#3481-3499
I think this happens when starting WebIDE or the Browser Toolbox from the Tools menu as well.
I've learned some more about this: We continue in spite of failing to lock the callback because firefox.exe can be kept open as a zombie process. We would rather do a potentially dirty update than never update. I don't think this is still the right compromise to make but I don't have a better idea just yet.
Priority: -- → P3

I'm also hitting this, almost daily. I use two profiles on Nightly on Linux. Updates are not handled by the package manager.

  1. Have pending updates and one Nightly instance open
  2. Nightly updates in the background
  3. Open an additional Nightly instance on a different profile using the startup flag.
  4. Firefox update dialog appears before the instance opens
  5. In the old instance, navigate to a different page
  6. "Sorry we need to restart (...)"

It should be possible to use the update sync manager multi-instance lock (from bug 1553982) to check if another instance is running before doing an update. Bug 1695797 implemented this specifically when running with background tasks, since the background update agent scheduled task would otherwise make this issue extremely common. This special case didn't need to deal with fallback/timeout and user notification in case the lock was stuck somehow, because the ability to update from a background task is considered supplemental to normal foreground updates.

If we decide not to apply an update on startup, we need to be able to allow the user to force this to go ahead anyway. The normal "restart to update" doorhanger message is changed to an appropriate warning message, that might be sufficient, but I don't think we show that message unless Firefox has just downloaded or staged the update. We may want to show that if we come up and see the update is still pending, maybe only if isOtherInstanceRunning(), similar to the check for an older update left pending [1].

The other half is knowing when to force the update on startup. I don't think we have any existing signal that this is an explicit restart to update. There is an old environment variable, MOZ_APP_RESTART [2], that gets set when we restart, including when we explicitly restart to apply an update, that might be a sufficiently strong signal that the user is intending to force the update. There are other ways to restart that we might not want to include, but they may be rare enough? If nothing else we can just add another env var or command line arg (assuming I didn't miss one that already exists).

[1] I had thought there was a general purpose handler for an existing update still pending, but I think it just falls through to handleFallbackToCompleteUpdate() from the bottom of _postUpdateProcessing here. I haven't verified that.

[2] I hadn't noticed MOZ_APP_RESTART before, some archaeological notes: It comes all the way from bug 628363, originally it was a timestamp so the process creation time could be measured even if the process was being reused across a restart. This was changed to just a flag in bug 852250. It was reworked somewhat and exposed in nsIAppStartup.wasRestarted in bug 911146, that was intended to replace to replace browser.sessionstore.resume_session_once in bug 914062, but that was abandoned.

Whiteboard: [iu_tracking]

A lot of the discussion around this issue has happened face-to-face, over internal channels, or has been fragmented across many different bugs. I am going to try to consolidate this into one post.

The Underlying Issue

In the beginning, Firefox ran as a single process. Over time, we have made large efforts to parallelize Firefox so that it can distribute work across many different processes. Now, when Firefox opens a website, it sometimes needs to start a new process to do so. When we need to start a process, we invoke the binary at the location that our process was launched from.

These processes talk to each other using a custom IPC (interprocess communication) protocol. Because we often make changes to this protocol, processes from different versions of Firefox cannot properly communicate with each other.

This means that if we end up in a situation where Firefox has been updated while it is running, it will eventually open a new process and find that it is speaking an incompatible version of IPC. At this point, Firefox has no way of launching new processes that it is compatible with, so it is unable to load more websites until it is restarted. Once it is restarted, of course, all the processes are the same version and the problem goes away.

Bug Activation Mechanisms

This problem can basically be caused in two known ways. If you are encountering this problem, and none of these situations seem to apply to you, please let us know!

  1. While Firefox is running, another instance of Firefox using the same installation updates. This is typically the result of running multiple profiles simultaneously. This is also theoretically possible on a system with multiple simultaneous users but, to my knowledge, no one has ever reported this problem to us.
  2. While Firefox is running, something else updates Firefox. Usually a Linux package manager. But occasionally we have seen Windows Antivirus programs (try to) do this. In theory, there are package managers for non-Linux systems that could probably cause this as well.

Note that Bug 1705217 tracks occurrences of this problem caused by package managers.

Known Possible Solutions

Don't have Firefox self-update if another instance is running.

This mechanism is described in some detail just above, in Comment 7. This could potentially solve the "updated by another instance of Firefox" bug, but not the "updated externally" bug.

This solution raises some concerns. It may enable malware to pretend to be a running instance of Firefox in order to attempt to prevent Firefox updates. Another concern is that users with many profiles may cycle through them and never give Firefox the opportunity to update. Given the increasing hostility of the internet and the fact that browsers are given a lot of trust run potentially malicious code while still protecting users, we would prefer a solution that does a better job of keeping Firefox up-to-date.

Fork Server

On some operating systems, like Linux, you can simply fork() a process to get a new one without needing to bother with the executable. We could keep a process around for the purpose of forking it into other processes when we need them.

The problem here is that we have many types of processes and they are all set up slightly differently. We would need the fork server to be an "undifferentiated" process that could be turned into any type of process needed. Then we would need to implement mechanisms to actually turn such a process into the necessary types.

This probably will not work on Windows, because Windows does not have a fork() command, as far as I know.

Start Processes with File Descriptors

It may be possible to start the processes that we want by launching Firefox from its file descriptor rather than using its path. fexecve, for example, does this. Unfortunately, the Firefox executable is not the only thing we would have to load this way. We would need fdlopen (which currently exists only on FreeBSD) to dynamically load libxul.

There may also be issues around code accessing the installation directory directly. It seems like we should be hitting those issues now, actually, so I'm a little confused about that. It's possible that this can cause a problem now and we've mostly just gotten lucky so far. But it is likely that all of these consumers would need to be fixed to open things with file descriptors in order for this solution to work properly.

Mitigation via IPC versioning

It is theoretically possible to limit this problem to when there are breaking changes in the IPC protocol. It might even be possible to make IPC backwards compatible.

The first issue with this is that it seems, from the conversations that I have had around this, that breaking changes to the IPC protocol happen all the time. Only very, very small security patches would likely be able to take advantage of this without a lot of effort being put into backwards compatibility. Testing would also become a problem, since we cannot realistically test the interaction between any two versions of processes that might interact.

Versioned Installations

We could potentially keep multiple versions of Firefox around at once. We would then have a launcher to pick the newest version when you go to start Firefox. Old versions would eventually be removed.

On Windows, this would be implemented with versioned directories (ex: "C:\Program Files\Mozilla Firefox" might contain directories for "89.0", "89.1", and "90").
On macOS, this would be implemented with versioned frameworks. This would allow us to install multiple without breaking the signature on the app.
I haven't spent all that much time looking into this on Linux yet. I suspect that versioned directories might not play nicely with package managers. We may want to have the Firefox package just be a launcher that depends on another package that has the browser. Then both packages could be installed simultaneously and updating might not be a problem? It could still be a problem though if the package manager uninstalls the old version. Obviously this needs further investigation if we want to go this route.

This would have the advantage of making it potentially possible to handle a new version crashing on startup by falling back to an older version. However, this also ends up being a disadvantage because it is effectively impossible to tell if the reason we crashed is our fault, or the fault of malware that wants us to downgrade. Also, when falling back, profile downgrade prevention may kick in, causing the user to lose access to their profile until they can be successfully updated.

Because this mechanism requires a launcher, it is likely that implementing it without regressing startup performance would be tough. Windows already sort of has a launcher process, which might make this slightly easier. But changing the launcher will likely require careful optimization.

This implementation has some of the biggest advantages, but it would be an extremely large effort. The launcher process would take time to write, it would be very platform-specific, and will probably be difficult or impossible to update. It would also need to handle forwarding command line options to Firefox - it's not immediately clear to me if that would be easy or problematic. Migrating users to the new mechanism would require a lot of careful updater work. I don't even know exactly what is involved in shipping the entirety of Firefox as a framework on macOS, but I have been told that it wouldn't be easy. And the Linux story is still a big unknown.

Where Do We Go From Here?

This is still an open question. We are continuing to search for other solutions to this problem and refine the ones that we have.

I'm very reluctant to implement the first solution for the reasons already given. And the other solutions are most likely more involved than the Update team can handle without lots of additional help. In other words, they are probably impractical without buy-in from the rest of the organization. Since, for better or for worse, this problem affects a relatively small group of people, this is a tough sell.

With respect to versioned installations on linux, having the firefox installation in a versioned directory, however on most distros, I think the old package will be removed before the new package is added. Possibly you could have a different package for each version of firefox and have the launcher package depend on both, but that would be kind of hacky, would require having two installations all of the time, and would be difficult for packagers.

Another potential solution could be to copy the executables and libraries needed to s temporary location on startup, and use those to start new processes. That would definitely add time to startup though. Or maybe this process could be done lazily. Before an update happens, if there is a Firefox instance running, copy the existing files into a temporary directory, and use IPC to notify the running process to use the new directory, then install on the base install directory. Of course, the temporary directory would need to be cleaned up somehow. There are several mechanisms to do that on linux, but I'm not sure about other OSes.

Finally, It may be worth looking to see how Chromium handles this, since it has multiple processes but doesn't run into this problem on out-of-band updates (such as from a linux package manager update).

See Also: → 1553982
See Also: → 1711143

(In reply to Kirk Steuber (he/him) [:bytesized] from comment #8)

This problem can basically be caused in two known ways. If you are encountering this problem, and none of these situations seem to apply to you, please let us know!

  1. While Firefox is running, another instance of Firefox using the same installation updates. This is typically the result of running multiple profiles simultaneously. This is also theoretically possible on a system with multiple simultaneous users but, to my knowledge, no one has ever reported this problem to us.

Is it expected that this can happen even when both (or all) profiles involved have "Check for updates but let you choose to install them" selected in about:preferences?

(In reply to Botond Ballo [:botond] from comment #10)

Is it expected that this can happen even when both (or all) profiles involved have "Check for updates but let you choose to install them" selected in about:preferences?

That setting actually controls when an update is downloaded, not installed. Once an update is downloaded, it doesn't matter what any of your settings say. We will always try to install it on Firefox startup, possibly causing this bug.

Blocks: 1724242

(In reply to Kirk Steuber (he/him) [:bytesized] from comment #11)

(In reply to Botond Ballo [:botond] from comment #10)

Is it expected that this can happen even when both (or all) profiles involved have "Check for updates but let you choose to install them" selected in about:preferences?

That setting actually controls when an update is downloaded, not installed. Once an update is downloaded, it doesn't matter what any of your settings say. We will always try to install it on Firefox startup, possibly causing this bug.

I see. I can see how this would be confusing for users, though. Maybe a good first step would be to document that this is the expected behaviour somewhere :) I've been hit by this (and the related bugs) quite often.

Severity: normal → S3

The severity field for this bug is relatively low, S3. However, the bug has 6 duplicates and 8 See Also bugs.
:Amir, could you consider increasing the bug severity?

For more information, please visit auto_nag documentation.

Flags: needinfo?(ahabibi)

The last needinfo from me was triggered in error by recent activity on the bug. I'm clearing the needinfo since this is a very old bug and I don't know if it's still relevant.

Flags: needinfo?(ahabibi)
Flags: needinfo?(ahabibi)

I had an interesting experience the other day with several of my Firefox processes (all running different profiles, I have 8 in regular use, and a few others in occasional use) telling me to restart to update when I tried to open a new tab, and when I did, the old tabs were there (but not the new one). This actually seems to be an improvement over just hanging and not working, as often happened before.

(In reply to Robin Steuber (they/them) [:bytesized] from comment #11)

(In reply to Botond Ballo [:botond] from comment #10)

Is it expected that this can happen even when both (or all) profiles involved have "Check for updates but let you choose to install them" selected in about:preferences?

That setting actually controls when an update is downloaded, not installed. Once an update is downloaded, it doesn't matter what any of your settings say. We will always try to install it on Firefox startup, possibly causing this bug.

On all of my machines this setting only works sporadically. Firefox usually will download and install updates without any notification and force me to restart without warning and lose everything in my private tabs. I do not have multiple profiles.

(In reply to Steve from comment #23)

(In reply to Robin Steuber (they/them) [:bytesized] from comment #11)

(In reply to Botond Ballo [:botond] from comment #10)

Is it expected that this can happen even when both (or all) profiles involved have "Check for updates but let you choose to install them" selected in about:preferences?

That setting actually controls when an update is downloaded, not installed. Once an update is downloaded, it doesn't matter what any of your settings say. We will always try to install it on Firefox startup, possibly causing this bug.

On all of my machines this setting only works sporadically. Firefox usually will download and install updates without any notification and force me to restart without warning and lose everything in my private tabs. I do not have multiple profiles.

Then this bug does not apply to you at all. If you file a new bug, we can dig into this further.

Duplicate of this bug: 1849967
Duplicate of this bug: 1882211
Duplicate of this bug: 1867393

Firstly, thanks to @Robin Steuber for this exemplary post; very informative!

While Firefox is running, another instance of Firefox using the same installation updates. This is typically the result of running multiple profiles simultaneously.

Likely, that's me; see --> https://bugzilla.mozilla.org/show_bug.cgi?id=1867393

I'm not entirely sure that the UX I experience is covered here, but it seems pretty close. I'm on Win10, so no linux package managers and my AV is malwarebytes (I do NOT use their browser guard thing), though I've never suspected AV interference.

I use at least four profiles plus an unknown number of others "generated" by geckodriver/marionette via Selenium etc. <-- these can be the first to make me aware there's an update pending: marionette issues suddenly appear in my output console -- but not a peep from my main Firefox -- until I open the about dialog: "Firefox is being updated by another instance". ME: Well give me a button to update then!

The poor UX (that's an understatement!) is centered around the fear that Firefox may not save my current tabs/session.
  -- I just counted them up: on my main profile, across 9 Firefox windows, I have 47 tabs active. I have another 3 Firefox windows (that I consider "apps") each using separate/different profiles each with 1 tab open (like I said, they're apps not used for browsing purposes). Even so, closing them is easy -- nothing to remember. Having closed them, Firefox in my main profile will still tell me "Firefox is being updated by another instance". <cue-anger-management-lessons>

So, in light of the fact this is not going to be fixed anytime soon, short of me saving a ton of bookmarks, what moves can I make to SAVE my tabs/session in my main profile and somehow force the update to happen? I mean really, that's ALL I NEED. I read somewhere that about:restartrequired would do the trick. It does restart the browser with my tabs/session intact but it doesn't perform the update. <cue-more-anger-management-lessons>

If there's any more info you need, don't hesitate...

Thanks.

(In reply to Codacoder from comment #29)

So, in light of the fact this is not going to be fixed anytime soon

We actually are hoping to address this soon.

what moves can I make to SAVE my tabs/session in my main profile and somehow force the update to happen?

Just close all the browser instances except one and update that one.

Just close all the browser instances except one and update that one.

And my session?

And, just for clarity, how do you define "browser instance"? A browser window? A Firefox window attached to one profile?

(In reply to Codacoder from comment #31)

Just close all the browser instances except one and update that one.

And my session?

I don't understand this question, sorry.

(In reply to Codacoder from comment #32)

And, just for clarity, how do you define "browser instance"? A browser window? A Firefox window attached to one profile?

Generally you want to pick a single profile to keep open and then close every window using a different profile from that one.

Technically, it is possible to have a single profile with multiple windows running in different Firefox instances by using the --no-remote or possibly the --new-instance switch. But unless you are going out of your way to launch Firefox with one of those switches, that shouldn't really be a concern.

There is technically one other thing you could do here. I'm a little reluctant to mention it, but I will for completeness's sake. I believe that you can prevent the multi-instance update lockout by setting app.update.checkOnlyInstance.enabled to false. Note that we still do not allow update to be driven by multiple Firefox instances simultaneously. Which basically means that setting this pref will allow one of the Firefox instances to update, not all of them. The other(s) will continue to give the same message.

But there is a reason that this is not the default behavior: the problem described by this exact bug. It is quite likely that if you disable app.update.checkOnlyInstance.enabled, you are going to encounter "Restart Required" pages more often.

And my session?

I don't understand this question, sorry.

It is my experience that closing Firefox windows and then updating from the last one, will not (always) allow me to restore my previous session on restart (that's the nub of the complaint behind my reporting the linked bug, that Firefox tells me to update, but (A), Firefox won't let me due to "Firefox is being updated by another instance" and (B), fear losing my session trying to get it to update).

That advice, sadly, puts me back at square one (and invites me to repeat my question: what to do to safely force the update without session loss).

You didn't actually give me a clear answer to: define "instance of Firefox". I'm guessing it's a window that was started by any given firefox.exe. Correct? Not new tabs, for example. But what about torn-off tabs; are they new instances?.

I didn't know about --new-instance. I only use --no-remote on those apps I mentioned, each having their own profile. I stopped using --no-remote in my day-to-day a few years ago, hoping that would cure my update woes. Didn't work, obviously.

FYI, my startup batch statements for the apps stuff I mentioned:

%FIREFOX% -wait-for-browser -profile %jQPROFILE% -no-remote http://127.0.0.1:57/Patches/%DOCFILE%.html
REM %FIREFOX% -profile %jQPROFILE% -no-remote http://127.0.0.1:57/Patches/%DOCFILE%.html

Every so often, I have to use -wait-for-browser, a few releases later, guaranteed, I have to remove it again -- hence the REM.

There is technically one other thing you could do here. I'm a little reluctant to mention it...

Sheesh. I'm not surprised. Thanks but no thanks ;)

Lastly, I'm pretty sure I have never encountered the "Restart Required" message in my normal day-to-day. Clue?

Thanks for reading and thanks for the engagement!

(In reply to Codacoder from comment #34)

And my session?

I don't understand this question, sorry.

It is my experience that closing Firefox windows and then updating from the last one, will not (always) allow me to restore my previous session on restart (that's the nub of the complaint behind my reporting the linked bug, that Firefox tells me to update, but (A), Firefox won't let me due to "Firefox is being updated by another instance" and (B), fear losing my session trying to get it to update).

Ah, okay I understand. Let me revise my instructions.

Choose one profile to keep open. For the windows belonging to other profiles, go to the Hamburger Menu and press Exit or use the hotkey Control+Shift+Q. This will shut down the whole instance at once rather than closing each window so that all windows will be in session restore next time.

When you relaunch, session restore can be accessed in several ways. You can go to Hamburger Menu -> "History" -> "Restore Previous Session". Or you can navigate to about:sessionrestore. Or you can navigate to about:preferences and in the "General" -> "Startup" section check the "Open previous windows and tabs" to automatically restore your session every time the profile starts.

You didn't actually give me a clear answer to: define "instance of Firefox". I'm guessing it's a window that was started by any given firefox.exe. Correct? Not new tabs, for example. But what about torn-off tabs; are they new instances?.

Unfortunately the answer is a bit complicated. In general, when you launch Firefox and it is not yet running using that profile, that's a new instance.

When you launch Firefox and there is already an instance running using that profile, this generally does not create a new instance. A Firefox process is briefly created, but it exits immediately after passing a message to the existing instance telling it what the user wanted to do (open a URL, create a new window, etc). This is why it's possible to click on a link from, say, Thunderbird and have it open a new tab in an existing Firefox window.

When you open a window, say, via Hamburger Menu's "New window" or "New private window" button, that doesn't start a new instance either. All of the windows using the same profile are ultimately controlled by the same Firefox parent process (i.e. they are part of the same instance). Which is why you can do things like dragging tabs between them.

But, --new-instance and --no-remote cause Firefox to work differently. I'm not especially knowledgeable about the specifics, but using them disables some or all of the message passing behavior that I mentioned above. Potentially this allows you to start many instances in parallel which can be useful for testing. But I don't believe that this is particularly useful to anyone outside of testing. In fact, if you try to use this to launch a second instance of a regular profile without any other switches, you are probably just going to get a "Firefox is already running" dialog when the second instance tries to lock the profile and finds that it is already locked by the other instance.

So, basically the answer here is that each running profile represents a Firefox instance. If you use --new-instance or --no-remote, this potentially gets more complicated. But you probably shouldn't use those switches anyways.

Lastly, I'm pretty sure I have never encountered the "Restart Required" message in my normal day-to-day. Clue?

Probably just means that the "You are being updated by another instance" feature is working properly. It's intended to address exactly this. It's just not perfect at its job nor is it ideal in terms of getting people updated quickly.

Thank you Robin, for the clarity and diligence. Very much appreciated.

You need to log in before you can comment on or make changes to this bug.