Open Bug 1614384 Opened 6 years ago Updated 7 months ago

Can't handle change of network connection - Clearing the DNS cache on a network change and work offline On/Off and clear cache and OS flush DNS actions...

Categories

(MailNews Core :: Networking, defect)

Product:
MailNews Core
Component:
Networking
Type:
defect

Tracking

(Not tracked)

Status:
UNCONFIRMED

People

(Reporter: mark, Unassigned)

References

Details

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:72.0) Gecko/20100101 Firefox/72.0

Steps to reproduce:

68.4.2 There are some actions that cause TB to not be able to connect to the current network connection. Change the network by closing Wi-Fi and enabling Ethernet. Start up a VPN after TB has started up. Unplug the Ethernet connection, thereby changing to a Wi-Fi connection, if available or vice versa. Plug the Ethernet connection back in after disconnecting the cable.

Actual results:

In those situations, TB can no longer make an outside connection to the mail servers (any of them). It is necessary, then, to Quit TB and restart. Often it is the "startup" applications that conflict with one another. Example, TB starts quickly, VPN starts a bit later. Result is that TB cannot connect to the outside world. TB reports that it is unable to connect, or a "timeout" occurs, etc.

Expected results:

Any change in the network should be sensed by TB and not assume that there is only one available connection. TB should then "find" the available connection and use it. The same happens in Firefox (I will report same feature request in that forum).

In my opinion this is a bug and not a feature request. It exists both on Windows and MacOS. Can this ticket be changed to a bug report?

Is there some reason you can't take Thunderbird offline using the icon in the status bar or the File > Offline > Work Offline menu item before doing all this reconfiguring?

Then go back online after.

Does the issue still present itself using the currently supported 91.3.2 version? Version 68 is EOL and no longer supported.

Flags: needinfo?(mz)

That workaround would probaby work, but changes can happen anytime, for example when I'm at work and remove my laptop from a docking station and it connects over WiFi instead of Ethernet, or when I have to connect to a different location through a VPN.

This happens in the latest version (91.3.2).

Flags: needinfo?(mz)
Type: enhancement → defect
Summary: [Feature request} Can't handle change of network connection → Can't handle change of network connection
Summary: Can't handle change of network connection → Can't handle change of network connection - Clearing the DNS cache on a network change and work offline On/Off and clear cache and OS flush DNS actions...

I don't think I have seen this.

Antony, m_kato, can you reproduce?

Flags: needinfo?(m_kato)
Flags: needinfo?(acdp)

I can reproduce this easily - it happens very often since I must now disconnect my VPN for certain other needs (certain web sites will not permit a VPN connection, etc.). All I need to do is to start up the system with VPN, start TB and other applications and use, then disconnect VPN in order to access a "particular" web site. TB is now unable to check for mail - timeout or other connection error. Workaround (as I have experienced) is to quit TB and restart. Using 91.7.0 (64-bit) now on Mac.

Gecko provides nsINetworkLinkService to detect network connection, but comm-central won't check whether network is online/offline automatically using it. So I guess that this will occur, but I think that this is a kind of feature request.

Flags: needinfo?(m_kato)

So, as I understand it, the workaround is to take TB off-line, then change network connections, then bring TB back on-line. Did I get that right? Thx.

(In reply to mark from comment #8)

So, as I understand it, the workaround is to take TB off-line, then change network connections, then bring TB back on-line. Did I get that right? Thx.

I did myself encountered such issue with TB when using vpn especially in work environent where IP of mail server (or other intranet services) change due to vpn connection to an internal ip (traffic routed via vpn) vs external IP (direct Internet access).

Upon network interface change events (such as connection/disconnection), I noticed that putting TB offline and back online never sorted the issue because it does not clear/update/invalifate TB internal DNS cache, nor ipconfig /flushdna on Windows does (because TB uses its own cache mechanism, it seems), nor clearing TB cache.

Workarounds always been to exit TB and restart it or wait for TTL to expire (internal IP TTL set to 5mn to reduce inconvenience) but none of those are ideal. TB should be able to detect changes in network connectivity and "adapt" automatically to the environnent.

This issue has already been reported but never been fixed. I am glad to see it is reported here and that there is now a glimpse of explanation about the possible cause. My view on the matter is that it should be considered as a bug not as a feature request ;-)

(In reply to Wayne Mery (:wsmwk) from comment #5)

I don't think I have seen this.

Antony, m_kato, can you reproduce?

I'll have a go a bit later and report - me also (AdP)

Flags: needinfo?(acdp)
Severity: normal → S3
Component: Untriaged → Networking
Flags: needinfo?(acdp)
Product: Thunderbird → MailNews Core

Is the bug around the use of a VPN only or in the main, or just one aspect?

Or is it mainly caused by switching networks between WiFi/Ethernet?

I do not use VPN as a rule, but switching networks does not cause TB to stop connecting to the server, I was switching whilst TB was still on-line

Ah, I now find that I have reporduced this oissue, I had to restart TB to get new emails. Off/On-line switching was not enough

Flags: needinfo?(acdp)

It seems TB 131.0b4 (64-bit) is still not able to properly handled network status change and/or to trigger Get Message automatically.

I found two other use case with IMAP mailbox:

  • Keep Thunderbird opened, disconnect from network, put computer to sleep. Next day, wake up computer, connect to network, Thunderbird does not trigger the Get Message function to check for new emails. You either have to click manually the Get Message button or wait for auto-check as per timer setting set.

  • While in Inbox, Write a New Message to yourself... Upon Send the message is saved in Sent items, but it does not appear in Inbox until Get Message is triggered manually or time for auto check is reached...

It feels that Thunderbird is still kind of buggy and does not always properly and immediately update Inbox by itself in a smooth manner.

Sometime it also feels that the Inbox is not properly updated you seems to have received no new email and then suddenly a lot of them appears after a long while... Either Get Message seems not triggered properly or Inbox view UI not properly updated after Get Message triggered by the timer. That usually happens when Thunderbird is iddle for a while not used while the view still in Inbox. The work around being either to trigger Get Message manually (which I feel I have to do regularly not to miss messages) or click on another folder eg Sent and come back to Inbox... which seems to refresh the view and possibly trigger the Get Message... so new messages appear.

Edit: Some of this is just WRONG. See next comment 14.
Hi Richard,
I think what you observe in your two use cases is pretty much expected behavior. Say the computer sleeps or hibernates. After a while (30 minutes per imap spec, but some servers quicker) the server times out the connection due to inactivity. When the computer then wakes up, the connections are all half-closed since TB doesn't know they are closed (but the server does). So TB does nothing until the next biff time (check for new mail time) occurs. I think the biff time is measured as TB operating time and not based on actual wall time, but not sure. So then at first biff after wake up, TB tries to check for new mail (typically just sends NOOP) and receives an error that there is no connection. So only now does TB know the connection is closed and then creates a new one to retry the NOOP command and normal checks for new mail cycles resume.

So if biff time is set to, say, 25m, you might not receive new mail until 25m after the computer wakes up (unless you click get messages or re-select inbox, as you describe).
Note: I recommended to set biff time to no more than 30m so the server doesn't close/timeout the Inbox imap connection while computer is awake. This will still allow the imap IDLE command to signal there are new messages immediately when they arrive, provided the server supports IDLE and it is enabled with TB's server setting "allow immediate notification when new mail arrives.

What I wrote in comment 13 is not completely correct. I did an actual test and let the systems sleep for several hours and looked at what is happening with IMAP:4 log and wireshark. I have the biff time set to 28m and the server does not support IDLE. When the system sleeps or hibernates, TB receives a signal that tells the imap threads to shutdown, this causes imap logout and/or close to occur and TB attempts to close the TCP/IP connections to the server. It appear that hibernate/sleep usually occurs before the server receives or acks the tcp FIN signal to terminate the connection, but TB knows that the connections are now gone and the task loops of each imap thread are exited and the threads are destroyed.
Before sleeping, I made sure the root folder (the account) is selected and not Inbox or other folder. On wake up, there is no delay in re-establishing the connection to Inbox. I thought with 28m biff time there may be a delay of up to 28m, but I don't see that. Instead, Inbox is imap SELECTed and any new message headers are immediately detected and fetched (I had two new messages and was immediately notified on wake-up).

Keep Thunderbird opened, disconnect from network, put computer to sleep. Next day, wake up computer, connect to network, Thunderbird does not trigger the Get Message function to check for new emails. You either have to click manually the Get Message button or wait for auto-check as per timer setting set.

I missed that you are first disconnecting from network. I didn't try that with my test and never do that before sleeping/hibernating. Do you see delay in receiving new messages on wake-up if you don't disconnect from network? Maybe I will repeat my test and add those steps and see if it matters.

Maybe I will repeat my test and add those steps and see if it matters.

Disconnecting from network (e.g., turn off wifi) causes the same signal as sleeping/hibernating: TB imap threads are exited and TB attempts to shutdown connections.
Then I sleep the system and wait a while and then wake up.
On wake up, tb tries to immediately connect to server but fails because network is down (this is the first biff attempt). Then I connect the network again. This doesn't cause TB to immediately poll (biff) for new messages but it waits for the next biff interval to occur (the first biff failed because the network was down). In my case this is 28m so only after 28m can new messages arrive.
So I don't think there is a problem if you don't shutdown the network before sleeping the system.
Maybe I'm not understanding why you think the network needs to be shutdown before sleeping/hibernating the system?

Maybe I'm not understanding why you think the network needs to be shutdown before sleeping/hibernating the system?

I mostly use personal hotspot at home, meaning that my computer do not necessarily reconnect to network immediately after wake up. Also often I may put my computer to sleep, change location, and when I wake up computer at new location I may need to connect to a new wifi network I never used before. I may also choose not to connect automatically to known network, there are many use case where upon wake up the computer may not necessarily be connected to the network.

TB shall not try to reconnect unless the network is ready and functional, and should do so as soon as it happens.

In all those case I would expect TB to run automatically a Get Message command shortly after the network is connected so I would immediately get new message automatically without having to wait 28mn (or what ever timer is set to for auto checks) or having to press the button manually or change folder to trigger such reaction.

Currently it just feels TB is buggy and does not work as expected "ideally"...

Does that make more sense?

The alternative use case could also be a switch of network for example after a VPN connection that may completely change IP range (connection via local IP rather than external IP).

TB shall not try to reconnect unless the network is ready and functional, and should do so as soon as it happens.

Right now that doesn't happen. An attempt is made to connect and it fails and there is no real retry other than try again at the next biff time. Also, don't think there is a signal that there is no network available at TB startup or after an OS wake-up. Or if there is a signal (network up or down), it is not currently hooked onto in imap code (but I could be wrong :)).

The alternative use case could also be a switch of network for example after a VPN connection that may completely change IP range (connection via local IP rather than external IP).

I don't have a VPN but I do have a cell phone wifi "hot spot" that I use sometimes when electrical power is off (storms, etc). I think when I use it I can switch between it and normal wifi and TB still keeps going OK. This changes the route but not the end IP address of the server so maybe not the same issues as VPN?

(In reply to gene smith from comment #17)

TB shall not try to reconnect unless the network is ready and functional, and should do so as soon as it happens.

Right now that doesn't happen. An attempt is made to connect and it fails and there is no real retry other than try again at the next biff time. Also, don't think there is a signal that there is no network available at TB startup or after an OS wake-up. Or if there is a signal (network up or down), it is not currently hooked onto in imap code (but I could be wrong :)).

It should be down to the automatic triggering of Get Message button command upon network up event. This action do re-connect (I believe) to the IMAP server as part as the process and check for new emails.

The alternative use case could also be a switch of network for example after a VPN connection that may completely change IP range (connection via local IP rather than external IP).

I don't have a VPN but I do have a cell phone wifi "hot spot" that I use sometimes when electrical power is off (storms, etc). I think when I use it I can switch between it and normal wifi and TB still keeps going OK. This changes the route but not the end IP address of the server so maybe not the same issues as VPN?

I am more referring to the case where the end IP address of the server change in the process of change of network. E.g local IP vs remote IP for example. Which may be likely to happen if you connect to a corporate VPN for example, especially if set to route all traffic via VPN.

Upon network status change, Thunderbird shall not only Get Message when network up event, but clear existing connection and re-connect with a new one to take into consideration network changes. That include clearing connection cache such as socket, DNS, etc...

Recent version of windows clear the Windows DNS cache upon network changes (before it was not) but because Thunderbird use its own cache (rather than the system one) its DNS cache is not cleared. Still trying to use old cached end IP address, instead of looking up again to get the new one and re-connect with it to the server.

The mechanism may already be available in Thunderbird, they may just not be properly triggered upon network up event.

That said, for the change of IP issue, there may need some adjustments as putting TB offline and back online does not seems to clear cached connection and reset it. Which I would expect should happen in that case as well.

Does it make more sense now?

In a nutshell...

A network up and IP change (or route table change?) event shall cause TB to:

  • try to contact the server via existing cached connection, if exists, and if it fails...
  • terminate any cached connection to the server, if exists (JavaScript object, network socket)
  • clear cache to remove any settings related to old connection, if any (eg server end IP from DNS cache)
  • run a new DNS lookup to get server end IP
  • re-connect to the server with new network settings - if that fails stop here and status notify user (retry after timer), if that succeed...
  • trigger automatically Get Message command to check for new emails, events, etc...
    which will update the views eg Inbox folder to show new msg

The same process could be triggered when you turn TB online via the TB toggle offline/online button. Currently this does not seems to re-initialise connection to the server while it should.

I suspect all features are there, it may just be done to have them properly triggered, in an orderly manner, upon the proper event(s).

That said, for the change of IP issue, there may need some adjustments as putting TB offline and back online does not seems to clear cached connection and reset it. Which I would expect should happen in that case as well.

Not sure what you mean by "cached connection[s]" here. What tb imap code calls "cached connections" do get closed (network FIN is sent) and the connection thread is exited when you go offline. So what is called "cached connections" are cleared and closed.
Then when you go back online and access a folder, a new "cached connection" is established (network SYN, etc) in a new thread.
But maybe you are referring to the DNS cache when you say cached connection? The current imap code doesn't deal with that at all.

Maybe I'm wrong, but if TB needs to make a new imap connection and the ip address of the server has changed, the mozilla network code should notice the failure (using the cached ip address) and do a new DNS lookup of the server hostname and try again with the correct ip address. TB should only see a failure if mozilla network code can't find the ip address that will allow the connection.

(In reply to gene smith from comment #20)

That said, for the change of IP issue, there may need some adjustments as putting TB offline and back online does not seems to clear cached connection and reset it. Which I would expect should happen in that case as well.
new DNS lookup of the server hostname and try again with the correct ip address.

I am referring to something cached in TB/Firefox (network connection "object" or settings) that prevent TB IMAP connection to the new end server IP when it changes. Eg from external to internet for example.

I can't confirm exactly what as I don't directly know, but I have observed it numerous time. Putting TB offline and back online does not fix the issue till you exit and restart Thunderbird or if you wait for a long while (dns cache record ttl?). Something with old connection or settings is cached or not reset properly. Maybe network socket issue? DNS cache issue? DNS request issue?

If TB/Firefox use their own DNS cache, which I believe they do separate from the system DNS cache (correct me if I am wrong here), the DNS lookup answer may return the old end IP rather than the new one until cache record times out. That is my believe but it could well be something else like not terminated or reset properly to re-establish connection to the server via new IP address.

You also mentioned that when you put TB back online, you have to access a folder to re-establish connection to the server. Why such connection is not triggered automatically after few seconds shortly after TB is back online (back online event), eg by triggering automatically the Get message command rather than waiting for end-user to change folder manually, trigger the command manually or timer expires for auto-check? If Thunderbird is already in Inbox folder view, it is unlikely user would change folder... The view should refresh itself automatically as TB is back online, the same way it is done when you starts TB, it automatically check with the server for changes.

When user sets tb offline, there is a direct command that tells cached connections to be dropped and imap threads to shutdown (on observed event "network:offline-about-to-go-offline"). This also happens when "sleep" event is observed. However, when user goes back online or wakes up, there is currently nothing that happens and it depends on the either biff occurring or user doing "get messages" or selecting a folder to trigger a url that causes a re-connect.
To do what Richard suggests in comment 21, which seems like a good suggestion, I think there would need to be some "observe" on event "network:offline-status-changed" to take the appropriate action (i.e., simulate what happens when TB starts up with a good network and make connections for accounts having "check for new mail on startup" set). But how to do this and if it is really something that is needed is beyond me.

Came across the below if that can help?

Option 1 - Online event - Monitor UP/DOWN network status

According to this article https://developer.mozilla.org/en-US/docs/Web/API/Navigator/onLine, there is an "online" event that can be observed.

To see changes in the network state, use addEventListener to listen for the events on window.online and window.offline, as in the following example:

window.addEventListener("offline", (e) => {
console.log("offline");
});

window.addEventListener("online", (e) => {
console.log("online");
});

Of course if you disconnect VPN it does not necessarily that you become offline from a Windows point of view. But at least it may be useful to handle the case where network connection is back online after being offline entirely.

Option 2 - Change event - Monitor CHANGE network status (include UP/DOWN?)

addEventListener("change", (event) => {});

onchange = (event) => {};

Source: https://developer.mozilla.org/en-US/docs/Web/API/NetworkInformation/change_event

If network status changed (including IP settings ans DNS - cache to be reset), or up from down, TB should be able to detect it and trigger automatically reconnection and sync of emails (get message command) then calendars (reload and sync changes command), then address books (sync changes command).

Option 3 - NetworkLinkService (or similar?) - Monitor UP/DOWN/CHANGE network status

Someone seems to have allowed Firefox to react to network changes, years ago... but some people were complaining of the same issue and I must admit I have seen it in recent version of Firefox. So may not be totally resolved in Firefox either, but the pref option and feature shall be present and usable by TB.
bug 939318
It could be that Option 1 and 2 are related to that fix.

The fix author wrote an article describing well the issue...
https://daniel.haxx.se/blog/2014/09/26/changing-networks-with-firefox-running

See Also: → 1439379

Magnus can something in comment 23 be used?

Flags: needinfo?(mkmelin+mozilla)

I think partly though would need some more exploration.

For network seems there is network:link-status-changed and some more topics that can observed
For online/offline not sure more is needed. mail-offline.js probably handles it.

Flags: needinfo?(mkmelin+mozilla)

(In reply to Magnus Melin [:mkmelin] from comment #25)

For online/offline not sure more is needed. mail-offline.js probably handles it.

From past experience with Thunderbird, putting Thunderbird offline, waiting, and back online has never fixed this bug, so no it does not seems to handle it. This action does not clear/reset DNS cache, or network connection even if the underlying network has changed, you always had to Exit Thunderbird and restart it so the network socket (and possibly Thunderbird DNS cache) would be re-initialised inline with new underlying network settings.

Even though I agree with you it should handle it as well.

Though that does not change the fact that Thunderbird shall also be able to handle network changes on the fly while remaining online at all time. It shall not be required to put Thunderbird offline and back online when underlying network change occurs in the operating system, it should happen automatically within Thunderbird.

Currently I don't think if you clear DNS cache on Windows "ipconfig /flushdns" it would clear DNS cache within Thunderbird, nor would Thunderbird detect such network changes.

You need to log in before you can comment on or make changes to this bug.