1321841 - Firefox network loss after connection to Cisco VPN.

Status: RESOLVED INCOMPLETE

(Core :: Networking, defect, P3)

Description

[Baner]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0
Build ID: 20161129173726

Steps to reproduce:

Normally browsing internet, and  connect to corporate VPN using Cisco AnyConnect Secure Mobility Client Ver: 3.1.11004


Actual results:

Firefox Connection is lost, and the only way to get it working again is:

1: Restart Firefox.
2: Goto  settings - advance - Network - connection-setting -  and then 'toggle' between "Auto detect proxy settings" & "Use system proxy settings".


Expected results:

Firefox should keep connected.

Comment 1

[Dragana Damjanovic [:dragana]]

Can you make a Http log:
https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging

Just change line:
set MOZ_LOG=timestamp,rotate:200,nsHttp:5,nsSocketTransport:5,nsStreamPump:5,nsHostResolver:5
into:
set MOZ_LOG=timestamp,nsHttp:5,nsSocketTransport:5,nsStreamPump:5,nsHostResolver:5

Thanks!

Comment 2

[Baner]

Attachment: log.zip

Open Youtube.com - OK
Connect to corporate VPN
Open Youtube.com - OK
Disconnect from VPN
Connect to Youtube.com - Not OK
Goto  Options - Advanced - Network - Settings -  and then 'toggle' between "Auto detect proxy settings" & "Use system proxy settings".
Open Youtube.com - OK

Comment 4

[Dragana Damjanovic [:dragana]]

From log, I can see that we still return a proxy with OnProxyAvailable even after the vpn is disconnected. Looking at our proxyservice we do query the system for every connection, no caching (I have not look at this code for a very long time maybe I am wrong), and we query the windows system using system function InternetGetConnectedStateExW and according to the response we set up a proxy connection or not. I do not have a Windows machine to try it and we only have pac logging but not windows specific logging. To be more certain we will need proxy log  as well.

Baner, can you make the same log with:
set MOZ_LOG=timestamp,nsHttp:5,nsSocketTransport:5,nsStreamPump:5,nsHostResolver:5,proxy:5

Daniel, you looked at the proxy/pac code recently. Am I correct here?

Comment 5

[Daniel Stenberg [:bagder]]

This problem with Cisco AnyConnect VPN seems to have been around since basically forever. See Bug 651333 for what sounds like exactly the same symptoms (and a few others through history since)

It sounds like your description matches what we've seen before Dragana.

So does Firefox always use a proxy when AnyConnect is active? I would expect a VPN to mostly work on network interface level.

I think we really need someone in the necko team to setup and run Cisco AnyConnect on a test machine and see what exactly is going on so that we can finally nail this long-going nuisance to users.

Comment 7

[Gary Chen [:xeonchen]]

(In reply to Daniel Stenberg [:bagder] from comment #5)
> I think we really need someone in the necko team to setup and run Cisco
> AnyConnect on a test machine and see what exactly is going on so that we can
> finally nail this long-going nuisance to users.

I'm going to set up the environment soon.

Comment 8

[Gary Chen [:xeonchen]]

Attachment: log.txt.zip

I'm using:

- Windows 7 SP1 x64
- Firefox 50.1.0 x86
- Firefox 50.1.0 x64
- Cisco AnyConnect Secure Mobility Client 3.1.13015

to test, but I can't reproduce by following steps:

1. open Firefox
2. open [1], success
3. connect AnyConnect to VPN
3.1 I can see "route print" command under prompt adds a new default route with lower metric
4. open [2], success
5. disconnect VPN
6. open [3], still success

Tried following proxy settings:
- Auto-detect proxy setting for this network
- Use system proxy settings (which uses a remote PAC file)

Not sure if it needs a more complicated network environment because I set up the VPN server in the same LAN.

[1] http://example.com/
[2] http://www.ipchicken.com/
[3] http://www.google.com.tw/

Comment 9

[Shian-Yow Wu [:swu]]

According to bug 651333 comment 3, the problem is related to system proxy setting been changed by VPN disconnection.  Did you encounter the same situation?

Comment 10

[Gary Chen [:xeonchen]]

(In reply to Shian-Yow Wu [:swu] from comment #9)
> According to bug 651333 comment 3, the problem is related to system proxy
> setting been changed by VPN disconnection.  Did you encounter the same
> situation?

I missed that comment, thank you swu.

Comment 11

[Gary Chen [:xeonchen]]

OK, I just made following modification, still cannot reproduce.

- Client
  - Network 1 is wired network
    - IP: 10.0.0.2
    - Proxy: 10.0.0.1
  - Network 2 is VPN network
    - IP: 192.168.1.2
    - Proxy: 192.168.1.1

1. Connect to VPN, open Wireshark to observe network 2
   -> http traffic is through 192.168.1.1
2. Disconnect from VPN, open Wireshark to observe network 1
   -> http traffic is through 10.0.0.1 now

Both Chrome 55.0.2883.87m and Firefox 50.1.0 works.

Comment 12

[Laurens Vets]

I will try and get a log file this weekend. Chrome always works for me.

Steps I do:
1. Connect corporate VPN.
2. Surf in Firefox.
3. VPN disconnects or reconnects because of a network issue.
4. Surfing doesn't work anymore in Firefox.

Comment 13

[Laurens Vets]

Ok, I've made a HTTP log.

Cisco AnyConnect client 3.1.14018
Firefox 50.1.0
System proxy set to corporate proxy

Steps to reproduce:

1. Corporate VPN Client is connected.
2. Open Firefox, homepage is https://www.google.com/ -> ok.
3. Surf to www.slashdot.org -> ok.
5. Disconnect & reconnect VPN
6. Click link on open slashdot page -> "Unable to connect. Firefox can’t establish a connection to the server at games.slashdot.org."
7. Try and surf to www.tweakers.net -> Same error message.
8. Try and surf to www.cnn.com -> Same error message.
9. Surf to www.hackaday.com in a new tab -> Same error message.
10. Implement workaround (Go to settings - advance - Network - connection-setting -  and then 'toggle' between "Auto detect proxy settings" & "Use system proxy settings".)
11. Refreshing www.hackaday.com now works.

Anything else I can provide?

Comment 14

[Laurens Vets]

Attachment: log_20170113.zip

Comment 15

[Gary Chen [:xeonchen]]

Hi Laurens,

Would you tell me your proxy setting type (auto-detect, use system, ...etc) before/after connected to VPN?

Comment 16

[Laurens Vets]

In Firefox: "Use system proxy settings"

Under LAN Settings in Windows 7: "Use automatic configuration script" with the Address filled in ("http://pac.domain.com/standard.pac").
"Automatically detect settings" is ticked off as well as "Use a proxy server for yourLAN (..."

As far as I know, this is default setting here and I don't manually change it beofre/during or after VPN reconnection.

Comment 17

[Gary Chen [:xeonchen]]

Attachment: 0001-Bug-1321841-Use-xor-to-avoid-checksum-overflow-r-bag.patch

I still don't reproduce the bug, but I found a possible root cause.

|NetworkChanged| is called only the first time VPN connected or disconnected because the variable |sum|, which is used to calculate checksum of network interfaces, will overflow when there are more than one interfaces because of the left shift operation.
The value of |sum| is always the same whether the VPN is connected or not.

Daniel, do you think it could be the root cause and it's worthy to land this patch and see if the bug is fixed?

Comment 18

[Daniel Stenberg [:bagder]]

Attachment: 0001-bug-1321841-better-checksum-for-live-adapters.patch

Argh! Yes I do think it can at least contribute to this problem as the logic uses the checksum to avoid signalling if the situation is identical to before. Good catch!

I think the primary mistake is that the left shift is done there for each letter in the adapter name, and not just once per adapter. But I also like your XOR suggestion. The point of course being that every letter in all names of the adapters that are UP need to have the same significance.

I'd like to suggest a combination: fixing the shift to happen less frequent (but letting 'sum' range over all adapters so that we'll use more bits) and the XOR.

Thoughts?

Comment 19

[Gary Chen [:xeonchen]]

Attachment: 0001-Bug-1321841-better-checksum-for-live-adapters-r-bagd.patch

Comment 20

[Gary Chen [:xeonchen]]

(In reply to Daniel Stenberg [:bagder] from comment #18)
> Created attachment 8827885 [details] [diff] [review]
> 0001-bug-1321841-better-checksum-for-live-adapters.patch
> 
> Argh! Yes I do think it can at least contribute to this problem as the logic
> uses the checksum to avoid signalling if the situation is identical to
> before. Good catch!
> 
> I think the primary mistake is that the left shift is done there for each
> letter in the adapter name, and not just once per adapter. But I also like
> your XOR suggestion. The point of course being that every letter in all
> names of the adapters that are UP need to have the same significance.
> 
> I'd like to suggest a combination: fixing the shift to happen less frequent
> (but letting 'sum' range over all adapters so that we'll use more bits) and
> the XOR.
> 
> Thoughts?

Looks great, let's give it a try.

Comment 21

[Gary Chen [:xeonchen]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ace09413753b0f23b2142a329bef9ef2de2248c9

Please checkin attachment #8827898 [details] [diff] [review]

Comment 22

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/111109712e77

Comment 23

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/111109712e77

Comment 24

[Gary Chen [:xeonchen]]

Hi, the patch has landed to the Nightly build, would you please try if this works?

Comment 25

[Baner]

Hi, i have tested the nightly build 54.0a1 (2017-01-23) (64-bit) and is working after connection and disconnection of my corporate VPN.

Comment 26

[Gary Chen [:xeonchen]]

(In reply to Baner from comment #25)
> Hi, i have tested the nightly build 54.0a1 (2017-01-23) (64-bit) and is
> working after connection and disconnection of my corporate VPN.

Thanks for your report!
Looks like we are almost there.

Comment 27

[Laurens Vets]

Did a quick test with a nightly build. I still seem to be having the same problem. I'll test further.

Comment 28

[Baner]

Ok it only worked a couple of times for me... after several tests is not working after VPN disconnection.

Comment 29

[Gary Chen [:xeonchen]]

(In reply to Laurens Vets from comment #27)
> Did a quick test with a nightly build. I still seem to be having the same
> problem. I'll test further.

(In reply to Baner from comment #28)
> Ok it only worked a couple of times for me... after several tests is not
> working after VPN disconnection.

oh no... :'(

would you follow comment 1 and provide the log with:

MOZ_LOG=timestamp,nsHttp:5,nsSocketTransport:5,nsStreamPump:5,nsHostResolver:5,proxy:5,nsNotifyAddr:5

and attach the file?

Comment 31

[Gary Chen [:xeonchen]]

lower down the priority until we get more information

Comment 32

[Ton]

I have the same problem on:
Oracle Linux 7.3
Firefox 52.1.0 (64-bit)

But in my case I have "Automatic proxy configuration URL: file:///home/ton/wpad.dat" and reload helps me on VPN Connect or Disconnect

Comment 33

[Ton]

Attachment: log.zip

1. Open ya.ru -> OK
2. Connect VPN
3. Open ya.ru -> Nothing
4. Reload Automatic proxy configuration
5. Open ya.ru -> OK
6. Disconnect VPN
7. Open ya.ru -> Nothing
8. Reload Automatic proxy configuration
9. Open ya.ru -> OK

Comment 34

[Ton]

Attachment: examlpe of wpad.dat

Comment 35

[Gary Chen [:xeonchen]]

(In reply to Ton from comment #32)
> I have the same problem on:
> Oracle Linux 7.3
> Firefox 52.1.0 (64-bit)
> 
> But in my case I have "Automatic proxy configuration URL:
> file:///home/ton/wpad.dat" and reload helps me on VPN Connect or Disconnect

Can you help to try Firefox Nightly (or v55+) and see if it's fixed by bug 1337336?

Comment 36

[Gary Chen [:xeonchen]]

We cannot reproduce this bug, no more information from reporters, and it is potentially fixed by bug 1337336, so I'm going to close this.
If anyone still has this issue, please feel free to reopen it.

Comment 37

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.

Comment 38

[wlfbck]

Attachment: log.txt-main.3932.moz_log

Comment 39

[wlfbck]

I have this problem often with AnyConnect and Firefox on my work PC. Interesting part is that Chrome is also affected, but Edge (where i'm typing this right now) is not.

Unfortunately the fix that Baner posted in his OP is not working for me, I have to restart the PC for Firefox and Chrome to work again, obviously not ideal.

Just to be clear, i can access Intranet sites with FF and Chrome, just nothing outside. But Edge can do both just fine.

I uploaded the network log some of the earlier posts requested, i tried connecting to https://www.leo.org/englisch-deutsch but it doesn't really matter which URL, only internal ones work.

If you need anything else just tell :)

Comment 40

[Gary Chen [:xeonchen]]

Assign to :kershaw since I'm not actively working on this.

Comment 41

[k.klewinowski1983]

Is there any progress on bug handling? I have the same problem.

Comment 42

[lallafa]

I have the same problem: after starting a vpn connection with Cisco AnyConnect all connections made by firefox are blocking.

AnyConnect: 4.10.05111
Firefox: 114.0.1
MacOS Ventura 13.4

Toggling proxy settings in firefox does not help (no proxy is used)

Restart of firefox does help

Comment 43

[Kershaw Chang [:kershaw]]

Hi Reporter,

We've recently added a new preference, network.http.http2.move_to_pending_list_after_network_change, aimed at improving how Firefox handles changes in network connections, such as when a VPN is activated. Could you please download the latest Firefox Nightly build, enable this preference, and test again to see if the issue persists?

Thanks.

Comment 44

[k.klewinowski1983]

It still didn't work for me. Firefox 125.0a1 (2024-02-25) (64-bit), Windows 10 Pro

Comment 45

[BugBot [:suhaib / :marco/ :calixte]]

Redirect a needinfo that is pending on an inactive user to the triage owner.
:kershaw, since the bug has recent activity, could you please find another way to get the information or close the bug as INCOMPLETE if it is not actionable?

For more information, please visit BugBot documentation.

Comment 46

[Kershaw Chang [:kershaw]]

Hi Reporter,

Could you try to capture a http log and also file a new bug?

Thanks.