Firefox network loss after connection to Cisco VPN.

RESOLVED INCOMPLETE

Status

()

defect
RESOLVED INCOMPLETE
2 years ago
a year ago

People

(Reporter: jdsandoval1809, Assigned: xeonchen)

Tracking

50 Branch
x86_64
Windows 7
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [necko-next][proxy])

Attachments

(6 attachments, 2 obsolete attachments)

(Reporter)

Description

2 years ago
User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0
Build ID: 20161129173726

Steps to reproduce:

Normally browsing internet, and  connect to corporate VPN using Cisco AnyConnect Secure Mobility Client Ver: 3.1.11004


Actual results:

Firefox Connection is lost, and the only way to get it working again is:

1: Restart Firefox.
2: Goto  settings - advance - Network - connection-setting -  and then 'toggle' between "Auto detect proxy settings" & "Use system proxy settings".


Expected results:

Firefox should keep connected.
(Reporter)

Updated

2 years ago
OS: Unspecified → Windows 7
Hardware: Unspecified → x86_64

Updated

2 years ago
Component: Untriaged → Networking
Product: Firefox → Core
Can you make a Http log:
https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging

Just change line:
set MOZ_LOG=timestamp,rotate:200,nsHttp:5,nsSocketTransport:5,nsStreamPump:5,nsHostResolver:5
into:
set MOZ_LOG=timestamp,nsHttp:5,nsSocketTransport:5,nsStreamPump:5,nsHostResolver:5

Thanks!
Flags: needinfo?(jdsandoval1809)
Whiteboard: [necko-active]
(Reporter)

Comment 2

2 years ago
Posted file log.zip
Open Youtube.com - OK
Connect to corporate VPN
Open Youtube.com - OK
Disconnect from VPN
Connect to Youtube.com - Not OK
Goto  Options - Advanced - Network - Settings -  and then 'toggle' between "Auto detect proxy settings" & "Use system proxy settings".
Open Youtube.com - OK
Flags: needinfo?(jdsandoval1809)
Duplicate of this bug: 1324879
Assignee: nobody → dd.mozilla
From log, I can see that we still return a proxy with OnProxyAvailable even after the vpn is disconnected. Looking at our proxyservice we do query the system for every connection, no caching (I have not look at this code for a very long time maybe I am wrong), and we query the windows system using system function InternetGetConnectedStateExW and according to the response we set up a proxy connection or not. I do not have a Windows machine to try it and we only have pac logging but not windows specific logging. To be more certain we will need proxy log  as well.

Baner, can you make the same log with:
set MOZ_LOG=timestamp,nsHttp:5,nsSocketTransport:5,nsStreamPump:5,nsHostResolver:5,proxy:5

Daniel, you looked at the proxy/pac code recently. Am I correct here?
Flags: needinfo?(jdsandoval1809)
Flags: needinfo?(daniel)
This problem with Cisco AnyConnect VPN seems to have been around since basically forever. See Bug 651333 for what sounds like exactly the same symptoms (and a few others through history since)

It sounds like your description matches what we've seen before Dragana.

So does Firefox always use a proxy when AnyConnect is active? I would expect a VPN to mostly work on network interface level.

I think we really need someone in the necko team to setup and run Cisco AnyConnect on a test machine and see what exactly is going on so that we can finally nail this long-going nuisance to users.
Flags: needinfo?(daniel)
(Assignee)

Updated

2 years ago
Duplicate of this bug: 651333
(Assignee)

Comment 7

2 years ago
(In reply to Daniel Stenberg [:bagder] from comment #5)
> I think we really need someone in the necko team to setup and run Cisco
> AnyConnect on a test machine and see what exactly is going on so that we can
> finally nail this long-going nuisance to users.

I'm going to set up the environment soon.
Assignee: dd.mozilla → xeonchen

Updated

2 years ago
Whiteboard: [necko-active] → [necko-active][proxy]
(Assignee)

Comment 8

2 years ago
Posted file log.txt.zip
I'm using:

- Windows 7 SP1 x64
- Firefox 50.1.0 x86
- Firefox 50.1.0 x64
- Cisco AnyConnect Secure Mobility Client 3.1.13015

to test, but I can't reproduce by following steps:

1. open Firefox
2. open [1], success
3. connect AnyConnect to VPN
3.1 I can see "route print" command under prompt adds a new default route with lower metric
4. open [2], success
5. disconnect VPN
6. open [3], still success

Tried following proxy settings:
- Auto-detect proxy setting for this network
- Use system proxy settings (which uses a remote PAC file)

Not sure if it needs a more complicated network environment because I set up the VPN server in the same LAN.

[1] http://example.com/
[2] http://www.ipchicken.com/
[3] http://www.google.com.tw/
According to bug 651333 comment 3, the problem is related to system proxy setting been changed by VPN disconnection.  Did you encounter the same situation?
(Assignee)

Comment 10

2 years ago
(In reply to Shian-Yow Wu [:swu] from comment #9)
> According to bug 651333 comment 3, the problem is related to system proxy
> setting been changed by VPN disconnection.  Did you encounter the same
> situation?

I missed that comment, thank you swu.
See Also: → 1207436
(Assignee)

Comment 11

2 years ago
OK, I just made following modification, still cannot reproduce.

- Client
  - Network 1 is wired network
    - IP: 10.0.0.2
    - Proxy: 10.0.0.1
  - Network 2 is VPN network
    - IP: 192.168.1.2
    - Proxy: 192.168.1.1

1. Connect to VPN, open Wireshark to observe network 2
   -> http traffic is through 192.168.1.1
2. Disconnect from VPN, open Wireshark to observe network 1
   -> http traffic is through 10.0.0.1 now

Both Chrome 55.0.2883.87m and Firefox 50.1.0 works.

Comment 12

2 years ago
I will try and get a log file this weekend. Chrome always works for me.

Steps I do:
1. Connect corporate VPN.
2. Surf in Firefox.
3. VPN disconnects or reconnects because of a network issue.
4. Surfing doesn't work anymore in Firefox.

Comment 13

2 years ago
Ok, I've made a HTTP log.

Cisco AnyConnect client 3.1.14018
Firefox 50.1.0
System proxy set to corporate proxy

Steps to reproduce:

1. Corporate VPN Client is connected.
2. Open Firefox, homepage is https://www.google.com/ -> ok.
3. Surf to www.slashdot.org -> ok.
5. Disconnect & reconnect VPN
6. Click link on open slashdot page -> "Unable to connect. Firefox can’t establish a connection to the server at games.slashdot.org."
7. Try and surf to www.tweakers.net -> Same error message.
8. Try and surf to www.cnn.com -> Same error message.
9. Surf to www.hackaday.com in a new tab -> Same error message.
10. Implement workaround (Go to settings - advance - Network - connection-setting -  and then 'toggle' between "Auto detect proxy settings" & "Use system proxy settings".)
11. Refreshing www.hackaday.com now works.

Anything else I can provide?

Comment 14

2 years ago
Posted file log_20170113.zip
(Assignee)

Comment 15

2 years ago
Hi Laurens,

Would you tell me your proxy setting type (auto-detect, use system, ...etc) before/after connected to VPN?
Flags: needinfo?(laurens)

Comment 16

2 years ago
In Firefox: "Use system proxy settings"

Under LAN Settings in Windows 7: "Use automatic configuration script" with the Address filled in ("http://pac.domain.com/standard.pac").
"Automatically detect settings" is ticked off as well as "Use a proxy server for yourLAN (..."

As far as I know, this is default setting here and I don't manually change it beofre/during or after VPN reconnection.
Flags: needinfo?(laurens)
(Assignee)

Comment 17

2 years ago
I still don't reproduce the bug, but I found a possible root cause.

|NetworkChanged| is called only the first time VPN connected or disconnected because the variable |sum|, which is used to calculate checksum of network interfaces, will overflow when there are more than one interfaces because of the left shift operation.
The value of |sum| is always the same whether the VPN is connected or not.

Daniel, do you think it could be the root cause and it's worthy to land this patch and see if the bug is fixed?
Attachment #8827870 - Flags: feedback?(daniel)
Argh! Yes I do think it can at least contribute to this problem as the logic uses the checksum to avoid signalling if the situation is identical to before. Good catch!

I think the primary mistake is that the left shift is done there for each letter in the adapter name, and not just once per adapter. But I also like your XOR suggestion. The point of course being that every letter in all names of the adapters that are UP need to have the same significance.

I'd like to suggest a combination: fixing the shift to happen less frequent (but letting 'sum' range over all adapters so that we'll use more bits) and the XOR.

Thoughts?
Attachment #8827870 - Flags: feedback?(daniel) → feedback+
(Assignee)

Comment 19

2 years ago
Attachment #8827870 - Attachment is obsolete: true
Attachment #8827898 - Flags: review?(daniel)
(Assignee)

Comment 20

2 years ago
(In reply to Daniel Stenberg [:bagder] from comment #18)
> Created attachment 8827885 [details] [diff] [review]
> 0001-bug-1321841-better-checksum-for-live-adapters.patch
> 
> Argh! Yes I do think it can at least contribute to this problem as the logic
> uses the checksum to avoid signalling if the situation is identical to
> before. Good catch!
> 
> I think the primary mistake is that the left shift is done there for each
> letter in the adapter name, and not just once per adapter. But I also like
> your XOR suggestion. The point of course being that every letter in all
> names of the adapters that are UP need to have the same significance.
> 
> I'd like to suggest a combination: fixing the shift to happen less frequent
> (but letting 'sum' range over all adapters so that we'll use more bits) and
> the XOR.
> 
> Thoughts?

Looks great, let's give it a try.
Attachment #8827898 - Flags: review?(daniel) → review+
(Assignee)

Updated

2 years ago
Attachment #8827885 - Attachment is obsolete: true
(Assignee)

Comment 24

2 years ago
Hi, the patch has landed to the Nightly build, would you please try if this works?
Flags: needinfo?(laurens)
(Reporter)

Comment 25

2 years ago
Hi, i have tested the nightly build 54.0a1 (2017-01-23) (64-bit) and is working after connection and disconnection of my corporate VPN.
Flags: needinfo?(jdsandoval1809)
(Assignee)

Comment 26

2 years ago
(In reply to Baner from comment #25)
> Hi, i have tested the nightly build 54.0a1 (2017-01-23) (64-bit) and is
> working after connection and disconnection of my corporate VPN.

Thanks for your report!
Looks like we are almost there.

Comment 27

2 years ago
Did a quick test with a nightly build. I still seem to be having the same problem. I'll test further.
(Reporter)

Comment 28

2 years ago
Ok it only worked a couple of times for me... after several tests is not working after VPN disconnection.
(Assignee)

Comment 29

2 years ago
(In reply to Laurens Vets from comment #27)
> Did a quick test with a nightly build. I still seem to be having the same
> problem. I'll test further.

(In reply to Baner from comment #28)
> Ok it only worked a couple of times for me... after several tests is not
> working after VPN disconnection.

oh no... :'(

would you follow comment 1 and provide the log with:

MOZ_LOG=timestamp,nsHttp:5,nsSocketTransport:5,nsStreamPump:5,nsHostResolver:5,proxy:5,nsNotifyAddr:5

and attach the file?
(Assignee)

Updated

2 years ago
See Also: → 1337336
See Also: → 1284378
Duplicate of this bug: 1339761
(Assignee)

Comment 31

2 years ago
lower down the priority until we get more information
Whiteboard: [necko-active][proxy] → [necko-next][proxy]

Comment 32

2 years ago
I have the same problem on:
Oracle Linux 7.3
Firefox 52.1.0 (64-bit)

But in my case I have "Automatic proxy configuration URL: file:///home/ton/wpad.dat" and reload helps me on VPN Connect or Disconnect

Comment 33

2 years ago
Posted file log.zip
1. Open ya.ru -> OK
2. Connect VPN
3. Open ya.ru -> Nothing
4. Reload Automatic proxy configuration
5. Open ya.ru -> OK
6. Disconnect VPN
7. Open ya.ru -> Nothing
8. Reload Automatic proxy configuration
9. Open ya.ru -> OK

Comment 34

2 years ago
Posted file examlpe of wpad.dat
(Assignee)

Comment 35

2 years ago
(In reply to Ton from comment #32)
> I have the same problem on:
> Oracle Linux 7.3
> Firefox 52.1.0 (64-bit)
> 
> But in my case I have "Automatic proxy configuration URL:
> file:///home/ton/wpad.dat" and reload helps me on VPN Connect or Disconnect

Can you help to try Firefox Nightly (or v55+) and see if it's fixed by bug 1337336?
Flags: needinfo?(66Ton99)
(Assignee)

Comment 36

2 years ago
We cannot reproduce this bug, no more information from reporters, and it is potentially fixed by bug 1337336, so I'm going to close this.
If anyone still has this issue, please feel free to reopen it.
Status: UNCONFIRMED → RESOLVED
Last Resolved: 2 years ago
Flags: needinfo?(laurens)
Flags: needinfo?(66Ton99)
Resolution: --- → INCOMPLETE
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
You need to log in before you can comment on or make changes to this bug.