1121999 - frequent, intermittent vpn drops

Status: RESOLVED WORKSFORME

(Infrastructure & Operations :: Corporate VPN: Support requests, task)

Description

[[e-mail] (:bhearsum)]

I've been experiencing this for at least a couple of months now. It's not consistent, but it can happen anywhere from 1 to 10 times a day. The drops happen without any notification on my side. Ie, Network Manager still thinks it's connected. This generally shows up as being unable to resolve Mozilla sites (because I can no longer route to the DNS server). Disconnecting and reconnecting to the VPN fix it every time. Here's what little syslog info I have from this morning. The events at 9:30:22 appear to be when the drop happened. 9:37 is when i disconnected/reconnected manually. (Times are all in EST):
Jan 15 09:10:19 dib nm-openvpn[6728]: [openvpn.scl3.mozilla.com] Inactivity timeout (--ping-restart), restarting
Jan 15 09:10:19 dib nm-openvpn[6728]: SIGUSR1[soft,ping-restart] received, process restarting
Jan 15 09:10:21 dib nm-openvpn[6728]: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Jan 15 09:10:21 dib nm-openvpn[6728]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Jan 15 09:10:21 dib nm-openvpn[6728]: UDPv4 link local: [undef]
Jan 15 09:10:21 dib nm-openvpn[6728]: UDPv4 link remote: [AF_INET]63.245.214.137:1194
Jan 15 09:10:23 dib nm-openvpn[6728]: [openvpn.scl3.mozilla.com] Peer Connection Initiated with [AF_INET]63.245.214.137:1194
Jan 15 09:10:26 dib nm-openvpn[6728]: Preserving previous TUN/TAP instance: tun0
Jan 15 09:10:26 dib nm-openvpn[6728]: /usr/lib/NetworkManager/nm-openvpn-service-openvpn-helper tun0 1500 1558 10.22.248.46 10.22.248.45 restart
Jan 15 09:10:26 dib nm-openvpn[6728]: NOTE: Pulled options changed on restart, will need to close and reopen TUN/TAP device.
Jan 15 09:10:27 dib nm-openvpn[6728]: TUN/TAP device tun0 opened
Jan 15 09:10:27 dib nm-openvpn[6728]: /usr/lib/NetworkManager/nm-openvpn-service-openvpn-helper tun0 1500 1558 10.22.248.134 10.22.248.133 init
Jan 15 09:10:27 dib nm-openvpn[6728]: Initialization Sequence Completed
Jan 15 09:16:20 dib nm-openvpn[6728]: SIGTERM[hard,] received, process exiting
Jan 15 09:16:23 dib nm-openvpn[8458]: OpenVPN 2.3.2 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [eurephia] [MH] [IPv6] built on Dec  1 2014
Jan 15 09:16:23 dib nm-openvpn[8458]: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Jan 15 09:16:23 dib nm-openvpn[8458]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Jan 15 09:16:23 dib nm-openvpn[8458]: WARNING: file '/home/bhearsum/vpn/key.key' is group or others accessible
Jan 15 09:16:23 dib nm-openvpn[8458]: WARNING: file '/home/bhearsum/vpn/ta.key' is group or others accessible
Jan 15 09:16:23 dib nm-openvpn[8458]: Control Channel Authentication: using '/home/bhearsum/vpn/ta.key' as a OpenVPN static key file
Jan 15 09:16:23 dib nm-openvpn[8458]: UDPv4 link local: [undef]
Jan 15 09:16:23 dib nm-openvpn[8458]: UDPv4 link remote: [AF_INET]63.245.214.137:1194
Jan 15 09:16:25 dib nm-openvpn[8458]: [openvpn.scl3.mozilla.com] Peer Connection Initiated with [AF_INET]63.245.214.137:1194
Jan 15 09:16:28 dib nm-openvpn[8458]: TUN/TAP device tun0 opened
Jan 15 09:16:28 dib nm-openvpn[8458]: /usr/lib/NetworkManager/nm-openvpn-service-openvpn-helper tun0 1500 1558 10.22.248.138 10.22.248.137 init
Jan 15 09:16:28 dib nm-openvpn[8458]: Initialization Sequence Completed
Jan 15 09:30:22 dib nm-openvpn[8458]: [openvpn.scl3.mozilla.com] Inactivity timeout (--ping-restart), restarting
Jan 15 09:30:22 dib nm-openvpn[8458]: SIGUSR1[soft,ping-restart] received, process restarting
Jan 15 09:30:24 dib nm-openvpn[8458]: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Jan 15 09:30:24 dib nm-openvpn[8458]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Jan 15 09:30:24 dib nm-openvpn[8458]: UDPv4 link local: [undef]
Jan 15 09:30:24 dib nm-openvpn[8458]: UDPv4 link remote: [AF_INET]63.245.214.137:1194
Jan 15 09:30:26 dib nm-openvpn[8458]: [openvpn.scl3.mozilla.com] Peer Connection Initiated with [AF_INET]63.245.214.137:1194
Jan 15 09:30:28 dib nm-openvpn[8458]: Preserving previous TUN/TAP instance: tun0
Jan 15 09:30:28 dib nm-openvpn[8458]: /usr/lib/NetworkManager/nm-openvpn-service-openvpn-helper tun0 1500 1558 10.22.248.138 10.22.248.137 restart
Jan 15 09:30:28 dib nm-openvpn[8458]: NOTE: Pulled options changed on restart, will need to close and reopen TUN/TAP device.
Jan 15 09:30:29 dib nm-openvpn[8458]: TUN/TAP device tun0 opened
Jan 15 09:30:29 dib nm-openvpn[8458]: /usr/lib/NetworkManager/nm-openvpn-service-openvpn-helper tun0 1500 1558 10.22.248.14 10.22.248.13 init
Jan 15 09:30:29 dib nm-openvpn[8458]: Initialization Sequence Completed
Jan 15 09:37:18 dib nm-openvpn[8458]: SIGTERM[hard,] received, process exiting
Jan 15 09:37:20 dib nm-openvpn[8910]: OpenVPN 2.3.2 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [eurephia] [MH] [IPv6] built on Dec  1 2014
Jan 15 09:37:21 dib nm-openvpn[8910]: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Jan 15 09:37:21 dib nm-openvpn[8910]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Jan 15 09:37:21 dib nm-openvpn[8910]: WARNING: file '/home/bhearsum/vpn/key.key' is group or others accessible
Jan 15 09:37:21 dib nm-openvpn[8910]: WARNING: file '/home/bhearsum/vpn/ta.key' is group or others accessible
Jan 15 09:37:21 dib nm-openvpn[8910]: Control Channel Authentication: using '/home/bhearsum/vpn/ta.key' as a OpenVPN static key file
Jan 15 09:37:21 dib nm-openvpn[8910]: UDPv4 link local: [undef]
Jan 15 09:37:21 dib nm-openvpn[8910]: UDPv4 link remote: [AF_INET]63.245.214.137:1194
Jan 15 09:37:23 dib nm-openvpn[8910]: [openvpn.scl3.mozilla.com] Peer Connection Initiated with [AF_INET]63.245.214.137:1194
Jan 15 09:37:25 dib nm-openvpn[8910]: TUN/TAP device tun0 opened
Jan 15 09:37:25 dib nm-openvpn[8910]: /usr/lib/NetworkManager/nm-openvpn-service-openvpn-helper tun0 1500 1558 10.22.248.138 10.22.248.137 init
Jan 15 09:37:25 dib nm-openvpn[8910]: Initialization Sequence Completed


I know Chris and Rail have been hitting this as well.

Comment 1

[Dan Parsons [:lerxst]]

Can you show me your openvpn config?

Comment 2

[[e-mail] (:bhearsum)]

Here's what I get when I export it from NetworkManager:
client
remote 63.245.214.137
ca /home/bhearsum/vpn/ca.crt
cert /home/bhearsum/vpn/cert.crt
key /home/bhearsum/vpn/key.key
auth-user-pass
cipher AES-256-CBC
comp-lzo yes
dev tun
proto udp
tls-auth /home/bhearsum/vpn/ta.key 1
nobind
auth-nocache
script-security 2
persist-key
persist-tun
user nobody
group nogroup

Comment 3

[Dan Parsons [:lerxst]]

Can you please try adding the following line to your config and retesting?:

reneg-sec 0

Also, change "comp-lzo" to "no" - I doubt it makes much of a difference in any regard, but my config has it set to no and I've seen "yes" cause bizarre problems before.

Comment 4

[[e-mail] (:bhearsum)]

(In reply to Dan Parsons [:lerxst] from comment #3)
> Can you please try adding the following line to your config and retesting?:
> 
> reneg-sec 0
> 
> Also, change "comp-lzo" to "no" - I doubt it makes much of a difference in
> any regard, but my config has it set to no and I've seen "yes" cause bizarre
> problems before.

I just added that, I'll see how it goes next week.

Comment 5

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #4)
> (In reply to Dan Parsons [:lerxst] from comment #3)
> > Can you please try adding the following line to your config and retesting?:
> > 
> > reneg-sec 0
> > 
> > Also, change "comp-lzo" to "no" - I doubt it makes much of a difference in
> > any regard, but my config has it set to no and I've seen "yes" cause bizarre
> > problems before.
> 
> I just added that, I'll see how it goes next week.

Unfortunately, it didn't help.

Comment 6

[Guillaume Destuynder [:kang] [this account is no longer active]]

as per https://www.yammer.com/mozilla.com/#/Threads/show?threadId=486619113 switching to TCP seems to help. It looks like the UDP packets are dropped after some time, or too many are lost or something like that.

Looking at logs it seems like there has been no renegotiation started and the current session is the one that time out (detected by the keep-alive). In addition to that, Network Manager doesn't auto-reconnect (it doesn't support that feature for openvpn yet AFAIK).

There's some work-arounds when auto-reconnect is needed such as https://bbs.archlinux.org/viewtopic.php?id=107509 (albeit it's not all that useful when/if using MFA since it will ask for the 2nd authentication token upon reconnect)

Comment 7

[[e-mail] (:bhearsum)]

(In reply to Guillaume Destuynder [:kang] from comment #6)
> as per https://www.yammer.com/mozilla.com/#/Threads/show?threadId=486619113
> switching to TCP seems to help. It looks like the UDP packets are dropped
> after some time, or too many are lost or something like that.

I got my first drop with TCP today actually :(. I'm willing to write that off as a freak occurence if it doesn't happen again though.

Comment 8

[Justin Dow [:jabba]]

Is this still occurring? We've had a few changes and updates since the last comment on this bug. I personally cannot reproduce, as I regularly go 30 days on a single connection (and then only get dropped because I need to re-auth with MFA).

Comment 9

[[e-mail] (:bhearsum)]

I've been off for a bit so it's hard to say. I'll have a better idea in a few weeks time. Let's resolve for now and re-open if it continues to be an issue...