848012 - Please grant access to RelEng VPN

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Pete Moore [:pmoore][:pete]]

Hi,

Please can you grant me access to the RelEng VPN (vpn1.releng.scl3.mozilla.com:1194).

Many thanks,
Pete

Comment 1

[Dumitru Gherman [:dumitru]]

Coop, John, ack?

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

We need a releng ack first, then no problem!

Comment 3

[Aki Sasaki (not active)]

Peter's a Releng new hire.  Ack.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

done.  It takes a bit (like an hour) to propagate.

Comment 5

[Pete Moore [:pmoore][:pete]]

Many thanks Dustin!

Comment 6

[Pete Moore [:pmoore][:pete]]

Hi guys,

My access seems to have been revoked - I cannot access now - it was working before.

I also got this email within a short time of my access disappearing (I was connected this morning, then it dropped, and now I cannot get back on):




On Mar 14, 2013, at 11:07 AM, root <[e-mail]> wrote:

mail=[e-mail],o=net,dc=mozilla
mail=[e-mail],o=com,dc=mozilla
mail=[e-mail],o=com,dc=mozilla
mail=[e-mail],o=com,dc=mozilla
mail=[e-mail],o=com,dc=mozilla
mail=[e-mail],o=com,dc=mozilla
mail=[e-mail],o=com,dc=mozilla
mail=[e-mail],o=com,dc=mozilla
mail=[e-mail],o=com,dc=mozilla
mail=[e-mail],o=com,dc=mozilla
mail=[e-mail],o=com,dc=mozilla
mail=[e-mail],o=com,dc=mozilla
mail=[e-mail],o=com,dc=mozilla
mail=[e-mail],o=com,dc=mozilla
mail=[e-mail],o=com,dc=mozilla
mail=[e-mail],o=com,dc=mozilla
mail=[e-mail],o=net,dc=mozilla
mail=[e-mail],o=com,dc=mozilla
mail=[e-mail],o=com,dc=mozilla
mail=[e-mail],o=com,dc=mozilla



Please note my account is not listed in this email above. It was listed in the email last week (from 7 March) so it looks like something has changed.

Please also note, Simone Bruno is having the same problem.

Many thanks,
Pete

Comment 7

[Amy Rich [:arr] [:arich]]

You both still have access to the VPN according to LDAP.  You disappeared form that list because you're now a member of the releng group and that only lists non-releng and non-sysadmins.

What error do you get when you try to connect to the vpn?

Comment 8

[Pete Moore [:pmoore][:pete]]

Hi Amy,

Thanks for looking into that. It looks like Simone's access is working again now - but I still have the same problem. My log file from Viscosity looks like this:


Mar 14 16:54:47: Viscosity Mac 1.4.3 (1114)
Mar 14 16:54:47: Viscosity OpenVPN Engine Started
Mar 14 16:54:47: Running on Mac OS X 10.8.2
Mar 14 16:54:47: ---------
Mar 14 16:54:47: Checking reachability status of connection...
Mar 14 16:54:47: Connection is reachable. Starting connection attempt.
Mar 14 16:54:49: OpenVPN 2.2.1 x86_64-apple-darwin10.8.0 [SSL] [LZO2] [PKCS11] [eurephia] built on Aug  1 2011
Mar 14 16:54:51: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Mar 14 16:54:51: NOTE: OpenVPN 2.1 requires '--script-security 2' or higher to call user-defined scripts or executables
Mar 14 16:54:51: Control Channel Authentication: using 'ta.key' as a OpenVPN static key file
Mar 14 16:54:51: LZO compression initialized
Mar 14 16:54:51: UDPv4 link local: [undef]
Mar 14 16:54:51: UDPv4 link remote: 63.245.214.124:1194
Mar 14 16:55:52: TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Mar 14 16:55:52: TLS Error: TLS handshake failed
Mar 14 16:55:52: SIGUSR1[soft,tls-error] received, process restarting
Mar 14 16:56:02: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Mar 14 16:56:02: NOTE: OpenVPN 2.1 requires '--script-security 2' or higher to call user-defined scripts or executables
Mar 14 16:56:02: Re-using SSL/TLS context
Mar 14 16:56:02: LZO compression initialized
Mar 14 16:56:02: UDPv4 link local: [undef]
Mar 14 16:56:02: UDPv4 link remote: 63.245.214.124:1194
Mar 14 16:57:02: TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Mar 14 16:57:02: TLS Error: TLS handshake failed
Mar 14 16:57:02: SIGUSR1[soft,tls-error] received, process restarting
Mar 14 16:57:13: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Mar 14 16:57:13: NOTE: OpenVPN 2.1 requires '--script-security 2' or higher to call user-defined scripts or executables
Mar 14 16:57:13: Re-using SSL/TLS context
Mar 14 16:57:13: LZO compression initialized
Mar 14 16:57:13: UDPv4 link local: [undef]
Mar 14 16:57:13: UDPv4 link remote: 63.245.214.124:1194
Mar 14 16:58:14: TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Mar 14 16:58:14: TLS Error: TLS handshake failed
Mar 14 16:58:14: SIGUSR1[soft,tls-error] received, process restarting
Mar 14 16:58:24: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
Mar 14 16:58:24: NOTE: OpenVPN 2.1 requires '--script-security 2' or higher to call user-defined scripts or executables
Mar 14 16:58:24: Re-using SSL/TLS context
Mar 14 16:58:24: LZO compression initialized
Mar 14 16:58:24: UDPv4 link local: [undef]
Mar 14 16:58:24: UDPv4 link remote: 63.245.214.124:1194

Thanks,
Pete

Comment 9

[Amy Rich [:arr] [:arich]]

Can you ssh to vpn1.releng.scl3.mozilla.com successfully?

Comment 10

[Pete Moore [:pmoore][:pete]]

Hi Amy,

Yes that seems ok:

pmoore@Peters-MacBook-Pro-2:~ $ ssh vpn1.releng.scl3.mozilla.com
The authenticity of host 'vpn1.releng.scl3.mozilla.com (63.245.214.124)' can't be established.
RSA key fingerprint is c4:e1:71:61:a6:cf:61:47:a4:07:15:82:b2:a8:5e:85.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'vpn1.releng.scl3.mozilla.com,63.245.214.124' (RSA) to the list of known hosts.
[[e-mail]3 ~]$ 

I just checked again (to make sure the connection problem still exists), and I can confirm there is still the same problem with the TLS key negotiation when attempting to connect to the RelEng VPN using Viscosity.

In order to help troubleshoot, I also tried a different VPN profile (Mozilla-MPT-Office) and I have the same problem there - so not just the RelEng VPN but it looks like a general VPN connectivity problem.

Thanks,
Pete

Comment 11

[Amy Rich [:arr] [:arich]]

If you can ssh to the vpn, that points to a configuration problem with openvpn on your local desktop/laptop.

Comment 12

[Pete Moore [:pmoore][:pete]]

Thanks Amy. I'm not aware that anything has changed there since this morning when it was working - although of course anything is possible. Does this mean the only technical requirement for the VPN connection to work is that I have ssh connectivity to that box? Is there no other server-side configuration that could be related?

If there is any further logging or troubleshooting information I can provide that might be useful, please let me know.

I also think it may not be the VPN configuration for the RelEng profile, since more than one profile is not working, which suggests something outside of the individual VPN profiles (although this could of course be a general Viscosity setting, but perhaps not so likely).

Another possibility might be a firewall issue - do you know which network flows (ports, protocols) are involved in the TLS handshake? I could check whether this is a firewall issue somewhere between here and there.

Many thanks for your support.

Kind regards,
Pete

Comment 13

[Justin Dow [:jabba]]

Our VPN servers are configured to listen on UDP port 1194. Are you able to verify that you have outgoing connectivity for that port? Since the releng and the mpt-vpn are in different data centers with different server side configs (same client configs but different endpoint IP), and neither is working I think it must be a problem on your end, since I haven't seen any issues with other users, and I see many users connected to both mpt-vpn and releng vpn right now.

Comment 14

[Pete Moore [:pmoore][:pete]]

Hi guys,

It looks like I do have connectivity on UDP port 1194:

pmoore@Peters-MacBook-Pro-2:~ $ nc -vzu vpn1.releng.scl3.mozilla.com 1194
Connection to vpn1.releng.scl3.mozilla.com 1194 port [udp/openvpn] succeeded!
pmoore@Peters-MacBook-Pro-2:~ $

I will see if I can get help from the desktop support team, as it looks like some kind of client problem.

Thanks,
Pete

Comment 15

[Pete Moore [:pmoore][:pete]]

I also tried replacing the name (vpn1.releng.scl3.mozilla.com) with IP (63.245.214.124) to see if it might be a problem with DNS resolution, but that also did not help.

Not sure what to try next...

Comment 16

[Pete Moore [:pmoore][:pete]]

So I think it must have been a problem with the ISP dropping packets. Nothing changed on my end, but now working again today.

This was the traceroute from yesterday:


pmoore@Peters-MacBook-Pro-2:~ $ traceroute vpn1.releng.scl3.mozilla.com
traceroute to vpn1.releng.scl3.mozilla.com (63.245.214.124), 64 hops max, 52 byte packets
 1  speedport.ip (192.168.2.1)  0.692 ms  0.421 ms  0.397 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
31  * * *
32  * * *
33  * * *
34  * * *
35  * * *
36  * * *
37  * * *
38  * * *
39  * * *
40  * * *
41  * * *
42  * * *
43  * * *
44  * * *
45  * * *
46  * * *
47  * * *
48  * * *
49  * * *
50  * * *
51  * * *
52  * * *
53  * * *
54  * * *
55  * * *
56  * * *
57  * * *
58  * * *
59  * * *
60  * * *
61  * * *
62  * * *
63  * * *
64  * * *


This is the traceroute from today:


pmoore@Peters-MacBook-Pro-2:~ $ traceroute vpn1.releng.scl3.mozilla.com
traceroute to vpn1.releng.scl3.mozilla.com (63.245.214.124), 64 hops max, 52 byte packets
 1  192.168.2.1 (192.168.2.1)  0.760 ms  0.450 ms  0.432 ms
 2  217.0.118.253 (217.0.118.253)  17.671 ms  17.136 ms *
 3  87.186.254.22 (87.186.254.22)  20.749 ms  17.356 ms  19.820 ms
 4  hh-ea4-i.hh.de.net.dtag.de (62.154.33.34)  24.753 ms  25.439 ms  25.187 ms
 5  80.156.160.242 (80.156.160.242)  23.790 ms  24.714 ms
    80.150.168.162 (80.150.168.162)  24.142 ms
 6  hbg-bb2-link.telia.net (213.155.135.84)  24.058 ms
    hbg-bb2-link.telia.net (213.155.135.86)  25.921 ms
    hbg-bb1-link.telia.net (213.155.135.82)  24.497 ms
 7  nyk-bb1-link.telia.net (80.91.247.127)  101.229 ms  101.435 ms
    ash-bb4-link.telia.net (213.155.131.251)  110.124 ms
 8  sjo-bb1-link.telia.net (213.155.135.157)  184.862 ms
    sjo-bb1-link.telia.net (80.91.253.68)  185.228 ms
    sjo-bb1-link.telia.net (80.91.245.96)  179.484 ms
 9  mozilla-ic-155747-sjo-bb1.c.telia.net (62.115.8.162)  178.838 ms  185.275 ms  183.763 ms
10  xe-0-0-1.border2.scl3.mozilla.net (63.245.219.162)  189.707 ms  236.943 ms  185.645 ms
11  v-1032.core1.releng.scl3.mozilla.net (63.245.214.90)  181.403 ms  186.332 ms  183.891 ms
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
31  * * *
32  * * *
33  * * *
34  * * *
35  * * *
36  * * *
37  * * *
38  * * *
39  * * *
40  * * *
41  * * *
42  * * *
43  * * *
44  * * *
45  * * *
46  * * *
47  * * *
48  * * *
49  * * *
50  * * *
51  * * *
52  * * *
53  * * *
54  * * *
55  * * *
56  * * *
57  * * *
58  * * *
59  * * *
60  * * *
61  * * *
62  * * *
63  * * *
64  * * *


So all resolved.

Thanks for all your help with this. I guess one of those mysteries whose source will always remain unknown. :)

Thanks,
Pete