459407 - cannot access mail.mozilla.com while connected to mpt-vpn

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[[e-mail] (:bhearsum)]

same problem as https://bugzilla.mozilla.org/show_bug.cgi?id=422225

Comment 1

[matthew zeier [:mrz]]

Bad arp entry, happened once before but hasn't in a long time and is hard to troubleshoot.

Quick fix was:

[root@cm-vpn01 ~]# arp -d mail.mozilla.com

Comment 2

[Chris AtLee [:catlee]]

Does the arp cache need another kick?  I can't access mail.m.o again.

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

I am also hitting the "can access mail without VPN; cannot access mail with VPN" problem.

+1 to the vote for kicking arp cache...

Comment 4

[matthew zeier [:mrz]]

Yes it does (need arp to be deleted).  Still not sure why this happens.

Comment 5

[Chris AtLee [:catlee]]

Please kick again

Comment 6

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

I'm also having the same problem as catlee. Dropping VPN connection and restarting VPN client didnt fix it for me. Any chance of a VPN kick?

Comment 7

[matthew zeier [:mrz]]

I've been here since 8:30am, haven't had a problem.  Typically you can be certain this is a problem if you login to mpt-vpn and try to ping mail.mozilla.com.

John says this is working after a reconnect though.  I've been connected since my last reboot (days) and it's been working fine.


[root@cm-vpn01 ~]# ping mail.mozilla.com                                                                                
PING mail.mozilla.com (10.2.72.15) 56(84) bytes of data.                                                                
64 bytes from mail.mozilla.com (10.2.72.15): icmp_seq=1 ttl=64 time=0.220 ms

Comment 8

[[e-mail] (:bhearsum)]

This is happening again

Comment 9

[Chris AtLee [:catlee]]

For me as well:
[catlee@cm-vpn01 ~]$ ping -c 5 mail.mozilla.com
PING mail.mozilla.com (10.2.72.15) 56(84) bytes of data.
From v72.fcore1.mozilla.com (10.2.72.1) icmp_seq=1 Destination Net Unreachable
From v72.fcore1.mozilla.com (10.2.72.1) icmp_seq=2 Destination Net Unreachable
From v72.fcore1.mozilla.com (10.2.72.1) icmp_seq=3 Destination Net Unreachable
From v72.fcore1.mozilla.com (10.2.72.1) icmp_seq=4 Destination Net Unreachable
From v72.fcore1.mozilla.com (10.2.72.1) icmp_seq=5 Destination Net Unreachable

--- mail.mozilla.com ping statistics ---
5 packets transmitted, 0 received, +5 errors, 100% packet loss, time 4002ms

Comment 10

[matthew zeier [:mrz]]

Fixed, again.

Comment 11

[matthew zeier [:mrz]]

dave, can you cron something to run every 15/30 mins and send a single icmp packet to mail.mozilla.com and if it fails, do an 'arp -d mail.mozilla.com'?  

I know, sucky fix but I don't have time to focus on why this is breaking.

Ben - you could also hard code mail.mozilla.com to 63.245.208.167.  You don't need to use the VPN to access that host (honestly that's what I've done at work).  Yeah I know, another sucky fix.

Comment 12

[[e-mail] (:bhearsum)]

Yeah, I think I'll put that in my hosts file. Is that something we can fix in DNS, though? Or maybe that would be dumb.

Comment 13

[matthew zeier [:mrz]]

Don't think that will work in DNS because if you're internal to the network you need to talk to the internal address.

Comment 14

[Dave Miller [:justdave]]

*/5 * * * * root ping -c1 mail.mozilla.com > /dev/null || arp -d mail.mozilla.com