Closed
Bug 1270322
Opened 8 years ago
Closed 7 years ago
Can't access google from Beijing office
Categories
(Infrastructure & Operations Graveyard :: NetOps: Office Other, task)
Infrastructure & Operations Graveyard
NetOps: Office Other
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: jxia, Unassigned)
Details
Hi,
www.google.com is unreachable from Beijing office, some informations as below:
1. DNS
jxia@ithp:~$ nslookup
> www.google.com
Server: 10.240.24.2
Address: 10.240.24.2#53
Non-authoritative answer:
Name: www.google.com
Address: 216.58.192.4
2. Ping status
jxia@ithp:~$ ping www.google.com
PING www.google.com (216.58.192.4) 56(84) bytes of data.
^C
--- www.google.com ping statistics ---
37 packets transmitted, 0 received, 100% packet loss, time 35995ms
3. tracert info
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 10.240.24.2 0.0% 74 0.6 0.6 0.5 1.3 0.1
2. ???
3. 10.2.22.1 0.0% 74 1.7 2.5 1.5 12.3 1.4
4. 10.2.2.57 0.0% 74 3.0 3.4 1.4 29.4 5.0
5. 119.253.5.133 1.4% 74 1.5 2.5 1.4 19.4 2.2
6. 119.253.4.85 4.1% 74 2.7 2.7 1.5 21.2 2.5
7. 119.253.5.86 0.0% 74 2.1 3.5 1.5 42.8 5.4
8. 192.168.51.73 0.0% 74 3.5 2.9 1.6 27.3 3.7
9. 159.226.253.109 0.0% 74 3.5 3.4 1.6 31.1 4.6
10. 159.226.253.50 0.0% 74 2.6 3.5 1.8 30.4 4.6
11. ???
4. compare with tracert info of smtp.gmail.com shown below, it seems that access www.gmail.com didn't go through the ipsec tunnel,
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 10.240.24.2 0.0% 2 0.5 0.5 0.5 0.5 0.0
2. 10.0.24.129 0.0% 2 4.3 4.9 4.3 5.4 0.8
3. ???
4. 10.0.24.10 0.0% 2 166.8 166.6 166.5 166.8 0.2
5. 63.245.219.169 0.0% 2 166.4 166.8 166.4 167.1 0.5
6. 67.92.171.49 0.0% 2 167.3 167.2 167.2 167.3 0.1
7. 207.88.13.224 0.0% 2 167.7 173.9 167.7 180.0 8.7
8. 207.88.13.229 0.0% 2 196.8 182.3 167.7 196.8 20.6
9. 216.156.84.30 0.0% 2 168.1 168.2 168.1 168.2 0.1
10. 209.85.249.5 0.0% 2 175.8 172.7 169.6 175.8 4.4
11. 209.85.246.38 0.0% 2 171.5 170.5 169.5 171.5 1.4
12. 216.239.49.198 0.0% 2 186.2 186.2 186.2 186.2 0.0
13. 74.125.37.208 0.0% 2 191.1 191.1 191.1 191.1 0.0
14. 74.125.37.219 0.0% 2 194.7 194.7 194.7 194.7 0.0
15. ???
16. 74.125.28.109 0.0% 1 189.2 189.2 189.2 189.2 0.0
|
||
Comment 1•8 years ago
|
||
You get to google.com via your provider in Beijing. You get to gmail via Mozilla's network. The output of your traceroutes confirms this. Please contact your provider in Beijing regarding this matter. If we can help further, do not hesitate to re-open this bug. Thank you.
Assignee: network-operations → dcurado
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → INVALID
|
Reporter | |
Comment 2•8 years ago
|
||
We can't access google sites via local provider in China, this is one of the reasons why route google sites to Mozilla's network. Could you help to check why the ip of google.com/www.google.com not be routed to Mozilla's network？ Thanks.
Status: RESOLVED → REOPENED
Resolution: INVALID → ---
|
|
||
Comment 3•8 years ago
|
||
Ah, OK, I think I misread your request. Currently we send packets to some parts of Google over Mozilla's network. To date, packets to google.com have not been routed that way. But now it looks like Google itself is accepting packets from your provider's IP space. Your traceroutes to google.com actually make it to the US, but then fail, as if google isn't carrying the routes for your provider's IP space. I wonder why that would be. From your original comment, www.google.com resolves to 216.58.192.4 for you. The CIDR block google announces for that IP range is 216.58.192.0/19. I will add a route for that, so that it's like what we do for gmail. Hopefully that will fix this for you. I have made this change. Please let me know if this helped? Thanks - and apologies for my confusion over your request.
Status: REOPENED → UNCONFIRMED
Ever confirmed: false
Flags: needinfo?(jxia)
|
|
Reporter | |
Comment 4•8 years ago
|
||
Thanks Dave, but we still can't access google.com/www.google.com, here are some informations： 1.DNS: Both www.google.com/google.com resolves to an ip which in 216.58.192.0/19 subnet, so it should be routed to the Mozilla's network. > www.google.com Server: 10.240.24.2 Address: 10.240.24.2#53 Non-authoritative answer: Name: www.google.com Address: 216.58.194.196 > google.com Server: 10.240.24.2 Address: 10.240.24.2#53 Non-authoritative answer: Name: google.com Address: 216.58.194.206 2. Tracert: got no response, even no response from 10.240.24.2 root@ithp:/home/jxia# traceroute 216.58.194.196 traceroute to 216.58.194.196 (216.58.194.196), 30 hops max, 60 byte packets 1 * * * 2 * * * 3 * * * 4 * * * 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * 3. ping root@ithp:/home/jxia# ping 216.58.194.196 PING 216.58.194.196 (216.58.194.196) 56(84) bytes of data. ^C --- 216.58.194.196 ping statistics --- 69 packets transmitted, 0 received, 100% packet loss, time 68542ms
Flags: needinfo?(jxia)
|
|
||
Comment 5•8 years ago
|
||
Hm. OK, sorry about that. I dug around and found there is security policy on the firewall in the PEK2 office which needs to allow packets for gmail (and now www.google.com) out via the PEK1 office. I've added the new prefix (216.58.192.0/19) to that security policy. Hopefully that will fix this. Thanks for including the traceroute output! Please let me know what it looks like now? Thanks!
Flags: needinfo?(jxia)
|
|
Reporter | |
Comment 6•8 years ago
|
||
It's working now, I really thank you for your help.
Flags: needinfo?(jxia)
|
|
||
Comment 7•8 years ago
|
||
Great! Apologies it took a few bounces and back and forth for me to get this working. Glad that it's working now.
Status: UNCONFIRMED → RESOLVED
Closed: 8 years ago → 8 years ago
Resolution: --- → FIXED
|
|
Reporter | |
Comment 8•8 years ago
|
||
:Dcurado, www.google.com/google.com is not working again, this time google sites are resolved to a new prefix 172.217.0.0/19, traceroute shows that this prefix is not be routed to Mozilla's network either, could you please help to add a route for this？ Thanks. Here are DNS and tracroute info: 1. DNS root@ithp:/home/jxia# nslookup > www.google.com Server: 10.240.24.2 Address: 10.240.24.2#53 Non-authoritative answer: Name: www.google.com Address: 172.217.0.68 > google.com Server: 10.240.24.2 Address: 10.240.24.2#53 Non-authoritative answer: Name: google.com Address: 172.217.0.78 2. Tracroute Packets Pings Host Loss% Snt Last Avg Best Wrst StDev 1. 10.240.24.2 0.0% 1133 0.6 0.5 0.4 1.2 0.1 2. ??? 3. 10.2.22.1 3.2% 1133 3.0 4.8 1.4 61.3 5.3 4. 10.2.2.57 2.0% 1133 2.3 5.1 1.3 61.7 7.1 5. 119.253.5.133 2.6% 1133 1.5 5.1 1.3 73.1 6.6 6. 119.253.4.93 2.0% 1133 19.7 4.7 1.3 53.2 6.0 7. 119.253.5.86 2.6% 1133 5.2 4.4 1.4 51.2 5.4 8. 192.168.51.73 2.2% 1133 5.1 5.5 1.5 60.0 7.2 9. 159.226.253.113 3.3% 1132 2.1 4.7 1.5 54.0 5.3 10. 159.226.253.54 2.7% 1132 3.6 6.7 1.9 83.3 8.4 11. 159.226.254.250 2.0% 1132 38.3 39.0 36.5 77.3 3.6 12. 72.14.221.138 2.1% 1132 37.4 39.8 36.6 119.4 7.5 13. 209.85.241.58 2.6% 1132 40.4 39.6 37.1 77.4 3.6 14. 216.239.40.13 2.1% 1132 35.1 37.9 34.9 77.2 4.7 15. 72.14.238.35 1.6% 1132 83.8 85.9 83.4 120.4 3.3 16. 209.85.242.89 2.3% 1132 178.8 182.6 177.7 232.0 8.5 17. 64.233.174.207 2.6% 1132 188.3 188.3 186.1 213.7 2.8 18. 209.85.249.62 2.6% 1132 184.9 185.3 183.1 218.2 3.0 19. 72.14.232.35 2.4% 1132 187.0 188.3 186.0 215.9 3.1 20. ??? 3. ping root@ithp:/home/jxia# ping www.google.com PING www.google.com (172.217.0.68) 56(84) bytes of data. ^C --- www.google.com ping statistics --- 30 packets transmitted, 0 received, 100% packet loss, time 29230ms
Status: RESOLVED → REOPENED
Ever confirmed: true
Resolution: FIXED → ---
|
||
Comment 9•8 years ago
|
||
Jiechen, it looks like mocprobe1.corp.pek2.mozilla.com may be unreachable. Would you be able to check on this host?
|
|
Reporter | |
Comment 10•8 years ago
|
||
Traffic from pek1 to US is not so good, I've reached ISP to research into this issue.
|
|
||
Comment 11•8 years ago
|
||
I'll add yet another static route for google's latest IP address change. You may want to consider some sort of long term strategy here, other than re-opening this bug every time google's IP addressing changes.
|
|
Reporter | |
Comment 12•8 years ago
|
||
Thank you dcurado, I'm not clear about how we get the initial ip subnet of google sites, but it seems our DNS server give a lot of new ip recently. I think the long term strategy is either keep the DNS resolve the ip to known ip or automatically add the new ip to firewall.
|
|
||
Comment 13•8 years ago
|
||
Google has data centers all over the world, and they load balance work between those data centers. That is probably why you see the IP addresses changing. I don't know of a process that can track the IP address that www.google.com resolves to, and automatically enter a static route for it. If Google changes the IP address frequently, that process could run out of control. The solution we have in place feels like a band-aid. A long term solution may be to change how you get to the Internet, like a private line to an ISP in Hong Kong or something like that? I am going to close this bug for now, as there is no action for me to take at this time.
Status: REOPENED → RESOLVED
Closed: 8 years ago → 8 years ago
Resolution: --- → FIXED
|
|
Reporter | |
Comment 14•7 years ago
|
||
www.google.com is unreachable in Beijing office again. Mtr info as below: # mtr -rn www.google.com HOST: ithp Loss% Snt Last Avg Best Wrst StDev 1.|-- 10.240.24.2 0.0% 10 0.6 0.6 0.5 0.6 0.1 2.|-- 10.0.24.129 90.0% 10 3.7 3.7 3.7 3.7 0.0 3.|-- 223.202.6.2 0.0% 10 4.2 4.3 4.2 4.4 0.1 4.|-- 223.202.72.121 0.0% 10 4.9 5.0 4.9 5.3 0.1 5.|-- 223.202.72.58 0.0% 10 4.6 5.0 4.5 7.8 1.0 6.|-- 172.18.0.101 0.0% 10 7.7 7.6 7.3 8.8 0.4 7.|-- 222.35.253.153 0.0% 10 5.0 5.1 5.0 5.3 0.1 8.|-- 61.233.9.222 0.0% 10 8.4 7.8 6.0 10.0 1.3 9.|-- 61.237.126.210 0.0% 10 39.8 38.7 36.6 40.0 1.3 10.|-- 61.237.123.210 0.0% 10 35.8 35.9 35.7 36.1 0.1 11.|-- 72.14.194.170 0.0% 10 42.9 43.0 42.9 43.1 0.1 12.|-- 108.170.241.70 0.0% 10 43.2 43.5 42.9 44.4 0.5 13.|-- 72.14.238.91 0.0% 10 42.3 42.6 42.3 43.0 0.2 14.|-- 209.85.247.146 0.0% 10 63.6 74.4 62.8 105.6 14.7 15.|-- 209.85.248.33 0.0% 10 178.5 178.5 178.4 178.8 0.1 16.|-- 216.239.49.199 0.0% 10 177.1 176.9 176.6 177.2 0.2 17.|-- 209.85.249.62 0.0% 10 177.8 177.7 177.5 177.9 0.1 18.|-- 108.170.242.225 10.0% 10 177.7 177.6 177.5 177.7 0.1 19.|-- 72.14.235.3 10.0% 10 177.7 177.7 177.6 177.8 0.1 20.|-- 172.217.6.36 10.0% 10 220.8 217.6 211.6 221.4 3.8 Firewall in PEK2 do forward packets to PEK1 at hop 2, but then the packcets didn't go through the ipsec tunnel, but go directly to 223.202.6.2(gateway of PEK1). This is why www.google.com is unaccessable. At the meantime, imap.googlemail.com go through the ipsec tunnel properly: # mtr -rn imap.googlemail.com HOST: ithp Loss% Snt Last Avg Best Wrst StDev 1.|-- 10.240.24.2 0.0% 10 0.6 0.6 0.5 0.6 0.0 2.|-- 10.0.24.129 90.0% 10 3.9 3.9 3.9 3.9 0.0 3.|-- 10.0.24.10 0.0% 10 239.2 248.7 239.2 252.1 4.0 4.|-- 63.245.219.169 0.0% 10 237.0 248.3 237.0 253.2 4.7 5.|-- 67.92.171.49 10.0% 10 237.7 248.8 237.7 251.7 4.9 6.|-- 207.88.13.224 0.0% 10 236.9 249.4 236.9 254.5 5.0 7.|-- 207.88.13.229 0.0% 10 232.5 249.0 232.5 256.1 6.4 8.|-- 216.156.84.86 0.0% 10 231.5 249.7 231.5 254.1 6.6 9.|-- 108.170.243.2 10.0% 10 233.5 249.6 233.5 255.3 6.3 10.|-- 209.85.246.20 10.0% 10 236.5 253.9 236.5 261.8 7.7 11.|-- 72.14.232.63 0.0% 10 255.8 272.4 255.8 296.0 10.0 12.|-- 216.239.40.144 10.0% 10 296.6 285.6 274.2 303.6 10.2 13.|-- 209.85.142.18 10.0% 10 268.8 273.5 268.8 277.6 2.3 14.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 15.|-- 74.125.199.16 10.0% 10 265.5 272.5 265.5 275.4 2.7
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
|
||
Updated•7 years ago
|
Flags: needinfo?(jbarnell)
|
|
||
Comment 15•7 years ago
|
||
Any luck with this request?
|
|
Reporter | |
Comment 16•7 years ago
|
||
This issue still remains unsolved.
|
||
Comment 17•7 years ago
|
||
JXIA ... I think the question here is ... Is Gmail, and associated apps (docs, sheets, cal, etc) working in the office. We need to ensure we're meeting the need here. If they are then this bug should be closed.
Flags: needinfo?(jbarnell)
|
|
Reporter | |
Comment 18•7 years ago
|
||
Gmail and associated apps is not working, because www.google.com will be requested when access Gmail(docs,sheets, etc) via web, here is the request flow： https://sso.mozilla.com/gmail https://mozilla.okta.com/home/google/ https://mozilla.okta.com/login/login.htm https://mozilla.okta.com/app/google/ky/sso/saml/mail https://www.google.com/a/mozilla.com/acs #process die here https://accounts.google.com/signin/continue?ui=2&view=bsp&ver= #packets of accounts.google.com forward to 223.202.6.2 as well Thanks.
|
|
||
Updated•7 years ago
|
Assignee: dcurado → network-operations
|
|
||
Comment 19•7 years ago
|
||
Netops this looks to be a routing issue, we should confirm the ip addresses and fix any routes. This is both on pek2 and pek1 routers.
|
|
||
Comment 20•7 years ago
|
||
Can you please check this. I've added some additional routes to walk the traffic back:
PEK1:
+ route 172.217.0.0/19 {
+ next-hop 10.0.24.10;
+ qualified-next-hop 10.0.24.14 {
+ preference 20;
+ }
+ }
+ route 108.177.96.0/19 {
+ next-hop 10.0.24.10;
+ qualified-next-hop 10.0.24.14 {
+ preference 20;
+ }
+ }
PEK2
set route 108.177.8.0/21 next-hop 10.0.24.129
set route 108.177.96.0/19 next-hop 10.0.24.129Flags: needinfo?(jxia)
|
|
Reporter | |
Comment 21•7 years ago
|
||
www.google.com resolved to 172.217.5.100, it route to 10.0.24.129, but there is a high packets lost rates at 10.0.24.129.
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 10.240.24.2 0.0% 63 0.6 0.5 0.4 0.6 0.0
2. 10.0.24.129 80.6% 63 3.7 3.7 3.6 3.9 0.1
3. ???
ping www.google.com
PING www.google.com (172.217.5.100) 56(84) bytes of data.
^C
--- www.google.com ping statistics ---
30 packets transmitted, 0 received, 100% packet loss, time 29214ms
as a result, www.google.com is still unaccessable.Flags: needinfo?(jxia)
|
|
Reporter | |
Comment 22•7 years ago
|
||
here is the tracert info of smtp.gmail.com:
mtr -n smtp.gmail.com
My traceroute [v0.80]
ithp (0.0.0.0) Tue Apr 18 11:22:33 2017
Keys: Help Display mode Restart statistics Order of fields quit
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 10.240.24.2 0.0% 13 0.5 0.5 0.5 0.6 0.0
2. 10.0.24.129 76.9% 13 3.7 3.8 3.7 3.9 0.1
3. 10.0.24.10 0.0% 13 222.9 228.8 216.2 243.1 7.5
4. 63.245.219.169 0.0% 13 223.4 230.4 214.4 244.2 8.3
5. 67.92.171.49 8.3% 13 228.7 232.3 217.3 244.1 7.9
6. 207.88.13.224 0.0% 13 229.9 232.7 217.3 242.9 7.0
7. 207.88.13.227 0.0% 13 232.8 233.9 217.4 245.6 7.2
8. 216.156.85.86 0.0% 12 230.2 234.6 221.7 244.8 6.2
9. 108.170.242.243 0.0% 12 228.5 234.6 217.8 246.2 7.4
10. 209.85.249.63 0.0% 12 228.0 235.2 219.2 245.3 7.2
11. 216.239.49.198 0.0% 12 244.6 251.6 234.0 261.0 8.1
12. 216.239.40.142 0.0% 12 250.8 272.5 250.8 292.0 12.9
13. 209.85.143.127 0.0% 12 257.5 255.4 242.0 267.0 7.3
14. ???
ping smtp.gmail.com
PING gmail-smtp-msa.l.google.com (74.125.28.108) 56(84) bytes of data.
64 bytes from pc-in-f108.1e100.net (74.125.28.108): icmp_req=1 ttl=44 time=261 ms
64 bytes from pc-in-f108.1e100.net (74.125.28.108): icmp_req=2 ttl=44 time=269 ms
64 bytes from pc-in-f108.1e100.net (74.125.28.108): icmp_req=3 ttl=44 time=276 ms
64 bytes from pc-in-f108.1e100.net (74.125.28.108): icmp_req=4 ttl=44 time=270 ms
^C
--- gmail-smtp-msa.l.google.com ping statistics ---
5 packets transmitted, 4 received, 20% packet loss, time 4003ms
rtt min/avg/max/mdev = 261.299/269.434/276.485/5.422 ms|
|
||
Comment 23•7 years ago
|
||
Change 1 PEK2: jbarnell@fw1-0.ops.pek2.mozilla.net# show | compare [edit security address-book global] address infoblox-clusters_14 { ... } + address google-mail11 108.177.8.0/21; + address google-mail12 108.177.96.0/19; [edit security address-book global address-set google-mail] address www-google2 { ... } + address google-mail11; + address google-mail12; Change 2 PEK1: jbarnell@fw1.console.pek1.mozilla.net# show | compare [edit security zones security-zone vpn address-book] address admin1.scl3c { ... } + address google-mail12 172.217.0.0/19; + address google-mail13 108.177.8.0/21; + address google-mail14 108.177.96.0/19; [edit security zones security-zone vpn address-book address-set google-mail] address google-mail11 { ... } + address google-mail12; + address google-mail13; + address google-mail14; Please try again .
Flags: needinfo?(jxia)
|
|
Reporter | |
Comment 24•7 years ago
|
||
still not working.
Packets Pings
Host Loss% Snt Last Avg Best Wrst StDev
1. 10.240.24.2 0.0% 67 0.5 0.5 0.4 0.8 0.1
2. 10.0.24.129 75.8% 67 3.7 3.7 3.4 4.3 0.2
3. ???
ping www.google.com
PING www.google.com (172.217.5.100) 56(84) bytes of data.
^C
--- www.google.com ping statistics ---
58 packets transmitted, 0 received, 100% packet loss, time 57009msFlags: needinfo?(jxia)
|
|
||
Comment 25•7 years ago
|
||
One more time:
SJC2/PAO1
[edit security address-book global]
address svc.phx1-net_2 { ... }
+ address google-mail_12 172.217.0.0/19;
+ address google-mail_13 108.177.96.0/19;
[edit security address-book global address-set google-mail]
address google-mail_11 { ... }
+ address google-mail_12;
+ address google-mail_13;
Try again|
|
Reporter | |
Comment 26•7 years ago
|
||
It works, thanks James.
|
|
||
Comment 27•7 years ago
|
||
Great have a good day.
Status: REOPENED → RESOLVED
Closed: 8 years ago → 7 years ago
Resolution: --- → FIXED
|
||
Updated•2 years ago
|
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•