Can't access google from Beijing office

RESOLVED FIXED

Status

RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: jxia, Unassigned)

Tracking

Details

(Reporter)

Description

3 years ago
Hi,
www.google.com is unreachable from Beijing office, some informations as below:

1. DNS 
jxia@ithp:~$ nslookup
> www.google.com
Server:         10.240.24.2
Address:        10.240.24.2#53

Non-authoritative answer:
Name:   www.google.com
Address: 216.58.192.4

2. Ping status
jxia@ithp:~$ ping www.google.com
PING www.google.com (216.58.192.4) 56(84) bytes of data.
^C
--- www.google.com ping statistics ---
37 packets transmitted, 0 received, 100% packet loss, time 35995ms

3. tracert info
                                  Packets               Pings
 Host                           Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 10.240.24.2                  0.0%    74    0.6   0.6   0.5   1.3   0.1
 2. ???
 3. 10.2.22.1                    0.0%    74    1.7   2.5   1.5  12.3   1.4
 4. 10.2.2.57                    0.0%    74    3.0   3.4   1.4  29.4   5.0
 5. 119.253.5.133                1.4%    74    1.5   2.5   1.4  19.4   2.2
 6. 119.253.4.85                 4.1%    74    2.7   2.7   1.5  21.2   2.5
 7. 119.253.5.86                 0.0%    74    2.1   3.5   1.5  42.8   5.4
 8. 192.168.51.73                0.0%    74    3.5   2.9   1.6  27.3   3.7
 9. 159.226.253.109              0.0%    74    3.5   3.4   1.6  31.1   4.6
10. 159.226.253.50               0.0%    74    2.6   3.5   1.8  30.4   4.6
11. ???

4. compare with tracert info of smtp.gmail.com shown below, it seems that access www.gmail.com didn't go through the ipsec tunnel,
                           Packets               Pings
 Host                    Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 10.240.24.2           0.0%     2    0.5   0.5   0.5   0.5   0.0
 2. 10.0.24.129           0.0%     2    4.3   4.9   4.3   5.4   0.8
 3. ???
 4. 10.0.24.10            0.0%     2  166.8 166.6 166.5 166.8   0.2
 5. 63.245.219.169        0.0%     2  166.4 166.8 166.4 167.1   0.5
 6. 67.92.171.49          0.0%     2  167.3 167.2 167.2 167.3   0.1
 7. 207.88.13.224         0.0%     2  167.7 173.9 167.7 180.0   8.7
 8. 207.88.13.229         0.0%     2  196.8 182.3 167.7 196.8  20.6
 9. 216.156.84.30         0.0%     2  168.1 168.2 168.1 168.2   0.1
10. 209.85.249.5          0.0%     2  175.8 172.7 169.6 175.8   4.4
11. 209.85.246.38         0.0%     2  171.5 170.5 169.5 171.5   1.4
12. 216.239.49.198        0.0%     2  186.2 186.2 186.2 186.2   0.0
13. 74.125.37.208         0.0%     2  191.1 191.1 191.1 191.1   0.0
14. 74.125.37.219         0.0%     2  194.7 194.7 194.7 194.7   0.0
15. ???
16. 74.125.28.109         0.0%     1  189.2 189.2 189.2 189.2   0.0
You get to google.com via your provider in Beijing.
You get to gmail via Mozilla's network.

The output of your traceroutes confirms this.

Please contact your provider in Beijing regarding this matter.
If we can help further, do not hesitate to re-open this bug.
Thank you.
Assignee: network-operations → dcurado
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → INVALID
(Reporter)

Comment 2

3 years ago
We can't access google sites via local provider in China, this is one of the reasons why route google sites to Mozilla's network. Could you help to check why the ip of google.com/www.google.com not be routed to Mozilla's network?
Thanks.
Status: RESOLVED → REOPENED
Resolution: INVALID → ---
Ah, OK, I think I misread your request.  

Currently we send packets to some parts of Google over Mozilla's network.
To date, packets to google.com have not been routed that way.
But now it looks like Google itself is accepting packets from your provider's
IP space.  
Your traceroutes to google.com actually make it to the US, but then fail, as if
google isn't carrying the routes for your provider's IP space.
I wonder why that would be.

From your original comment, www.google.com resolves to 216.58.192.4 for you.
The CIDR block google announces for that IP range is 216.58.192.0/19.
I will add a route for that, so that it's like what we do for gmail.
Hopefully that will fix this for you.  

I have made this change.  Please let me know if this helped?
Thanks - and apologies for my confusion over your request.
Status: REOPENED → UNCONFIRMED
Ever confirmed: false
Flags: needinfo?(jxia)
(Reporter)

Comment 4

3 years ago
Thanks Dave, but we still can't access google.com/www.google.com, here are some informations:

1.DNS:
Both www.google.com/google.com resolves to an ip which in 216.58.192.0/19 subnet, so it should be routed to the Mozilla's network.
> www.google.com
Server:         10.240.24.2
Address:        10.240.24.2#53
Non-authoritative answer:
Name:   www.google.com
Address: 216.58.194.196

> google.com
Server:         10.240.24.2
Address:        10.240.24.2#53
Non-authoritative answer:
Name:   google.com
Address: 216.58.194.206

2. Tracert:
got no response, even no response from 10.240.24.2
root@ithp:/home/jxia# traceroute 216.58.194.196
traceroute to 216.58.194.196 (216.58.194.196), 30 hops max, 60 byte packets
 1  * * *
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

3. ping
root@ithp:/home/jxia# ping 216.58.194.196
PING 216.58.194.196 (216.58.194.196) 56(84) bytes of data.
^C
--- 216.58.194.196 ping statistics ---
69 packets transmitted, 0 received, 100% packet loss, time 68542ms
Flags: needinfo?(jxia)
Hm.  OK, sorry about that.  I dug around and found there is security policy on the firewall in the PEK2 office which needs to allow packets for gmail (and now www.google.com) out via the PEK1 office.
I've added the new prefix (216.58.192.0/19) to that security policy.  Hopefully that will fix this.

Thanks for including the traceroute output!  

Please let me know what it looks like now? Thanks!
Flags: needinfo?(jxia)
(Reporter)

Comment 6

3 years ago
It's working now, I really thank you for your help.
Flags: needinfo?(jxia)
Great!  Apologies it took a few bounces and back and forth for me to get this working.  Glad that it's working now.
Status: UNCONFIRMED → RESOLVED
Last Resolved: 3 years ago3 years ago
Resolution: --- → FIXED
(Reporter)

Comment 8

3 years ago
:Dcurado, www.google.com/google.com is not working again, this time google sites are resolved to a new prefix 172.217.0.0/19, traceroute shows that this prefix is not be routed to Mozilla's network either, could you please help to add a route for this? Thanks.

Here are DNS and tracroute info:

1. DNS
root@ithp:/home/jxia# nslookup
> www.google.com
Server:         10.240.24.2
Address:        10.240.24.2#53

Non-authoritative answer:
Name:   www.google.com
Address: 172.217.0.68
> google.com
Server:         10.240.24.2
Address:        10.240.24.2#53

Non-authoritative answer:
Name:   google.com
Address: 172.217.0.78 

2. Tracroute                                     Packets               Pings
 Host                                Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 10.240.24.2                       0.0%  1133    0.6   0.5   0.4   1.2   0.1
 2. ???
 3. 10.2.22.1                         3.2%  1133    3.0   4.8   1.4  61.3   5.3
 4. 10.2.2.57                         2.0%  1133    2.3   5.1   1.3  61.7   7.1
 5. 119.253.5.133                     2.6%  1133    1.5   5.1   1.3  73.1   6.6
 6. 119.253.4.93                      2.0%  1133   19.7   4.7   1.3  53.2   6.0
 7. 119.253.5.86                      2.6%  1133    5.2   4.4   1.4  51.2   5.4
 8. 192.168.51.73                     2.2%  1133    5.1   5.5   1.5  60.0   7.2
 9. 159.226.253.113                   3.3%  1132    2.1   4.7   1.5  54.0   5.3
10. 159.226.253.54                    2.7%  1132    3.6   6.7   1.9  83.3   8.4
11. 159.226.254.250                   2.0%  1132   38.3  39.0  36.5  77.3   3.6
12. 72.14.221.138                     2.1%  1132   37.4  39.8  36.6 119.4   7.5
13. 209.85.241.58                     2.6%  1132   40.4  39.6  37.1  77.4   3.6
14. 216.239.40.13                     2.1%  1132   35.1  37.9  34.9  77.2   4.7
15. 72.14.238.35                      1.6%  1132   83.8  85.9  83.4 120.4   3.3
16. 209.85.242.89                     2.3%  1132  178.8 182.6 177.7 232.0   8.5
17. 64.233.174.207                    2.6%  1132  188.3 188.3 186.1 213.7   2.8
18. 209.85.249.62                     2.6%  1132  184.9 185.3 183.1 218.2   3.0
19. 72.14.232.35                      2.4%  1132  187.0 188.3 186.0 215.9   3.1
20. ???

3. ping
root@ithp:/home/jxia# ping www.google.com
PING www.google.com (172.217.0.68) 56(84) bytes of data.
^C
--- www.google.com ping statistics ---
30 packets transmitted, 0 received, 100% packet loss, time 29230ms
Status: RESOLVED → REOPENED
Ever confirmed: true
Resolution: FIXED → ---
Jiechen, it looks like mocprobe1.corp.pek2.mozilla.com may be unreachable.  Would you be able to check on this host?
(Reporter)

Comment 10

3 years ago
Traffic from pek1 to US is not so good, I've reached ISP to research into this issue.
I'll add yet another static route for google's latest IP address change.
You may want to consider some sort of long term strategy here, other than re-opening this bug every
time google's IP addressing changes.
(Reporter)

Comment 12

3 years ago
Thank you dcurado, I'm not clear about how we get the initial ip subnet of google sites, but it seems our DNS server give a lot of new ip recently. I think the long term strategy is either keep the DNS resolve the ip to known ip or automatically add the new ip to firewall.
Google has data centers all over the world, and they load balance work between those data centers.
That is probably why you see the IP addresses changing.

I don't know of a process that can track the IP address that www.google.com resolves to, and automatically enter a static route for it.  If Google changes the IP address frequently, that process could run out of control.  

The solution we have in place feels like a band-aid.  A long term solution may be to change how you get to the Internet, like a private line to an ISP in Hong Kong or something like that?

I am going to close this bug for now, as there is no action for me to take at this time.
Status: REOPENED → RESOLVED
Last Resolved: 3 years ago3 years ago
Resolution: --- → FIXED
(Reporter)

Comment 14

2 years ago
www.google.com is unreachable in Beijing office again. Mtr info as below:
# mtr -rn www.google.com
HOST: ithp                        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 10.240.24.2                0.0%    10    0.6   0.6   0.5   0.6   0.1
  2.|-- 10.0.24.129               90.0%    10    3.7   3.7   3.7   3.7   0.0
  3.|-- 223.202.6.2                0.0%    10    4.2   4.3   4.2   4.4   0.1
  4.|-- 223.202.72.121             0.0%    10    4.9   5.0   4.9   5.3   0.1
  5.|-- 223.202.72.58              0.0%    10    4.6   5.0   4.5   7.8   1.0
  6.|-- 172.18.0.101               0.0%    10    7.7   7.6   7.3   8.8   0.4
  7.|-- 222.35.253.153             0.0%    10    5.0   5.1   5.0   5.3   0.1
  8.|-- 61.233.9.222               0.0%    10    8.4   7.8   6.0  10.0   1.3
  9.|-- 61.237.126.210             0.0%    10   39.8  38.7  36.6  40.0   1.3
 10.|-- 61.237.123.210             0.0%    10   35.8  35.9  35.7  36.1   0.1
 11.|-- 72.14.194.170              0.0%    10   42.9  43.0  42.9  43.1   0.1
 12.|-- 108.170.241.70             0.0%    10   43.2  43.5  42.9  44.4   0.5
 13.|-- 72.14.238.91               0.0%    10   42.3  42.6  42.3  43.0   0.2
 14.|-- 209.85.247.146             0.0%    10   63.6  74.4  62.8 105.6  14.7
 15.|-- 209.85.248.33              0.0%    10  178.5 178.5 178.4 178.8   0.1
 16.|-- 216.239.49.199             0.0%    10  177.1 176.9 176.6 177.2   0.2
 17.|-- 209.85.249.62              0.0%    10  177.8 177.7 177.5 177.9   0.1
 18.|-- 108.170.242.225           10.0%    10  177.7 177.6 177.5 177.7   0.1
 19.|-- 72.14.235.3               10.0%    10  177.7 177.7 177.6 177.8   0.1
 20.|-- 172.217.6.36              10.0%    10  220.8 217.6 211.6 221.4   3.8

 Firewall in PEK2 do forward packets to PEK1 at hop 2, but then the packcets didn't go through the ipsec tunnel, but go directly to 223.202.6.2(gateway of PEK1). This is why www.google.com is unaccessable.

 At the meantime, imap.googlemail.com go through the ipsec tunnel properly:
 # mtr -rn imap.googlemail.com
 HOST: ithp                        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 10.240.24.2                0.0%    10    0.6   0.6   0.5   0.6   0.0
  2.|-- 10.0.24.129               90.0%    10    3.9   3.9   3.9   3.9   0.0
  3.|-- 10.0.24.10                 0.0%    10  239.2 248.7 239.2 252.1   4.0
  4.|-- 63.245.219.169             0.0%    10  237.0 248.3 237.0 253.2   4.7
  5.|-- 67.92.171.49              10.0%    10  237.7 248.8 237.7 251.7   4.9
  6.|-- 207.88.13.224              0.0%    10  236.9 249.4 236.9 254.5   5.0
  7.|-- 207.88.13.229              0.0%    10  232.5 249.0 232.5 256.1   6.4
  8.|-- 216.156.84.86              0.0%    10  231.5 249.7 231.5 254.1   6.6
  9.|-- 108.170.243.2             10.0%    10  233.5 249.6 233.5 255.3   6.3
 10.|-- 209.85.246.20             10.0%    10  236.5 253.9 236.5 261.8   7.7
 11.|-- 72.14.232.63               0.0%    10  255.8 272.4 255.8 296.0  10.0
 12.|-- 216.239.40.144            10.0%    10  296.6 285.6 274.2 303.6  10.2
 13.|-- 209.85.142.18             10.0%    10  268.8 273.5 268.8 277.6   2.3
 14.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 15.|-- 74.125.199.16             10.0%    10  265.5 272.5 265.5 275.4   2.7
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Flags: needinfo?(jbarnell)
Any luck with this request?
(Reporter)

Comment 16

2 years ago
This issue still remains unsolved.
JXIA ... I think the question here is ... Is Gmail, and associated apps (docs, sheets, cal, etc) working in the office.  We need to ensure we're meeting the need here.  If they are then this bug should be closed.
Flags: needinfo?(jbarnell)
(Reporter)

Comment 18

2 years ago
Gmail and associated apps is not working, because www.google.com will be requested when access Gmail(docs,sheets, etc) via web, here is the request flow:

https://sso.mozilla.com/gmail
https://mozilla.okta.com/home/google/
https://mozilla.okta.com/login/login.htm
https://mozilla.okta.com/app/google/ky/sso/saml/mail
https://www.google.com/a/mozilla.com/acs        #process die here
https://accounts.google.com/signin/continue?ui=2&view=bsp&ver=  #packets of accounts.google.com forward to 223.202.6.2 as well

Thanks.
Assignee: dcurado → network-operations
Netops this looks to be a routing issue, we should confirm the ip addresses and fix any routes.  This is both on pek2 and pek1 routers.
Can you please check this.  I've added some additional routes to walk the traffic back:

PEK1:

+  route 172.217.0.0/19 {
+      next-hop 10.0.24.10;
+      qualified-next-hop 10.0.24.14 {
+          preference 20;
+      }
+  }
+  route 108.177.96.0/19 {
+      next-hop 10.0.24.10;
+      qualified-next-hop 10.0.24.14 {
+          preference 20;
+      }
+  }

PEK2

set route 108.177.8.0/21 next-hop 10.0.24.129
set route 108.177.96.0/19 next-hop 10.0.24.129
Flags: needinfo?(jxia)
(Reporter)

Comment 21

2 years ago
www.google.com resolved to 172.217.5.100, it route to 10.0.24.129, but there is a high packets lost rates at 10.0.24.129.
                                                                                                                                          Packets               Pings
 Host                                                                                                                                    Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 10.240.24.2                                                                                                                           0.0%    63    0.6   0.5   0.4   0.6   0.0
 2. 10.0.24.129                                                                                                                          80.6%    63    3.7   3.7   3.6   3.9   0.1
 3. ???


ping www.google.com
PING www.google.com (172.217.5.100) 56(84) bytes of data.
^C
--- www.google.com ping statistics ---
30 packets transmitted, 0 received, 100% packet loss, time 29214ms

as a result, www.google.com is still unaccessable.
Flags: needinfo?(jxia)
(Reporter)

Comment 22

2 years ago
here is the tracert info of smtp.gmail.com:
mtr -n smtp.gmail.com
                                                                               My traceroute  [v0.80]
ithp (0.0.0.0)                                                                                                                                             Tue Apr 18 11:22:33 2017
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                                                                                           Packets               Pings
 Host                                                                                                                                    Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 10.240.24.2                                                                                                                           0.0%    13    0.5   0.5   0.5   0.6   0.0
 2. 10.0.24.129                                                                                                                          76.9%    13    3.7   3.8   3.7   3.9   0.1
 3. 10.0.24.10                                                                                                                            0.0%    13  222.9 228.8 216.2 243.1   7.5
 4. 63.245.219.169                                                                                                                        0.0%    13  223.4 230.4 214.4 244.2   8.3
 5. 67.92.171.49                                                                                                                          8.3%    13  228.7 232.3 217.3 244.1   7.9
 6. 207.88.13.224                                                                                                                         0.0%    13  229.9 232.7 217.3 242.9   7.0
 7. 207.88.13.227                                                                                                                         0.0%    13  232.8 233.9 217.4 245.6   7.2
 8. 216.156.85.86                                                                                                                         0.0%    12  230.2 234.6 221.7 244.8   6.2
 9. 108.170.242.243                                                                                                                       0.0%    12  228.5 234.6 217.8 246.2   7.4
10. 209.85.249.63                                                                                                                         0.0%    12  228.0 235.2 219.2 245.3   7.2
11. 216.239.49.198                                                                                                                        0.0%    12  244.6 251.6 234.0 261.0   8.1
12. 216.239.40.142                                                                                                                        0.0%    12  250.8 272.5 250.8 292.0  12.9
13. 209.85.143.127                                                                                                                        0.0%    12  257.5 255.4 242.0 267.0   7.3
14. ???


ping smtp.gmail.com
PING gmail-smtp-msa.l.google.com (74.125.28.108) 56(84) bytes of data.
64 bytes from pc-in-f108.1e100.net (74.125.28.108): icmp_req=1 ttl=44 time=261 ms
64 bytes from pc-in-f108.1e100.net (74.125.28.108): icmp_req=2 ttl=44 time=269 ms
64 bytes from pc-in-f108.1e100.net (74.125.28.108): icmp_req=3 ttl=44 time=276 ms
64 bytes from pc-in-f108.1e100.net (74.125.28.108): icmp_req=4 ttl=44 time=270 ms
^C
--- gmail-smtp-msa.l.google.com ping statistics ---
5 packets transmitted, 4 received, 20% packet loss, time 4003ms
rtt min/avg/max/mdev = 261.299/269.434/276.485/5.422 ms
Change 1 PEK2:

jbarnell@fw1-0.ops.pek2.mozilla.net# show | compare 
[edit security address-book global]
     address infoblox-clusters_14 { ... }
+    address google-mail11 108.177.8.0/21;
+    address google-mail12 108.177.96.0/19;
[edit security address-book global address-set google-mail]
      address www-google2 { ... }
+     address google-mail11;
+     address google-mail12;

Change 2 PEK1:
jbarnell@fw1.console.pek1.mozilla.net# show | compare 
[edit security zones security-zone vpn address-book]
       address admin1.scl3c { ... }
+      address google-mail12 172.217.0.0/19;
+      address google-mail13 108.177.8.0/21;
+      address google-mail14 108.177.96.0/19;
[edit security zones security-zone vpn address-book address-set google-mail]
        address google-mail11 { ... }
+       address google-mail12;
+       address google-mail13;
+       address google-mail14;

Please try again .
Flags: needinfo?(jxia)
(Reporter)

Comment 24

2 years ago
still not working.


                                                                                                                                          Packets               Pings
 Host                                                                                                                                    Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 10.240.24.2                                                                                                                           0.0%    67    0.5   0.5   0.4   0.8   0.1
 2. 10.0.24.129                                                                                                                          75.8%    67    3.7   3.7   3.4   4.3   0.2
 3. ???


ping www.google.com
PING www.google.com (172.217.5.100) 56(84) bytes of data.
^C
--- www.google.com ping statistics ---
58 packets transmitted, 0 received, 100% packet loss, time 57009ms
Flags: needinfo?(jxia)
One more time:

SJC2/PAO1

[edit security address-book global]
     address svc.phx1-net_2 { ... }
+    address google-mail_12 172.217.0.0/19;
+    address google-mail_13 108.177.96.0/19;
[edit security address-book global address-set google-mail]
      address google-mail_11 { ... }
+     address google-mail_12;
+     address google-mail_13;

Try again
(Reporter)

Comment 26

2 years ago
It works, thanks James.
Great have a good day.
Status: REOPENED → RESOLVED
Last Resolved: 3 years ago2 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.