763478 - visits to mozilla.org in Paris are hitting the Phoenix servers and not the Amsterdam servers

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Pascal Chevrel (relman team) -> :pascalc]

I noticed that going to mozilla.org is slow, here is the result of my ping from the Mozilla Paris office:

ping www.mozilla.org

PING bedrock-prod.zlb.phx.mozilla.net (63.245.217.105) 56(84) bytes of data.
64 bytes from bedrock-prod.zlb.phx.mozilla.net (63.245.217.105): icmp_req=1 ttl=50 time=154 ms
64 bytes from bedrock-prod.zlb.phx.mozilla.net (63.245.217.105): icmp_req=2 ttl=50 time=158 ms

People in Europe should hit the Amsterdam server and get a more decent ping for our download pages, not the servers in the US. I don't know if the problem is only for the Paris area, the country or the whole European continent.

If you make config changes and need me to test from Paris, I am on IRC and will be glad to help (pascalc on #l10n)

Thanks!

Comment 1

[Jake Maul [:jakem]]

This is handled by 3crowd, and they use a pretty normal/simple GeoIP mapping scheme. It guesses where the user is based on the IP address that queries 3crowd... which is usually not the end user, but the end user's ISP's DNS server.

Can you check what you're using for a DNS resolver? If it's a Mozilla server/IP, I'm guessing it's actually somewhere in the US, and there won't be much we can do. If you just do a "dig www.mozilla.org" it should be in the output near the bottom. In fact, maybe just post the whole 'dig' output here. :)

If your resolver is a Mozilla IP and it really does live in France, we can possibly open a bug with 3crowd to have them tweak their mapping rules.

If your resolver is not a Mozilla IP (ex: your ISP), we can still alert them to a potential discrepancy... not sure we can do anything though. It would be between 3crowd and the operator of that resolver.

Comment 2

[Arzhel Younsi [:XioNoX]]

The resolvers in the office are admin1a/b, which are configured to use Google's DNS as forwarders.

[[e-mail]2 ~]# host www.mozilla.org
www.mozilla.org is an alias for mozorg.3crowd.mozilla.net.
mozorg.3crowd.mozilla.net has address 63.245.215.20
Host mozorg.3crowd.mozilla.net not found: 3(NXDOMAIN)
[[e-mail]2 ~]# host www.mozilla.org 8.8.8.8
www.mozilla.org is an alias for mozorg.3crowd.mozilla.net.
mozorg.3crowd.mozilla.net is an alias for bedrock-prod.zlb.phx.mozilla.net.
bedrock-prod.zlb.phx.mozilla.net has address 63.245.217.105

But if I use the ISP's DNS, it returns a correct answer
[[e-mail]2 slave-config]# host www.mozilla.org 212.27.40.240
www.mozilla.org is an alias for mozorg.3crowd.mozilla.net.
mozorg.3crowd.mozilla.net has address 63.245.213.92

The same thing happens at home in the UK, querying directly google's DNS returns something in the US, but the ISP's returns AMS1.

Can it be only related to Google's DNS?

Comment 3

[Jake Maul [:jakem]]

Yep, that's almost certainly the problem.

It looks like Google's DNS servers are anycast, and they do exist in a lot of datacenters: https://developers.google.com/speed/public-dns/faq#locations. My guess then is that whatever IP they use (individually) to connect to 3crowd is ultimately mapped back to the US. This may or may not be something that 3crowd could fix (it could be a Google issue or a 3crowd issue), but it's certainly out of our hands.

I'll try to reach someone at 3crowd and see if they have any interest (or ability!) in fixing this.

Comment 4

[Jake Maul [:jakem]]

I brought this up with 3crowd, and they're investigating it. The feeling right now is that it's probably just a mis-mapping of the GeoIP data being used, and might be easily be correctable as they update their data normally. In fact, I was told they did so in June, so it's possible this was corrected some time after this bug was verified.

When you get a chance, could you verify if this is still happening?

Comment 5

[Pascal Chevrel (relman team) -> :pascalc]

That looks fixed today for me:
ping www.mozilla.org
PING mozorg01.zlb.nl.mozilla.com (63.245.213.92) 56(84) bytes of data.
64 bytes from mozorg01.zlb.nl.mozilla.com (63.245.213.92): icmp_req=1 ttl=52 time=19.4 ms
64 bytes from mozorg01.zlb.nl.mozilla.com (63.245.213.92): icmp_req=2 ttl=52 time=19.7 ms

Comment 6

[Jake Maul [:jakem]]

Calling this one fixed... thanks!