1235068 - SPDY coalescing should not work across address families

Status: RESOLVED INVALID

(Core :: Networking: HTTP, defect)

Description

[Faidon Liambotis]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:42.0) Gecko/20100101 Firefox/42.0 Iceweasel/42.0
Build ID: 20151104064442

Steps to reproduce:

0. Use a computer that has a dual-stack IPv4/IPv6 connection. I used Linux but it probably doesn't matter.
1. Visit https://en.wikipedia.org/wiki/Main_Page
2. Open a new tab, visit https://geoiplookup.wikimedia.org/


Actual results:

Output is Geo = {IPv6:true}


Expected results:

Output should have been a json with a location that was calculated by geolocating an IPv4 address (e.g. Geo = {"city":"Athens","country":"GR","region":"35","lat":"37","lon":"23","IP":"192.0.2.235"})

This should happen, because in contrast with en.wikipedia.org, geoiplookup.wikimedia.org only has an A record and not an AAAA. It should never be reached over IPv6, on purpose, as the service currently can perform only GeoIP lookups for IPv4 addresses. The separate domain exists precisely for this purpose: to force dual-stacked clients to connect to it over IPv4, and hence be able to (more accurately) geolocate them.

I suspected this to be an effect of SPDY coalescing being too aggressive and I've confirmed this by setting network.http.spdy.coalesce-hostnames to false, after which the bug cannot be observed anymore. It also does not happen with any other browser — I've confirmed both locally with Chrome, but also from Wikimedia's logs (99.7% of the IPv6 requests to geoiplookup.wikimedia.org over a 24hr period are coming from Firefox).

What happens here is that both hostnames have A records pointing to the same IPv4 address, but one of the two also has an AAAA record to an IPv6 address. When requested to connect to the dual-stacked hostname, Firefox (obviously) prefers the IPv6 connects; then when a request to connect to the second hostname is performed, Firefox in an attempt to perform SPDY coalescing, compares the A record of the two hostnames, and after they match, performs the HTTP request to the second hostname over the same SPDY session. From a quick search, bug #1129571 sounds possibly related.

Note that because of this bug, geoiplookup.wikimedia.org may soon gets its very own IPv4 address and/or stop working entirely, and hence the steps to reproduce above may not work (FWIW, this is tracked over at Wikimedia's bug tracker with https://phabricator.wikimedia.org/T121922)

Comment 1

[Patrick McManus [:mcmanus]]

Thanks for taking the time to file this where I can see it. I'm going to close this as invalid, explain why, and give you a couple of other options to help with your issue.

web content is defined by origin (host, port, and scheme) and security is enforced via the certificates at the host level. Routing (i.e. where to fetch that data from) is a bit looser - we've always been willing to take content from a cache and h2 allows alternate hosts to be identified either through coalescing or through alt-svc. But that never changes the definition of the content (origin) or how it has to be secured (with a certificate that matches the origin); IP addresses aren't really a fundamental part of that definition.

So in a > H1 world, firefox has decided that these two sites share "routing overlap" because they have an intersecting member and the certificate presented when we connect covers both sites (confirmed):

en.wikipedia.org.       319     IN      A       208.80.154.224
en.wikipedia.org.       243     IN      AAAA    2620:0:861:ed1a::1
geoiplookup.wikimedia.org. 544  IN      A       208.80.154.224

At that point its an optimization to use an existing connection for whatever origins that it has proof of control of (back to the cert it is using). Those sets are not the same, but they do intersect and the assumption is that some kind of dynamic dns is being used that prevents us from seeing the whole set of valid content. (this is true e.g. with yahoo!).. I hope you can at least see the use case there.

so, if you don't want geoip to coalesce with en there are at least 3 things you can do, ordered from most to least severe:

1] make sure the address sets do not overlap (you're considering that one by giving it a new address)
2] make sure they use certs that don't overlap - this is pretty easy if you just want it for geoip
3] when you receive a query for host:geoip carried on the v6 address, return a "421 Misdirected" response code. Clients will replay the request on a fresh connection when that happens

hth.

Comment 2

[Faidon Liambotis]

Thanks for the quick & elaborate response and your suggestions, much appreciated. We have a few solutions in mind already and we'll work around this very soon -- this wasn't about our specific problem, but about the broader problems this may pose.

I understand the rationale -- I guessed this was done to work around dynamic/loadbalancing DNS responses and I can understand the use case, yeah.

That said, I'm not entirely sure that ignoring the presence of an address family entirely is such a good idea; skipping AAAAs is obviously not something that one would do in that use case. I can see the "origin definition" argument too, but that is not /entirely/ true as stated -- if it was, you'd always ask any *.wikimedia.org domain over the same SPDY/H2 session without even doing a DNS query, since the certificate in the existing session already provides you that proof of possession for the whole domain. The fact that solution (1) (giving the two endpoints non-intersecting address sets) will work is a proof that IPs /are/ a fundamental part of the definition.

Anyway, we were just caught by surprise by this Firefox-specific behavior, especially since we've been doing this geoiplookup trick for a quite a long time. If it's just our niche case and not somewhat of a wider problem indeed, I can understand your decision given the tradeoff at play here.