1222136 - HTTP/2 connection pool breaks SNI hosts

Status: RESOLVED INVALID

(Core :: Networking: HTTP, defect)

Description

[Julien Vehent]

I operate a few personal sites hosted on a single IPv4. One of these site uses HTTP/2 for a photo gallery. The TCP load balancer in front of all the sites (haproxy) looks at the SNI of the incoming handshake, and based on its content, sends the handshake to the right https backend.

The problem occurs when Firefox has an http/2 connection established on site 1, and I open a new tab on site 2 which is hosted on the same IP. Site 2 returns a 404. After doing some digging, it appears that the connection to site 2 was in fact sent to site 1, because Firefox reused the HTTP/2 connection that was already opened with site 1, and the load balancer just continue forward traffic to the backend it already had a connection established too.

                                  +--------------------+
                                  |site1.example.net   |
                                  +---------^----------+
+------+                                    |
|      |           +---------+              |
|firefox+--http/2->+haproxy  +----http/2----+
|      |           +---------+
+------+                          +--------------------+
                                  |site2.com           |
                                  +--------------------+

I think the expected behavior here, would be to not reuse an existing http/2 connection when the target Host is different. That would force Firefox to open a new connection to site2, which would allow the load balancer to inspect the SNI extension again, and send the request to the right backend.

Comment 1

[April King [:April]]

I actually think that your load balancer is behaving incorrectly here, according to the HTTP/2 specification:

HTTP/2 9.1:

A client MAY open multiple connections to the same IP address and TCP port using different Server Name Indication [TLS-EXT] values or to provide different TLS client certificates but SHOULD avoid creating multiple connections with the same configuration.

HTTP/2 9.1.1:

Connections that are made to an origin server, either directly or through a tunnel created using the CONNECT method (Section 8.3), MAY be reused for requests with multiple different URI authority components. A connection can be reused as long as the origin server is authoritative (Section 10.1). For TCP connections without TLS, this depends on the host having resolved to the same IP address.

For https resources, connection reuse additionally depends on having a certificate that is valid for the host in the URI. The certificate presented by the server MUST satisfy any checks that the client would perform when forming a new TLS connection for the host in the URI.


Notably, authority is determined via RFC 2818 (section 3) @ https://tools.ietf.org/html/rfc2818#section-3.


If I had to hazard a guess, I would guess that your haproxy server presented a certificate that was valid for both site1.example.net and site2.com, when the TLS connection was initially setup.  As such, Firefox reused that TLS connection for subsequent requests, as recommended by HTTP/2.

Comment 2

[u408661]

Indeed, what April said in comment 1 seems the most likely scenario. If, however, haproxy only presents a cert valid for site1.example.net, then we have a bug on our end, and one that should be fixed sooner rather than later. Julien, what does the cert that haproxy presents look like on the first connection?

Comment 3

[Julien Vehent]

I do use a single SAN cert for both sites. You can see the cert at jve.linuxwall.info.
I'll test again with separate certs, and close this bug as invalid if needed. It's still threw me off quite a bit though, but I don't think there's much we can do about it.

Comment 4

[Julien Vehent]

Confirmed: when using separate certificates, Firefox creates separate connections. Same in Chrome.

Comment 5

[Patrick McManus [:mcmanus]]

There are a bunch of ways to opt out of this coalescing
1] different certs as you've discovered
2] different dns ips
3] if a transaction is routed to a service that doesn't actually support that origin (the full origin is in the request header) it can return 421 and the client will retry on a different connection.

to level up we're definitely going to an era where authority is determined more by pki than by addressing. (enter content centric networking?)

there has been discussion at ietf94 this week about allowing requirement #2 above to be waived if an existing connection actually opts into additional origins during the connection (and has the cert to cover those origins, of course). Its basically doing the same thing through dns now.

fwiw, this is actually an exciting h2 feature because it allows us to de-shard the web in a backwards compatible way... people added multiple host names for the same server in order to get h1 parallelism that is counter productive in h2 and this feature allows that to be undone while still serving h1 clients in the old fashion.