1673231 - HTTP3 on Firefox 82.0 intermitently fails to establish TLS with google, disabling HTTP3 support in about:config fixes the problem imediatly

Status: RESOLVED FIXED

(Core :: Networking: HTTP, defect, P2)

Description

[undercover_black_hat]

User Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:82.0) Gecko/20100101 Firefox/82.0

Steps to reproduce:

Set network.http.http3.enabled to True, try and access Google or using the search bar to google something. I am running on Fedora 32 with firefox-82.0-5.fc32.x86_64.

Actual results:

It take a long time to work and most times (but not always) it says that it failed to establish a secure connection.

Expected results:

It should have loaded Google without any problem, it worked just fine in the previous version with HTTP3 enabled so I am pretty confident the latest version caused a regression.

Comment 1

[robsmith11]

I have the same problem with Firefox 82 on Arch Linux with both Google and Youtube. I've also been experiencing it on Firefox for Android Nightly for the past few weeks (disabling http3 also fixed it there).

Comment 2

[Alin Ilea, Desktop Test Engineering]

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:82.0) Gecko/20100101 Firefox/82.0

Hi,

I have tested your issue on Release version 82.0.2, Beta 83.0b5 and latest Nightly 84.0a1 (2020-10-29) using Windows 10 and could not reproduce it. Unfortunately we don't have a Fedora OS in order to test it in a more appropriate environment.
Further, I will move this over to a component so developers can take a look over it. If this is not the correct component please feel free to change it to an appropriate one.

Thanks for the report.

Comment 3

[Dominique Martinet]

I also have the problem on fedora firefox-82.0-5.fc33.x86_64; I think it happened after upgrading from ff 81 time-wise so it's a recent-ish regression, but it could very well be a change in behaviour of google servers instead.

I've debugged a bit through the network logger, and recompiled a curl with http3(quiche) support to replay with custom headers, it looks like google servers don't like some headers e.g. the following two requests fail for me (firefox has both headers set):

./curl --header 'Connection: keep-alive' --http3 'https://www.google.com/search?client=firefox-b-d&q=test+search' -vvv
./curl --header 'TE: Trailers' --http3 'https://www.google.com/search?client=firefox-b-d&q=test+search' -vvv

What i don't understand is that it seems to work in firefox if it sends the same request again, e.g. google servers would refuse when the http3 "connection" has a handshake but accept in further requests? It's fairly reliably, if I idle for a bit then start a google query I will get 400 or a timeout, then if I keep using google it will work rather well.
I tried reproducing the behaviour with curl by sending two requests in the same invocation but I'm not sure it doesn't clean its slate when it does that so it's not very conclusive... anyway I think both firefox and google are wrong here so I'll submit a google "feedback" as well:

• google should be more tolerant in the headers it accepts
• firefox might want not to send keep-alive for http3, unless the header makes sense for it? I'm not sure.

Thanks!

Comment 4

[Kershaw Chang [:kershaw]]

According to spec, we should not sent Connection: keep-alive. For http2, I think we skip the connection header here.
For http3, I am not sure we should do this in necko or neqo.

Like HTTP/2, HTTP/3 does not use the Connection header field to indicate connection-specific fields; in this protocol, connection-specific
metadata is conveyed by other means. An endpoint MUST NOT generate an HTTP/3 field section containing connection-specific fields; any
message containing connection-specific fields MUST be treated as malformed (Section 4.1.3).

Comment 5

[Dominique Martinet]

Ah, thanks for the quote.. so google servers are actually correct if the spec says messages containing these headers must be treated as malformed.

What happened to "Be conservative in what you do, be liberal in what you accept from others" robustness principle :(

Anyway, this isn't the place to debate that -- thanks for your work & keep it up. :)

Comment 6

[Dragana Damjanovic [:dragana]]

Thank you for testing.
All headers mentioned above are excluded in case of HTTP3 as well. Firefox will not send that headers.
This sound more like a different problem.
Can you try to reproduce the issue with Nightly 84? The version of HTTP3 in 82 is not ready for wide use.
I am interested in understanding the problem. Please try 84 and let me know if you can reproduce it.

Comment 7

[Dominique Martinet]

Ok for 82 not being ready yet, that might be all there is to it.

I just tried on nightly (84.0a1 (2020-11-04)), I see the same headers in the network traces -- are they filtered out after being displayed at the protocol level? That is rather misleading, if that's the case I don't know if the headers were really sent in 82 or not.

Here's the headers I see in the request subwindow of the network traces:

GET /search?q=test+search HTTP/3
Host: www.google.com
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:84.0) Gecko/20100101 Firefox/84.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://www.google.com/
DNT: 1
Alt-Used: www.google.com
Connection: keep-alive
Cookie: <removed>
Upgrade-Insecure-Requests: 1
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Fetch-User: ?1

That being said I can't reproduce right now with nightly, so the headers are probably not sent (I still reproduce with 82). I'm not 100% sure given it's intermittent but I'll keep trying from time to time and report if I hit it again.

Comment 8

[undercover_black_hat]

These are the headers of a failed request in 82 (I edited out parts of the cookie):

GET /search?client=firefox-b-d&q=ceva HTTP/3
Host: www.google.com
User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:82.0) Gecko/20100101 Firefox/82.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
DNT: 1
Alt-Used: www.google.com
Connection: keep-alive
Cookie: CGIC=CgtmaXJlZm94LWItZCJKdGV43hodG1sK3htbCxhcHBsaWNhdGlvbi94bWw7cT0wLjksaW1hZ2Uvd2VicCwqLyo7cT0wLjg; NID=204=eiSD7wNwf4vF4JFDQnWWtYVVDaik1x_7ZMm91tpK2utv0bNHYD0LRWhA91rqztxfVUwOaV2_VZxuZBMlvBSASIROzFcCxo7-PZ9qXoFJqcnoTCrDmycPRpIzjKyxDL7-SmaVsWb_RJU7SOueEm8zBIrKOIqFOnow7K-hA_fA4kZp9gs4_Q_S2uQ3D4AL0cwLDwSP9YNQnO2jeG03lXXk3qrH3vpWxXm6uX3IwHPNbMj_c8p1Zlogam_o9vjWyW2bcxqQjw-8HIDgtG11yhN6I6; SID=2wcn5VCpdsLKLpf_i9qzyQI2nz-G2srwfPlIb7L9s1xMaanzxQIOUhCpCLZxM6JA8pPMAQ.; __Secure-3PSID=2wcn5VCpdsLKLpf_i9qzyQI2nz-G2srwfPlIb7L9s1xMaanzDVUu97_ccN64uK1HNbef7w.; HSID=ArinCtFkwxGphA1mz; SSID=A0XGK95yA8aSnStjZ; APISID=aQeWsXop3s3o0xgV/Ag8_VlizKCJHJzZIf; SAPISID=78Me80y_ijbb68oO/AHuqI6Jj8ZbO8fJ1N; _Secure-3PAPISID=78Me80y_ijbb68oO/AHuqI6Jj8ZbO8fJ1N; CONSENT=YES+RO.en+20151207-13-0; SIDCC=AJi4QfE74B6wJYxJ-bYSbfWyS3Snc_2l-okYpG8PCZgeQITPLS9Bsoi2e2eeCGD3uo98qPxgSE; 1P_JAR=2020-11-05-06; __Secure-3PSIDCC=AJi4QfGQDmLXY3H9hzZI89guxFAfyoGNxXOkY10siWNivIor0HR7KpaIguqxGP1g; ANID=AHWqTUljXdf57BZgfRN_Q1vMZVo049HtrF9KZF9ipTFgzpK5N8TZvZ; OTZ=5689327_48_48_123900_44_436380; S=billing-ui-v3=uqze2qZzQS7RyD4Fq1IAMrFFlEU:billing-ui-v3-efe=uqze2D4Fq1IAMrFFlEU; DV=QzOXLEk3q_dCwOHtxWZdw4P5MyRCtwwAAAIAGwtOLQBGLSwAAAKxvDnFuwjx5VQAAAA
Upgrade-Insecure-Requests: 1

Comment 9

[Dominique Martinet]

That being said I can't reproduce right now with nightly, so the headers are probably not sent (I still reproduce with 82). I'm not 100% sure given it's intermittent but I'll keep trying from time to time and report if I hit it again.

Unfortunately I've hit the same error with nightly (still 2020-11-04) -- it didn't happen when testing again ~1h later but I just kept nightly open in the background all this time and this morning when doing a new search I also got the same 400 error.
I'd be tempted to trust the network traces box?

Thanks

Comment 10

[undercover_black_hat]

It happens intermittently on 82 and from what I can tell when the connection goes thru it is actually downgraded to HTTP/2.

On nightly I wasn't able to reproduce it yet, if it does happen it is a lot rarer than on 82.

Comment 11

[robsmith11]

I turned http3 back on with Firefox for Android nightly (84) and after a few hours got a 400 error with YouTube.

Comment 12

[u674127]

Almost the same on Ubuntu. Google-related sites loading forever with a blank screen, or respond with 400 bad request.

Comment 13

[Dragana Damjanovic [:dragana]]

Attachment: Bug 1673231 - Make sure to reset all members.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b4a6d7cdcc59
Make sure to reset all members. r=necko-reviewers,valentin

Comment 15

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/b4a6d7cdcc59