1739673 - Intermittent gmail POST requests with very long TLS setup times 58-59 seconds - 0-RTT

Status: RESOLVED DUPLICATE of bug 1735864

(Core :: Networking, defect)

Description

[Nick Alexander :nalexander [he/him]]

Intermittently, when I click on a folder in gmail.com (with my mozilla.com address), I see the yellow "Loading..." banner for a very long time before it succeeds. Subsequent requests seem normal.

I've taken to leaving the devtools networking panel open, and I have witnessed two instances with a TLS setup time of 58s. This seems excessive. I have saved a HAR archive, but it appears to include cookies, etc, so I can't easily share it. I see nothing about timings in the HAR, but I also didn't look too hard. Both requests were

POST https://mail.google.com/sync/u/1/i/s?hl=en&c=181

but I can't guarantee the c=181 parameter was the same.

I raised this issue on (private) Slack at https://mozilla.slack.com/archives/C4D9J9KGA/p1635181131001000. I was asked if I was using DoH. For the first slow TLS setup, I was not using DoH. I am in Canada, so I expected to be using DoH, but I was not at first; I subsequently enabled DoH. For the second slow TLS setup, I was using DoH.

I was asked if I was using HTTP/2 or HTTP/3. Per the networking tools, it is reported as version HTTP/2.

Since this issue is intermittent, it's not easy for me to capture network logs or a profile, but I will certainly try.

Comment 1

[tors42]

I too have seen "TLS Setup" times of 58 and 59 seconds in Firefox 94.0,
when solving chess puzzles at https://lichess.org/training

I couldn't reproduce it without being logged in to an account (free) on the site.
The immediate difference I can think of when beeing logged in is that one gets a
puzzle rating number which is displayed (when being logged in),
but a POST is still sent when not being logged in - so yeah - not sure what
makes the difference.

The problem seems to happen when I take more than 3 minutes to solve a puzzle,
i.e I just load the page and look at it for 3 minutes without making any input,
just about then the initial TCP connection will be closed.

And when I then make my attempt to solve the puzzle after this time has passed,
the resulting POST request being sent will now set up a new connection - and that takes this additional 58-59 seconds to establish.

One thing I see in packet capture (screenshot https://i.imgur.com/5r2KhZF.png),
is that the initial TLS handshake goes:

1 Client Hello
2 Server Hello, Change Cipher Spec
3 Change Cipher Spec

And when re-establishing the connection, it looks a bit different:

1 Client Hello
2 Change Cipher Spec
3 Server Hello, Change Cipher Spec

And after this there are 5 Keep-Alive packets sent,
10 seconds apart - seemingly adding up the "TLS Setup" time...

Screenshot of developer tools (not from same run, but I don't see them sharing any info) https://i.imgur.com/Zz2VJtt.png

Comment 3

[Martin Thomson [:mt:]]

I'm not sure that duplicating this to an SNMP bug is correct; gmail only uses HTTP. This seems to be related to Google's rollout of TLS early data (aka 0-RTT).

We've also had reports about connections that stall in Bug 1738102. That's not TCP and TLS, but QUIC, but if the problem is in the interaction between our early data logic and the way we poll() sockets, then the two have a similar cause.

Comment 4

[Richard Leger]

(In reply to Martin Thomson [:mt:] from comment #3)

I'm not sure that duplicating this to an SNMP bug is correct;

Bug 1735864 is not related to SMTP bug but to a TLS Setup delay as reported here in Comment 1, which is a Core > Security or Core > Networking issue, that is why I believe this bug is duplicate.

gmail only uses HTTP. This seems to be related to Google's rollout of TLS early data (aka 0-RTT).

The problem is likely related to the way Thunderbird and Firefox handle TLS/Handshake since recently... See Bug 1735864 Comment 64 where we have unfolded a list of handshake related patches recently pushed to Firefox > Core > Security... those are the clues unfolded for now...

We've also had reports about connections that stall in Bug 1738102. That's not TCP and TLS, but QUIC, but if the problem is in the interaction between our early data logic and the way we poll() sockets, then the two have a similar cause.

Bug 1738102 may likely be a regression related to one of those patches... but I don't know enough to know for sure... but it looks like it...

Comment 5

[Richard Leger]

Hi Martin,

(In reply to Richard Leger from comment #4)

(In reply to Martin Thomson [:mt:] from comment #3)

This seems to be related to Google's rollout of TLS early data (aka 0-RTT).

The problem is likely related to the way Thunderbird and Firefox handle TLS/Handshake since recently... See Bug 1735864 Comment 64 where we have unfolded a list of handshake related patches recently pushed to Firefox > Core > Security... those are the clues unfolded for now...

It looks like you got it right.

According to Bug 1735864 Comment 76 and Bug 1735864 Comment 76 it seems linked to 0-RTT:

"...
possibly it was a change wrt. 0rtt after changeset: 70cb14dba583380ec9a7a797ca7ee005bd9586f1

(...)

Possible workaround:
security.tls.enable_0rtt_data = FALSE
network.http.http3.enable_0rtt = FALSE

(...)

An interesting article on 0rtt here https://www.privateinternetaccess.com/blog/supercookey-a-supercookie-built-into-tls-1-2-and-1-3
..."

Regards,

Comment 6

[Andreas Wagner [:TheOne] [use NI]]

(In reply to Richard Leger from comment #5)

Possible workaround:
security.tls.enable_0rtt_data = FALSE
network.http.http3.enable_0rtt = FALSE

I have flipped those two prefs and haven't experienced the issue ever since.