1758870 - Broken TLS connection on POST from JavaScript

Status: ASSIGNED

(NSS :: Libraries, defect, P3)

Description

[peterpz]

Attachment: Issue.zip

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36

Steps to reproduce:

Our current company internal page is working properly in all of the browsers (IE, Chrome, Edge) apart from Firefox. The issue seems to be related to the versions 91.6.0esr and 91.7.0esr (32 bits). I suspect that it doesn't happen in 91.5.0esr, but I'm not sure of that. The situation is that site loads successfully when entered initial URL. Then it stops working when you just work inside this page (click some buttons). The buttons are just forwarding the request to another URLs (within the same domain) but the important thing is that navigation is done by Javascript and on HTTP level it is HTTP POST method.

Actual results:

Then the communicate is just that FF couldn't connect to the site. I attach zip file with all the details, this communicate is displayed in issue.png file (after decompressing it). To make the site work again you just need to go to the address bar and click ENTER because the issue IS NOT happening from address bar, always it is related to the requests send after initial page load, I suspect it always is from JavaScript. The issue is not network or nothing, I did a lot of tests in different browsers and everything is OK. The real issue can be found in the file ff_error.pcapng which shows the TLS exchange. You can see that, there after Change Cipher Spec message the firefox breaks the connection. And on GUI there is no error showing that there was something wrong with the TLS, just "cannot load the page".

I tried to debug a bit more this issue and I found that disabling two algorithms:
TLS_RSA_WITH_AES_256_GCM_SHA384
TLS_RSA_WITH_AES_128_GCM_SHA256

Which is shown in the file ff_workaround.png resolves this issue. Now everything work smoothly.

So I suppose that there is something wrong with those two cipher suites but stil I cannot understand a lot of things:

1. Why those ciphers are ok for FF for initial page load?
2. Why only for JavaScript there is an issue?
3. Also this issue is not 100% deterministic - some JS requests (mainly at the begginig after page load) are working OK but stop working after some time, and always on network there is the same behaviour of breaking TLS connection.

I cannot provide more details, because currently I didn't found the issue in the public visible domains but if I would find it I would post is in edit or comment.

Comment 1

[peterpz]

Attachment: ff_error.pcapng

Comment 2

[peterpz]

Attachment: issue.png

Comment 3

[peterpz]

Attachment: ff_workaround.png

Comment 4

[:Gijs (he/him)]

Dana, can you or someone else who knows this area of gecko take a look? (And does this need to be kept sec-sensitive?)

Comment 5

[peterpz]

One additional info from our side. The server side in this communication is F5 load balancer. That might be important (or not :) ). Today we've disabled on F5 two cipher suites which were detected by me that caused the issue:
16: 157 AES256-GCM-SHA384 256 TLS1.2 Native AES-GCM SHA384 RSA
17: 156 AES128-GCM-SHA256 128 TLS1.2 Native AES-GCM SHA256 RSA

And users confirm that the issue is resolved.

I'm still confused what was the real root cause of the issue, because upgrade to the 91.6.0 esr when we've checked users logs doesn't correlate to the start of the issue.

Comment 6

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Historically, odds are this is a problem with the F5 load balancer, but it could be an NSS issue. Also, this doesn't seem like a security issue.

Comment 7

[peterpz]

If this is related to the F5 load balancer, can I collect any additional logs to confirm what it is exactly?

Comment 8

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

You could set the environment variable SSLKEYLOGFILE to a file to log to and try to connect with Firefox Nightly while capturing packets with Wireshark.

Comment 9

[Dennis Jackson]

Hi Peter,

As Dana mentioned, a pcap with the TLS packets decrypted would be helpful. I'd also be interested in seeing a pcap with those ciphersuites disabled. Although there's likely an issue with the F5 Load Balancer, we shouldn't be tearing down the connection without an alert so there's likely an NSS bug here as well.

Comment 10

[peterpz]

I am trying to firstly generate this issue again, because we've did workaround on F5, so on production this issue is not visible any more. We've rolled back the configuration on our test F5 to see this issue again but because of using different computer - I'm not able to recreate it for now. The issue was visible only on our corporate computers, so I'm currently wondering if this is maybe only visible on some specific OS (corporate Windows 10 with updates aproved by corporate) + Firefox 91.6.0esr (with maybe some corporate edited settings) + specific F5.

Comment 11

[peterpz]

Hello. I attach the PCAP file and the nss.log file with the keys to decrypt traffic. Unfortunately I don't know if this is help, because I don't see anything more in that exchange that wireshark was able to show without keys. The other proper exchange is decrypted properly using those keys, so the keys from NSS are ok (unfortunately I cannot send you all traffic).

Comment 12

[peterpz]

Attachment: ff_error4.pcapng

Comment 13

[peterpz]

Attachment: ssl.log