1895659 - SSL_ERROR_RX_MALFORMED_SERVER_HELLO error on multiple websites with no clear pattern

Status: RESOLVED WORKSFORME

(Core :: Security: PSM, defect)

Description

[Aadhithya N.M]

Attachment: XHR Network requests made upon refreshing the app.schildi.chat webpage. Note the errors when it tries to "phone" to any matrix.org domain.

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0

Steps to reproduce:

1. Load Firefox Troubleshoot Mode, or Firefox on a new profile
2. Attempt to navigate to any of the following websites by just entering any of the following affected URLs (problematic websites from here on out):

https://broadband.forum/
https://cloudflare.com/
https://matrix.org/
https://nodejs.org/
https://element.io/

Trying to load the above pages in Firefox Android, on a device connected to the same Wi-FI network as the PC, yields no such error.
Trying to load the above pages in Google Chrome for PC (124.0.6367.119 (Official Build) (64-bit) (cohort: Stable)) Yields no such error on the same exact PC that is running the problematic Firefox.

The following about:config tweaks, gotten from this Reddit post ( https://www.reddit.com/r/firefox/comments/em250h/ssl_error_rx_malformed_server_hello_with_the_new/) Do not fix the issue:

setting security.osclientcerts.autoload to true
setting security.enterprise_roots.enabled to true
setting security.tls.hello_downgrade_check to false
setting security.tls.version.max from 4 to 3*

• This causes the above problematic websites to infinitely load/buffer.

This also seems to affect XHR requests: Matrix web clients such as SchildiChat (https://app.schildi.chat), Cinny (https://app.cinny.in), FluffyChat (https://fluffychat.im/web/) are unable to connect to the matrix.org homeserver, where my account is registered.*

Actual results:

I recieve a SSL_ERROR_RX_MALFORMED_SERVER_HELLO code error. "An error occurred during a connection to nodejs.org. SSL received a malformed Server Hello handshake message." Upon trying to directly connect to the problematic website.

Refreshing the page does not fix this.

*E.g The Matrix protocol web client, SchildiChat (https://app.schildi.chat) attempts to contact matrix.org when trying to log the user into their account but experiences "SSL Errors". See the attached image for the Firefox Developer Tools Network requests made when refreshing the page.

Expected results:

The problematic websites should have loaded normally, as they did until around 2-3 weeks ago.

Comment 1

[Aadhithya N.M]

Attachment: about:support troubleshooting data

Comment 2

[Aadhithya N.M]

Another problematic site:

https://readthedocs.io/

A new bit of information: This error does not occur on another PC on the same Wi-FI network, running the same Firefox Version and same OS (Windows 11).

Comment 3

[BugBot [:suhaib / :marco]]

The Bugbug bot thinks this bug should belong to the 'Core::Security: PSM' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 4

[Dennis Jackson]

Hi,

Your tests suggest there is a configuration problem with the OS or Firefox rather than the network. Is the impacted PC configured to use any kind of proxy, VPN or other security software? Another thing to try might be installing Firefox Extended Support Release to rule out any recent code changes in Firefox as a cause, although this seems unlikely given the nature of the error.

Comment 5

[Aadhithya N.M]

The impacted PC is not configured to use any proxy or VPN.
Firefox Troubleshoot Mode, running in the bone stock factory settings with no config changes, no addons, no settings changes etc, exhibits the same problems. Thus it is not a Firefox configuration error on my end.
Google Chrome running on the same PC and same OS exhibits no such problems. Thus it is not a OS/System configuration error on my end.
Firefox Troubleshoot Mode on a separate PC in the same Wifi network exhibits no such problems. Thus it is not a network issue on my end.

Comment 6

[Aadhithya N.M]

Tried Firefox ESR 115 from the given link. The same issue occus.

Comment 7

[Aadhithya N.M]

The error has suddenly disappeared for me. Should I close this ticket?

Comment 8

[Dennis Jackson]

Thanks for testing with ESR Aadhithya. It seems most likely this was due to a network issue of some kind. We'll close this for now, but if the issue returns please feel free to reopen it.

Comment 9

[Aadhithya N.M]

The following websites no longer exhibit the issue:
https://broadband.forum/
https://cloudflare.com/
https://matrix.org/
https://nodejs.org/
https://element.io/
https://readthedocs.io/

This issue still occurs on select sites, e.g:
https://www.explainxkcd.com

Comment 10

[BugBot [:suhaib / :marco]]

The severity field is not set for this bug.
:keeler, could you have a look please?

For more information, please visit BugBot documentation.

Comment 11

[Dana Keeler (she/her) [:keeler]]

Can you use wireshark to capture packets from a connection that fails and attach the results here or email them to me? Thanks!

Comment 12

[Aadhithya N.M]

Hi Dana,

Unfortunately my PC's drive was recently corrupted and I had to install Linux to a different drive. So I cannot reproduce the bug on Windows for now.

However, I seem to be experiencing a similar bug on select sites in Linux, namely "PR_END_OF_FILE_ERROR". The same exact text, complaining that "the authenticity of the recieved data could not be verified", is present on the error page. I suspect it is the same bug across both Windows and Linux, manifesting in different forms.

Further, the problematic sites keep rotating (i.e https://www.explainxkcd.com seems to work now), but the problematic pages do not seem to be saved in my browser history, making them hard to find after the fact.

I will email you with the output of my system inxi -F now. And I will email you with a wireshark capture of a site that fails to load for me, ASAP.

Comment 13

[Aadhithya N.M]

I am simply not able to reproduce this bug on Linux or Windows. Closing as resolved for now.