1910298 - (CVE-2025-5270) When I successfully enabled encrypted DNS, SNI was sometimes not encrypted

Status: RESOLVED FIXED

(Core :: Networking: DNS, defect, P2)

Description

[xiulou]

Attachment: Screenshots.7z

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 Edg/127.0.0.0
Firefox for Android

Steps to reproduce:

I have enabled an encrypted DNS in Firefox 128.03 and the status says Enabled, indicating that I am using encrypted DNS. Firefox was originally designed to enable DNS-over-https, start ECH, and then SNI will be encrypted, but I tried a few times and found that sometimes SNI is not encrypted.The website I used to detect if SNI was encrypted was https://crypto.cloudflare.com/cdn-cgi/trace/。Another thing is that when I select the default protection in the encrypted DNS, the SNI must be in the plaintext state, but I set the encrypted DNS at the Windows system level, and I can detect that the SNI is encrypted in the Edge, but Firefox with the default protection enabled cannot encrypt the SNI. Isn't Firefox supposed to use the DNS in the system when the default protection is turned on, and why can't SNI be encrypted?

Actual results:

The actual result is that the SNI is in plaintext

Expected results:

The normal situation is that the SNI is encrypted, and then occasionally the SNI is not encrypted, which should be a bug that happens by chance。Attached are some screenshots

Comment 1

[Kershaw Chang [:kershaw]]

Hi reporter,

could you try to record a http log for this?
In about:logging, please select logging to file and send the file to [e-mail] (the log file may contain PII, so please don't upload it to bugzilla).

Thanks.

Comment 2

[xiulou]

Logs have been sent in the past

Comment 3

[Kershaw Chang [:kershaw]]

Thanks for the log.
It shows that Firefox did set ECH to the socket, so it's still not clear why ECH was not working.

2024-07-30 16:02:39.967000 UTC - [Parent 7252: Socket Thread]: D/nsSocketTransport creating nsSocketTransport @1e6f1303a00
2024-07-30 16:02:39.967000 UTC - [Parent 7252: Socket Thread]: E/nsSocketTransport nsSocketTransport::Init [this=1e6f1303a00 host=crypto.cloudflare.com:443 origin=crypto.cloudflare.com:443 proxy=:0]
2024-07-30 16:02:39.967000 UTC - [Parent 7252: Socket Thread]: D/nsHostResolver Resolving host [crypto.cloudflare.com]<^partitionKey=%28https%2Ccloudflare.com%29> type 0. [this=1e6e8bb44c0]
2024-07-30 16:02:39.967000 UTC - [Parent 7252: Socket Thread]: D/nsHostResolver   Offline request for host [crypto.cloudflare.com]; ignoring.
2024-07-30 16:02:39.967000 UTC - [Parent 7252: Socket Thread]: V/nsHttp Setting Socket to use IP hint address
2024-07-30 16:02:39.967000 UTC - [Parent 7252: Socket Thread]: D/nsSocketTransport nsSocketTransport::SetConnectionFlags 1e6f1303a00 flags=73760
2024-07-30 16:02:39.967000 UTC - [Parent 7252: Socket Thread]: D/nsSocketTransport Reset callbacks for tlsSocketInfo=0 callbacks=1e6e8bca7f0
2024-07-30 16:02:39.967000 UTC - [Parent 7252: Socket Thread]: V/nsHttp Setting ECH

Are you able to reproduce this with a clean profile? If yes, could you set the log modules to timestamp,sync,nsHttp:5,nsSocketTransport:5,nsHostResolver:5,pipnss:5 and try to record a log again?
Also, could you open a new tab and type this about:telemetry#histograms-tab_search=ech in address bar and let me know what's the result of SSL_HANDSHAKE_RESULT_ECH?

Thanks.

Comment 4

[xiulou]

I copied those contents into the address bar and hit enter, which is equivalent to searching for them. But it's a pity that I can't send a screenshot here to show you

Comment 5

[xiulou]

I typed about:telemetry#histograms-tab_search=ech into the address bar, and I got some results, but how do I show you the screenshot and send it to my previous email address?

Comment 6

[xiulou]

The log file has been sent to the mailbox again

Comment 7

[Kershaw Chang [:kershaw]]

(In reply to xiulou from comment #5)

I typed about:telemetry#histograms-tab_search=ech into the address bar, and I got some results, but how do I show you the screenshot and send it to my previous email address?

The telemetry results do show that ECH was used successfully, so I am not sure why crypto.cloudflare.com indicates that SNI is not encrypted. I'll forward the screenshot to my colleagues and ask them to take a look.

Comment 8

[xiulou]

In crypto.cloudflare.com this website, it sometimes shows that the SNI is encrypted, sometimes it shows that it is not encrypted, and about 50% of the time it shows that the SNI is not encrypted

Comment 9

[Kershaw Chang [:kershaw]]

(In reply to xiulou from comment #8)

In crypto.cloudflare.com this website, it sometimes shows that the SNI is encrypted, sometimes it shows that it is not encrypted, and about 50% of the time it shows that the SNI is not encrypted

Unfortunately, the logs you sent suggest that ECH was used successfully. It would be very helpful if you could capture a log that shows ECH not working. If you do capture it, please send the log to [e-mail] again. Thanks.

Comment 10

[xiulou]

It's a bit difficult, I've tried several times when I was capturing the logs, and found that the SNI sometimes encrypts successfully and sometimes doesn't, and then just sends the logs over. It stands to reason that there should be a record of SNI and no encryption in the log. You guys can try the DNS I use
https://freedns.controld.com/x-1hosts-pro

Comment 11

[Kershaw Chang [:kershaw]]

(In reply to xiulou from comment #10)

It's a bit difficult, I've tried several times when I was capturing the logs, and found that the SNI sometimes encrypts successfully and sometimes doesn't, and then just sends the logs over. It stands to reason that there should be a record of SNI and no encryption in the log. You guys can try the DNS I use
https://freedns.controld.com/x-1hosts-pro

Unfortunately, I still can't reproduce this issue. Since you mentioned there's about a 50% chance of reproducing it, could you start logging immediately after Firefox starts and then stop logging when you notice the SNI is not encrypted?

Thanks.

Comment 12

[xiulou]

I'll just try one last time

Comment 13

[xiulou]

I've sent the log file to your mailbox

Comment 14

[Kershaw Chang [:kershaw]]

I think this might be related to speculative connection.

Could you try to open about:config and set network.http.speculative-parallel-limit to 0 and see if you can still reproduce this?
Thanks.

Comment 15

[xiulou]

I did as you said, and there is still a slight probability that SNI is not encrypted

Comment 16

[Kershaw Chang [:kershaw]]

(In reply to xiulou from comment #15)

I did as you said, and there is still a slight probability that SNI is not encrypted

It'd be very helpful if you could capture a http log again.
Could you help?
Thanks.

Comment 17

[xiulou]

I've sent the log file to your email. You check it out

Comment 18

[Kershaw Chang [:kershaw]]

Attachment: Bug 1910298 - Make EchConfig a part of the connection hash key, r=#necko

By including ECH as part of the connection hash key, we ensure that connections with ECH are not mixed with others.

Comment 19

[Kershaw Chang [:kershaw]]

Sorry for the delay.

I wasn’t 100% convinced that we needed this patch until I encountered bug 1947968. While reproducing the bug, I discovered a scenario where Firefox could not use ECH. This issue occurs when we receive an Alt-svc header and create a speculative connection to verify it.

2025-02-13 12:50:10.435071 UTC - [Parent 15999: Main Thread]: D/nsHttp AltSvcMapping ctor 11027c100 https://encryptedsni.com:443 to :443
2025-02-13 12:50:10.435076 UTC - [Parent 15999: Main Thread]: D/nsHttp AltSvcCache::LookupMapping 111254280 https:encryptedsni.com:443:.:^partitionKey=%28https%2Cencryptedsni.com%29:3
2025-02-13 12:50:10.435079 UTC - [Parent 15999: Main Thread]: D/nsHttp AltSvcCache::LookupMapping 111254280 MISS
2025-02-13 12:50:10.435081 UTC - [Parent 15999: Main Thread]: D/nsHttp AltSvcCache::UpdateAltServiceMapping 111254280 map 11027c100 existing 0 encryptedsni.com validated=0
2025-02-13 12:50:10.435092 UTC - [Parent 15999: Main Thread]: V/nsHttp Init nsHttpConnectionInfo @1613f13f0
2025-02-13 12:50:10.435098 UTC - [Parent 15999: Main Thread]: D/nsHttp AltSvcCache::UpdateAltServiceMapping 111254280 validation via speculative connect started
2025-02-13 12:50:10.435101 UTC - [Parent 15999: Main Thread]: D/nsHttp AltSvcMappingValidator ctor 10e947040 map 11027c100 [encryptedsni.com -> encryptedsni.com]

In AltSvcTransaction, we directly attempt to create an HTTP/3 connection without first performing a lookup for an HTTPS record, which results in the new connection not using ECH.

2025-02-13 12:50:10.436107 UTC - [Parent 15999: Socket Thread]: I/nsHttp Http3Session::Init origin=encryptedsni.com, alpn=h3, selfAddr=0.0.0.0, peerAddr=104.22.73.170, qpack table size=65536, max blocked streams=20 webtransport=0 [this=15d4aab00]
2025-02-13 12:50:10.436228 UTC - [Parent 15999: Socket Thread]: V/nsHttp Http3Session::DoSetEchConfig 15d4aab00 of length 0

I believe it would still be worthwhile to include ECH as part of the connection hash key. This way, whenever ECH changes, we’ll always create a new connection.

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ebe2678e73a4
Make EchConfig a part of the connection hash key, r=necko-reviewers,valentin

Comment 21

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/ebe2678e73a4

Comment 22

[Donal Meehan [:dmeehan]]

:kershaw could you add uplift requests for esr128 and beta?

Comment 23

[Kershaw Chang [:kershaw]]

Attachment: Bug 1910298 - Make EchConfig a part of the connection hash key, r=#necko

Original Revision: https://phabricator.services.mozilla.com/D231355

Comment 24

[Phabricator Automation]

esr128 Uplift Approval Request

• User impact if declined: In some rare cases, ECH might not be used for connections.
• Code covered by automated testing: yes
• Fix verified in Nightly: yes
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: N/A
• Risk associated with taking this patch: Low
• Explanation of risk level: Low risk, since this patch is straightforward.
• String changes made/needed: N/A
• Is Android affected?: yes

Comment 25

[Kershaw Chang [:kershaw]]

I’m only requesting an ESR128 uplift since I believe this patch fixes only some corner cases, not common ones. It should be fine to let this patch ride the train to release.

Comment 26

[BugBot [:suhaib / :marco]]

The patch landed in nightly and beta is affected.
:kershaw, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
• If no, please set status-firefox136 to wontfix.

For more information, please visit BugBot documentation.

Comment 27

[Ryan VanderMeulen [:RyanVM]]

Is this ready for ESR128 uplift now?

Comment 28

[Kershaw Chang [:kershaw]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #27)

Is this ready for ESR128 uplift now?

No. Apparently this patch caused a regression, so we plan to revert this change.
We'll try a different approach and put this behind a pref to mitigate the risk.

Comment 29

[Pascal Chevrel (relman team) -> :pascalc]

Reverted in https://hg.mozilla.org/releases/mozilla-beta/rev/6318cabd4f5c

Comment 30

[Kershaw Chang [:kershaw]]

The ECH issue mentioned in comment #0 should be resolved by bug 1953078, so I think we can close this bug.

The patch that landed (and was later reverted) attempted to include the ECH key as part of the connection info hash. However, it introduced a regression in the ECH success rate.

Since the main issue is already addressed, I suggest we close this bug and file a new one to investigate the ECH success rate regression.

Comment 31

[Jesse Schwartzentruber (:truber)]

Attachment: advisory.txt

Comment 32

[Jesse Schwartzentruber (:truber)]

Attachment: advisory.txt