1667801 - Cloudflare reports ESNI is no longer working

Status: RESOLVED WONTFIX

(Core :: Networking: HTTP, defect)

Description

[Kestrel]

STR:

1. Set network.trr.mode = 3 and network.security.esni.enabled = true.
2. Visit https://www.cloudflare.com/ssl/encrypted-sni/
3. Click "Check My Browser".

Expected:

Encrypted SNI: Your browser encrypted the SNI when visiting this page.

Actual:

Encrypted SNI: Your browser did not encrypt the SNI when visiting this page.

Regression range:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=9d511473dace227eddcd9b019f607d0087209406&tochange=4a66e4016c18203c9be3eefdc6888387f737cae2

Regressed by Bug 1652677.

Comment 1

[Kershaw Chang [:kershaw]]

I think we will only support ECH (bug 1654332) in the future.

Comment 3

[Kestrel]

Bug 1654332 is only P3 though, so we may be without encrypted SNI for some time. Is it not possible to follow the normal procedure of preffing off the new implementation and only removing the old code when it is done to avoid a regression?

Comment 4

[Troy Janda]

Seems that until the new code is fully implemented the old code should remain, or users that rely on ESNI might be reluctant to update to 83 until new code is done as in my case.

Comment 5

[uBlock-user]

I think we will only support ECH (bug 1654332) in the future.

Currently websites which're blocked by ISPs and can only be opened via DOH+ESNI no longer work. Shouldn't have removed ESNI. This is an issue.

Comment 6

[Kershaw Chang [:kershaw]]

(In reply to uBlock-user from comment #5)

I think we will only support ECH (bug 1654332) in the future.

Currently websites which're blocked by ISPs and can only be opened via DOH+ESNI no longer work. Shouldn't have removed ESNI. This is an issue.

Thanks for all the comments. We'll add ESNI code back.

Comment 7

[uBlock-user]

Thank you. Will the code be added back in Nightly ?

Comment 8

[Kershaw Chang [:kershaw]]

Yes. The change in bug 1652677 will be reverted.

Comment 9

[uBlock-user]

Fixed in Nightly build 83.0a1 (2020-09-30) (64-bit) 20200930214529

Comment 10

[uBlock-user]

I see this Pulsebot comment, so you brought those commits again and ESNI removed from nightly again ?

Comment 11

[Kershaw Chang [:kershaw]]

(In reply to uBlock-user from comment #10)

I see this Pulsebot comment, so you brought those commits again and ESNI removed from nightly again ?

No, I just add support to echConfig. The ESNI code is not modified.

Comment 12

[Troy Janda]

Just updated the latest nightly and seem to still be fixed in build Id 20201005093214.

Comment 13

[Kershaw Chang [:kershaw]]

NI Kevin to make sure he is aware of the requests here to support ESNI.

Comment 14

[Kershaw Chang [:kershaw]]

Hello,

I'd like to update our current plan about esni and echConfig. That is we will only support echConfig in the future.
esni is supported now, but will be replaced by echConfig when bug 1654332 is done.

Using ni to make sure you are aware of this information.
Thanks.

Comment 15

[uBlock-user]

esni is supported now, but will be replaced by echConfig when bug 1654332 is done.

Will it still function like ESNI does now ?

Comment 16

[Kershaw Chang [:kershaw]]

(In reply to uBlock-user from comment #15)

esni is supported now, but will be replaced by echConfig when bug 1654332 is done.

Will it still function like ESNI does now ?

Yes, I think so.
I'd suggest to take a look at the spec for more details.

Comment 17

[uBlock-user]

I read the spec and it's supposed to encrypt the SNI which is not done by TLS 1.3. So is the difference only in the name though ?

Comment 18

[Kershaw Chang [:kershaw]]

(In reply to uBlock-user from comment #17)

I read the spec and it's supposed to encrypt the SNI which is not done by TLS 1.3. So is the difference only in the name though ?

I think the difference is definitely more than naming, but I am not a NSS expert and I don't want to give wrong information here.

Just a heads-up that bug 1654332 is landed, so esni will be deprecated in a few days.

Comment 19

[Kevin Jacobs [:kjacobs]]

The changes are extensive. In ECH, an entire Client Hello message (containing the private server name) is encrypted and appended as an extension to an unencrypted Client Hello (which contains its own "public" server name).

As :kershaw stated, ESNI as it exists currently will be replaced soon. If you still need the old ESNI, Firefox ESR can be used.

Comment 20

[uBlock-user]

If you still need the old ESNI, Firefox ESR can be used.

As long as it works primarily as ESNI currently does, no need for you guys to give any heads up, unless it won't? ;)

Comment 21

[uBlock-user]

ESNI's no longer working in Nightly, so has it been removed completely as of today ?

Comment 22

[uBlock-user]

Can someone contact the team who implemented ESNI at the server-side at CloudFlare to update the ESNI to ECH draft -08 too, so as to avoid this breakage of functionlity now ?

Comment 23

[Troy Janda]

shouldn't ECH be fully implemented before the SNI is disabled? Makes no sense to remove SNI if the new code is not 100% implemented. So tired of this crap

Comment 24

[Kevin Jacobs [:kjacobs]]

(In reply to uBlock-user from comment #22)

Can someone contact the team who implemented ESNI at the server-side at CloudFlare to update the ESNI to ECH draft -08 too, so as to avoid this breakage of functionlity now ?

I can't speak to Cloudflare's rollout plans, but Cloudflare-go[0] already supports ESNI draft-08 (i.e. ECH), in case you want to run your own server. Draft -09 will soon be published[1] as well. This is the same development process that occurred for ESNI.

[0] https://github.com/cloudflare/go
[1] https://github.com/tlswg/draft-ietf-tls-esni/wiki/Draft--09-Interop

Comment 25

[uBlock-user]

but Cloudflare-go[0] already supports ESNI draft-08 (i.e. ECH) , and -09 will soon be published[1].

I don't know what CloudFlare Go is, but websites that have CloudFlare as their Authoritative DNS provider don't have ECH draft 08 enabled on their side as I'm not able to browse https://torrentz2.is/ and https://www.cloudflare.com/ssl/encrypted-sni/ test now reports that SNI didn't get encrypted on Nightly.

Further proof on https://www.cloudflare.com/cdn-cgi/trace sni is plaintext instead of encrypted. This has now brought back the initial issue I reported in https://bugzilla.mozilla.org/show_bug.cgi?id=1667801#c5 2 months ago, which is very unfortunate.

Comment 26

[rainmakerraw]

(In reply to uBlock-user from comment #25)

but Cloudflare-go[0] already supports ESNI draft-08 (i.e. ECH) , and -09 will soon be published[1].

I don't know what CloudFlare Go is, but websites that have CloudFlare as their Authoritative DNS provider don't have ECH draft 08 enabled on their side as I'm not able to browse https://torrentz2.is/ and https://www.cloudflare.com/ssl/encrypted-sni/ test now reports that SNI didn't get encrypted on Nightly.

Further proof on https://www.cloudflare.com/cdn-cgi/trace sni is plaintext instead of encrypted. This has now brought back the initial issue I reported in https://bugzilla.mozilla.org/show_bug.cgi?id=1667801#c5 2 months ago, which is very unfortunate.

I signed up to reply to this. I'm running the latest firefox-trunk Nightly (85.0a1 (2020-11-29) (64-bit)) on Ubuntu 20.04.1 LTS. Cloudflare reports ESNI isn't working, and indeed I can no longer access the (many) websites blocked by my ISP in the UK.

Launching Firefox v83 (same user.js, with privacy.resistFingerprinting enabled, TRR in mode 3, and ESNI enabled) means all the blocked sites work again. This is marked wontfix but it's a definite regression for the end user. There's surely no point breaking a working privacy/security feature when the replacement isn't fully implemented/working yet. At least have an overlap period so that users aren't disadvantaged (or worse, depending on their nation state of residence). It's all very well saying go back to ESR (or, for now, Stable) but then we *nix users lose out on VAAPI accelerated video playback... One surely shouldn't have to decide on which losses to juggle, for no real gain in return.

fl=21f259
h=www.cloudflare.com
ip=xx.xx.xx.xx
ts=1606688351.761
visit_scheme=https
uag=Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0
colo=LHR
http=http/2
loc=GB
tls=TLSv1.3
sni=plaintext
warp=off
gateway=off

Comment 27

[Masatoshi Kimura [:emk]]

I guess ECH will not work until bug 1678079 is fixed.

Comment 28

[uBlock-user]

(In reply to Masatoshi Kimura [:emk] from comment #27)

I guess ECH will not work until bug 1678079 is fixed.

Can you confirm if the CloudFlare team is still running their draft 01 on their side ? From what I read on GitHub, back in 2019, it was draft 01 indeed.

Comment 29

[Masatoshi Kimura [:emk]]

(In reply to uBlock-user from comment #28)

Can you confirm if the CloudFlare team is still running their draft 01 on their side ? From what I read on GitHub, back in 2019, it was draft 01 indeed.

I don't know. My comment is about our (Firefox) side.

Comment 30

[uBlock-user]

(In reply to Masatoshi Kimura [:emk] from comment #29)

I don't know. My comment is about our (Firefox) side.

Any ETA on the fix for that bug ?

Comment 31

[Masatoshi Kimura [:emk]]

(In reply to uBlock-user from comment #30)

Any ETA on the fix for that bug ?

I don't know. I'm just a volunteer contributor.

Comment 32

[Kestrel]

This upcoming Cloudflare stream might be of interest:

ECH Discussion
December 9, 2020, 5:00 – 5:30 PM (UTC) Live event
Join this special Privacy Week session featuring Cloudflare Research Engineer Chris Patton and Product Manager Wesley Evans discussing ECH.

https://cloudflare.tv/event/3LXGtbK4qmJe5lPbeyK1A8

Comment 33

[uBlock-user]

(In reply to Masatoshi Kimura [:emk] from comment #27)

I guess ECH will not work until bug 1678079 is fixed.

Updated to Nightly just now, no change, SNI is still not encrypted as per https://www.cloudflare.com/ssl/encrypted-sni/ and https://www.cloudflare.com/cdn-cgi/trace

Comment 34

[Masatoshi Kimura [:emk]]

As I already said, I don't know about ETA at Cloudflare's end.

Comment 35

[uBlock-user]

That's not what I'm talking about, you claimed that fixing a specific bug that you mentioned in comment 27 will fix from Firefox's side, but that's not the case apparently.

Comment 36

[Masatoshi Kimura [:emk]]

Does https://www.cloudflare.com/ssl/encrypted-sni/ already support testing ECH? If not, your check does not make sense. It's the reason I mentioned about Cloudflare's schedule.

Comment 37

[Leo_sk]

As a an average user, can someone just tell when DoH with esni or ECH is going to work as intended? I don't want to get into details that sites will have to update etc, just want to know when it works. If 90% of sites that support encrypted server name indication only only support ESNI (draft-08 or whatever), then it is the duty of web browser to support it, even if it is old or inferior method. We did not block every http site when https support was added. Even if number of sites affected in this case is smaller, it makes no sense to change approach

Comment 39

[vernonperry]

This is incredibly bad development practice. You could literally end up getting someone killed depending on where they reside on the planet by pulling ESNI functionality without any warning or notice. How long have people been using 85.0b4 without encrypted ESNI, and with the only indicator that it's broken being inaccessible websites. Absurd.

Comment 40

[prefetcher]

(In reply to vernonperry from comment #39)

This is incredibly bad development practice. You could literally end up getting someone killed depending on where they reside on the planet by pulling ESNI functionality without any warning or notice. How long have people been using 85.0b4 without encrypted ESNI, and with the only indicator that it's broken being inaccessible websites. Absurd.

And why do the developers break a working mechanism without providing any alternative? First of all, they must implement ECH, only then remove ESNI.

Comment 44

[GregoryVPerry]

Ah, and here we go. Instead of issuing an emergency security advisory about how two Mozilla developers just silently disabled production code related to secure browsing - with zero warning to the entire community that relies upon Firefox for private browsing - you would instead just flag critical posts as abuse and spam.

How many hundreds of thousands, if not millions of citizens of this planet did you just compromise with your actions. How many pulls does just jess/firefox get on a day-to-day basis for Dockerized Firefox containers, and that's just one repo. How many OS distributions are tracking nightly Firefox builds, just to have production code for ESNI simply removed without replacement or commentary from the Mozilla team.

Both of you, Masatoshi Kimura and Kershaw Chang are working for the Federal Bureau of Investigation to intentionally weaken the privacy protections of Firefox. Every single code commit you both have made should be scrutinized with a fine tooth comb as I guarantee you've both done way more than this to compromise the integrity of the entire Firefox project.

Comment 45

[phamminhduc0711]

When will this feature work again? I really don't want to downgrade to ESR to lose a bunch of new features (VA-API and screen sharing on Wayland...)

Comment 46

[vernonperry]

(In reply to phamminhduc0711 from comment #45)

When will this feature work again? I really don't want to downgrade to ESR to lose a bunch of new features (VA-API and screen sharing on Wayland...)

There will never be ESNI added back to Firefox, guaranteed. These clowns are working directly with the feds on this, to ensure that all websites visited by the Firefox browser can be easily observed with deep packet content inspection.

Firefox is trash, kowtowing to the feds and whatever other world governments they are now beholden to.

Comment 47

[Mike Hoye [:mhoye]]

That's quite enough of that.

I have locked this bug to editbugs-privileged users only. Consider this is a reminder that Bugzilla is our professional working environment as much as it is our issue tracker, and that sarcasm, hostility, attacking our engineers and repeatedly creating new accounts to avoid bans will not be tolerated.

I encourage everyone involved here to read our community participation guidelines and take them to heart. In the meantime, if you have a contribution to make that can move this issue forward please feel free to email me directly and I would be happy to add your comments or concerns to this thread.

Happy new year, everyone.