1797231 - Unicode cookie values can cause permanent denial-of-service attacks on multiple websites

Status: NEW

(Core :: Networking: Cookies, defect, P2)

Description

[April King [:April]]

VERSION
Firefox Version: Firefox 106.0.1 (affects all Firefox) stable
Operating System: macOS 12.6, but affects all operating systems

NOTE
I am reporting this bug after being encouraged to do so by Dan Veditz, and have filed a similar bug with the Chromium team. As it affects both browsers and many websites, please do not make this bug public.

REPRODUCTION CASE

I recently ran into an issue where a piece of JavaScript I was working with was causing issues with the cookie parsing library of the programming language I was using. When it encountered these characters, it would simply fail to parse any further cookies. Not only did this cause issues with code that interacted with the cookie in question, but it also broke any downstream code that relied on the presence of the other cookies.

This led me down a long and twisted road to understand how all browsers (Firefox, Chromium, and Safari) as well as all language libraries (Python, Go, etc.) behave in the presence of the sometimes confusing and conflicting language in RFC 6265. In was in the process of doing this research (please contact me offline for a copy of the manuscript) that I discovered a series of unusual bugs related to the intersection of how browser and language libraries parse cookies.

For this particular issue, I discovered that Firefox allows the following characters inside cookie values: htab, space, dquote, comma, backslash, and 0x80-0xFF + Unicode. While allowing these characters as per RFC 6265bis Section 5 is acceptable, it also seems to cause denial-of-service attacks across numerous websites across the web.

Running this code:

document.cookie='cookieUnicode=🍪';

Will cause many websites to simply fail to work at all. As of my submitting this bug, that includes both facebook.com (which will forever tell you that "something went wrong") and netflix.com (which will also tell you that "something went wrong", with error code NSES-500). The only way to fix an affected user is to either have them manually clear their cookies or to have the receiving web server / website's Javascript enumerate a user's cookies and invalidate them.

In most cases this wouldn't be a huge issue, as it requires code execution to set these "poisoned" cookies in the first place. Unfortunately, cookies are a bit of a special case as due to their cross-origin nature, a subdomain such as poorlysecuredsubdomain.example.com can execute a piece of code such as:

document.cookie='cookieUnicode=🍪; domain=.example.com; path=/';

Which will permanently break the primary site as well as any of its subdomains. This is a significant issue, as many websites today bind their sensitive cookies using host-only (__Host) cookies to TLD+1 and allow subdomains to exist with far poorer security practices. As such, XSS vulnerabilities or subdomain takeovers on poorly secured subdomains can semi-permanent denial-of-service on the primary site.

In my discussions with Dan Veditz (as well as his counterpart at Google), it is a bit unclear as to what the solution to this problem should be. If telemetry shows that not many sites are making use of cookies containing these characters, perhaps the solution is to bar the the creation of any cookies containing 0x00-08, 0x0A-x1F, and 0x7F-FF, including Unicode, and which would apply both to Set-Cookie and to document.cookie.

If the telemetry instead shows that these kinds of cookies commonly exist, then the fix will likely be quite a bit more painful: getting spot fixes across the numerous programming languages, websites, and parsing cookie parsing libraries that make up the web. In only extremely cursory testing, I was able to break both facebook.com and netflix.com, but I suspect the number of affected sites is quite large.

REPLICATION STEPS

Go to facebook.com or netflix.com (or any subdomain)
Open up your console
Run: document.cookie='cookieUnicode=🍪';
Reload the page

Comment 1

[April King [:April]]

With a bit more poking around tonight I found a couple other sites that seem to have permanent denial-of-service errors when encountering a cookie like this: AWS (marketplace, login, probably other pieces), as well as bestbuy.com.

Comment 2

[Daniel Veditz [:dveditz]]

April: please add the link to the Chromium bug so we can reference it when we talk to the Chrome folks.

Comment 3

[Daniel Veditz [:dveditz]]

The cookie emoji is outside the unicode base plane and made me wonder if the sites were trying to parse UTF-8 and choking on surrogate characters. But it seems like any non-ASCII value will trigger the behavior, even simple document.cookie = 'foo=é';

Comment 4

[April King [:April]]

I've been continuing to do research on this, and here are the list of websites I've found that are broken if you have a cookie value containing "🍪".

I'll edit this living comment as time goes along:

• Okta (every sign-in page backed by Okta throws a 400 error)
• Facebook (throws a something went wrong error on every page)
• Instagram (throws a 500 error)
• Netflix (throws an NSES-500 "something went wrong" error)
• Amazon (ratings are broken), gaming.amazon.com (produces 400 errors)
• AWS (can't login, amongst other things)
• Best Buy (searching is broken, throws errors)
• Home Depot (searching is broken, throws errors)
• Wells Fargo (fails to load at all)
• outlook.live.com (400 error on every request)
• eBay (400 errors on about half of their pages)
• zoom.us (400 errors on every request)
• Intuit / TurboTax (4XX errors on every page)
• Target (can neither login nor logout)

Comment 5

[April King [:April]]

I want to add that Okta is broken, so even organizations that don't have websites affected by this bug could still potentially be locked out of all their internal systems.

Comment 6

[Greg Hess]

Following Chrome bug. Valentin to chat with Google align strategies and decide next steps for us.

Comment 7

[Valentin Gosu [:valentin] (he/him)]

Could you CC me ([e-mail] or [e-mail]) on https://bugs.chromium.org/p/chromium/issues/detail?id=1378154 ?
Thanks!

Comment 8

[Valentin Gosu [:valentin] (he/him)]

Attachment: Bug 1797231 - Add telemetry for cookies containing unicode characters r=#necko

Comment 9

[Valentin Gosu [:valentin] (he/him)]

Attachment: Bug 1797231 - Add pref to block unicode chars in cookies r=#necko

Depends on D165953

Comment 10

[Valentin Gosu [:valentin] (he/him)]

Attachment: data-review for bug 1797231.md

Comment 11

[BugBot [:suhaib / :marco/ :calixte]]

There are some r+ patches which didn't land and no activity in this bug for 2 weeks.
:valentin, could you have a look please?
If you still have some work to do, you can add an action "Plan Changes" in Phabricator.
For more information, please visit auto_nag documentation.

Comment 12

[Valentin Gosu [:valentin] (he/him)]

Chris, is it OK to add telemetry in a sec bug? Or should I move that patch to a new one?

Comment 13

[Chris H-C :chutten]

(In reply to Valentin Gosu [:valentin] (he/him) from comment #12)

Chris, is it OK to add telemetry in a sec bug? Or should I move that patch to a new one?

The Data Collection Review (request and response) and the documentation of the instrumentation must both be public to the audience who're having their data collected for the duration it's being collected. The process doesn't need to be conducted in public (though we of course prefer it that way), but its final state and artefacts need to be publicly available. This has resulted in a few models you might wish to copy/adapt:

1. Time-delay Confidentiality: if the reasons and data collection itself shouldn't be revealed before the code collecting it is active, you can file a moco-conf bug for the Data Collection Review and then unconf it when the code ships.
2. See Also public bug: If the Data Review can be public starting immediately, file a public bug and do it there, referencing by number this secbug (without going into sensitive details).
3. Make this secbug public: If this bug can be made public when the instrumentation ships, we could conduct Data Review directly here and then disclose it at the appropriate moment.

I haven't looked up-bug to see which is best applicable here, but Model 2 is the most popular in my experience, if that helps.

None of these models require that the bug/review/etc of landing the instrumentation ever be made public, so go ahead and add your telemetry alongside your impl patches here if you'd like. Just be sure to, as always, get Data Review for new or expanded collections.

Comment 14

[Phabricator Automation]

Comment on attachment 9310655 [details]
Bug 1797231 - Add telemetry for cookies containing unicode characters r=#necko

Revision D165953 was moved to bug 1813469. Setting attachment 9310655 [details] to obsolete.

Comment 15

[Valentin Gosu [:valentin] (he/him)]

Comment on attachment 9311369 [details]
data-review for bug 1797231.md

Obsoletting in fav of attachment 9314792 [details]

Comment 16

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Landed: https://hg.mozilla.org/integration/autoland/rev/89ccb90407c3bb76ca3d6b2e34ccfead1eb1b19d

Backed out 2 changesets (Bug 1813469, Bug 1797231) for causing failures related to TestCookie.BlockUnicode on windows: https://hg.mozilla.org/integration/autoland/rev/18ea4d744ace13d7063a0b7826d9023bacebf8a3

Push with failures
Failure log

Comment 17

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Add pref to block unicode chars in cookies r=necko-reviewers,kershaw
https://hg.mozilla.org/integration/autoland/rev/a2246da1895f9be97e3ca2165274668cc184c70a
https://hg.mozilla.org/mozilla-central/rev/a2246da1895f

Comment 18

[Greg Hess]

Telemetry is just hitting release channel with Fx 111, will monitor to assess potential breakage impact.

Comment 19

[Valentin Gosu [:valentin] (he/him)]

Name	2023-03-09	2023-03-21
None	95,703,129	101,780,065
UnicodeValue	9,571	10,180
UnicodeName	0	0

Glam

It seems forbidding unicode names is quite risk free.
The count of unicode values is quite low (0.01%). Of course, the impact of blocking those values would vary between websites 🤷.
We could either block these cookies and let it ride the trains, or limit that to early beta. Ed, Dan, any thoughts?

Comment 20

[edgul]

The list in comment 4 is concerning by comparison to the low (0.01%) and probably even lower ratio of functional breakage, IMO.

I'm guessing we don't have a way to determine which sites are breaking and inform the owners without additional (and not-so-private) telemetry.
Are we throwing warning/error in devtools to at least give a hint to an affected web developer? Might it be worth the effort to add such a warning?

Comment 21

[Valentin Gosu [:valentin] (he/him)]

(In reply to Ed Guloien [:edgul] from comment #20)

I'm guessing we don't have a way to determine which sites are breaking and inform the owners without additional (and not-so-private) telemetry.

I don't think we can really do that. We could employ a crawler to do that, or maybe add a telemetry flag to say how often a unicode cookie is being set in a alexa top10K site - but even so, I don't know how much that would help.

Are we throwing warning/error in devtools to at least give a hint to an affected web developer? Might it be worth the effort to add such a warning?

Yes. I think it should be obvious what the invalid characters are.

document.cookie="unicode=🙂"
> Cookie “unicode” has been rejected for invalid characters in the value. debugger eval code:1
> "unicode=🙂"

document.cookie="🙂=value"
> Cookie “🙂” has been rejected for invalid characters in the name. debugger eval code:1
> "🙂=value"

I put the patch to let this ride the trains on bug 1827862 so we can add a release note and not make all of the details in this bug public. We can then close this bug as FIXED.

Comment 22

[Daniel Veditz [:dveditz]]

Yes. I think it should be obvious what the invalid characters are.

Will it? This bug and the tests all use cute emojis that stand out, but it's any non-ASCII character and may not be so obvious.

document.cookie = "foo=аре"
 ⚠️ Cookie “foo” has been rejected for invalid characters in the value.

It would be a lot clearer if the message said "non-ASCII" (there's no guarantee they're unicode, even UTF-8 encoded unicode). Plus control chars, plus a couple others -- OK, maybe "invalid" is better than something complicated. We need to explain the non-ASCII addition in release notes so SUMO folks can spread the word for people who are confused.

The telemetry is encouraging, but I'm concerned that Chrome isn't yet blocking these characters (current status of crbug 1378154 is they only recently landed metrics gathering)

A month later and the telemetry still looks more or less like comment 19. Those numbers could be explained by one or two security bug hunters trying lots of cookie values to see what breaks, although if that were the case I'd expect invalid names to show up as well.

Comment 23

[Valentin Gosu [:valentin] (he/him)]

There are some failing tests (including WPT) in Bug 1827862 - Set network.cookie.blockUnicode=true because the expectation is that unicode cookies would work.
I am inclined to wait for a signal from chrome too.

Comment 24

[Valentin Gosu [:valentin] (he/him)]

No signal from Chrome yet, so we'll hold off on landing Bug 1827862 for a while.

Comment 25

[April King [:April]]

Chrome is also seeing a very low rate of these cookies overall (0.01%), but a considerably higher rate in certain countries such as Argentina, Finland, and Mexico. I'm not sure what that will mean for their eventual decision, but I imagine it will slow it down.

Comment 26

[Kershaw Chang [:kershaw]]

Valentin, what do you think?

Comment 27

[Valentin Gosu [:valentin] (he/him)]

I agree. I think this is all about some websites keeping usernames or other user data in cookies that contain non-ascii charactes. As such, this change will disproportionately impact users in some countries, or users whose names contain non-ascii characters.
I do expect this to be a longer process, and I don't think Firefox can take the lead on this given the chance for regressions 🙁.

Comment 28

[Kershaw Chang [:kershaw]]

(In reply to Valentin Gosu [:valentin] (he/him) from comment #27)

I agree. I think this is all about some websites keeping usernames or other user data in cookies that contain non-ascii charactes. As such, this change will disproportionately impact users in some countries, or users whose names contain non-ascii characters.
I do expect this to be a longer process, and I don't think Firefox can take the lead on this given the chance for regressions 🙁.

Thanks, Valentin. Let's keep this in our monitor queue and wait for Chrome.

Comment 29

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:valentin, maybe it's time to close this bug?
For more information, please visit BugBot documentation.

Comment 30

[Valentin Gosu [:valentin] (he/him)]

Unfortunately I don't think we can make the first move here.
Chrome is also monitoring telemetry.
As mentioned earlier, some countries are more heavily impacted by this change than others.

Comment 31

[Valentin Gosu [:valentin] (he/him)]

Based on current telemetry, unicode cookies are still an issue in some parts of the world.
With this being a potential web-compat area of concern, I think we need to wait for Chrome to change behaviour first.

Comment 32

[April King [:April]]

The Chrome folks have pushed this one to the back burner, and I've asked if I can publish my paper since it has been two years since I opened these bugs. Their telemetry shows:

Overall our metrics are showing a bit less than 0.01% of all cookies have some sort of non-ascii unicode character.

However, when I break the data down more I see that some countries have a significantly higher number of clients reporting cookies with non-ascii unicode characters. Countries such as Argentina, Mexico, and Finland have as high as 12% of clients reporting such cookies.

I do see that since opening this bug that network.cookie.blockUnicode has been added as a preference, which is great because presumably it would allow a change to get pushed out.

I've been updating my draft of my article at a new URL, as I work towards updating all the things that have updated over the lifecycle of this bug:

https://grayduck.mn/drafts/handling-cookies-is-a-minefield/

Comment 33

[April King [:April]]

FYI, the article is set to be publish tomorrow morning (CST), after Google unlocks their bug and we can (hopefully) unlock this one.