1064466 - Firefox shares LocalStorage quota across all sub-domains of a domain name

Status: NEW

Description

[kaldari]

Unlike other web browsers, Firefox's LocalStorage quota is set per domain name, not per host. So, for example, de.wikipedia.org and en.wikipedia.org both share a single 5 MB LocalStorage quota. This means that if you use several different language Wikipedias, your LocalStorage quota will quickly be maxed out.

Ideally, the quota should be per host, not per domain name.

See the following thread for another report of the same problem:
https://support.mozilla.org/en-US/questions/963825

Comment 1

[Boris Zbarsky [:bzbarsky, bz on IRC]]

This is purposeful, to avoid people "working around" the quota with sockpuppet subdomains...

Comment 2

[kaldari]

Boris: Couldn't you just enact a 'same-origin' policy? In other words, only allow setting LocalStorage for a particular host from that same host?

Comment 3

[Krinkle]

Does "domain" here use the same definition as the cookie system? https://publicsuffix.org/ (e.g. co.uk is not one domain).

Not sure that should be applied to Storage as hosted services such as:

wordpress.com
stackexchange.com
wikia.com

.. then share one quota for all their instances.

There are plenty of site-specific user interfaces that use local storage (e.g. search suggestions). And larger objects saved by an individual site's admin panel (e.g. saving unsaved drafts in case of browser crash or when offline).

With a practically unlimited number of instances on these sites, there is no reasonable way they can use local storage in Firefox due to having a shared quota (visit enough sites and you'll hit it).

Since 5MB is considered reasonable for a single site, a few admin features together may use that up. And if you're an admin on multiple WordPress blogs, too bad. Quota reached...

And what about hosts like GitHub Pages (github.io), 5minfork.com, your typical No-IP subdomain, temporary tunnel service, etc.

I get the suckpuppet issue, but that only raises the bar on investment a little. One can just get more domains and escalate that way (even for free domains with dot.tk).

--

PS:

Continuing under the assumption that the Public Suffix list is indeed used for this...

Looks like github.io, appspot.com and blogspot.com are actually given special treatment by having an entry on that list. Makes sense security-wise, but may not work well with storage. E.g. Wikipedia, Wikia, WordPress, StackExchange etc. want and do use wildcard cookies for Single sign-on.

Plus, it negatively impacts other companies as their clients end up getting crippled access to local storage. I assume you trust larger parties like github.io and appspot.com to prevent abuse by monitoring their services (and handle abuse reports). But that means smaller organisations are at a disadvantage and have to get special permission first.

For cookies the list makes sense (it restricts the domain, not enable it). But for storage it'd be an enabling feature that essentially requires permission from Mozilla (net neutrality?, j/k *bait*).

Comment 4

[Krinkle]

(In reply to kaldari from comment #2)
> (In reply to Boris Zbarsky [:bz] from comment #1)
> > This is purposeful, to avoid people "working around" the quota with
> > sockpuppet subdomains...
> Boris: Couldn't you just enact a 'same-origin' policy? In other words, only
> allow setting LocalStorage for a particular host from that same host?

I think that's in place already and not what Boris meant.

He means this is to avoid abuse where a single domain is able to store infinite amount of space (well, as much as the browser's internal storage is limited to). E.g. the owner of example.org would set up *.example.org to point at 5mb-pump.example.org, then open lots of Math.random() .example.org's and they keep increasing occupied storage space.

Comment 5

[Olli Pettay [:smaug]]

http://feross.org/fill-disk/ was in the news last year.

Comment 6

[kaldari]

It looks like Chrome rejected adopting Firefox's solution (see comments at https://code.google.com/p/chromium/issues/detail?id=178980) and instead proposed the following strategy:
1) take a (temporary) protective action to avoid too aggressive localStorage usage from a renderer, which could be triggered with a reasonable threshold
2) purge memory for areas that are not used, or do so more aggressively when the usage is getting high, and
3) have proper Quota-like mechanism for managing the total amount of on-disk storage.

Comment 7

[Boris Zbarsky [:bzbarsky, bz on IRC]]

> Does "domain" here use the same definition as the cookie system? https://publicsuffix.org/

Pretty sure it does, though I haven't checked the code recently.

> 3) have proper Quota-like mechanism for managing the total amount of on-disk storage.

This may be the desired end state, yes.  Once we get that figured out, we can reduce the existing restriction, presumably.

Comment 8

[Jan Varga [:janv]]

(In reply to Boris Zbarsky [:bz] from comment #7)
> > Does "domain" here use the same definition as the cookie system? https://publicsuffix.org/
> 
> Pretty sure it does, though I haven't checked the code recently.
> 
> > 3) have proper Quota-like mechanism for managing the total amount of on-disk storage.
> 
> This may be the desired end state, yes.  Once we get that figured out, we
> can reduce the existing restriction, presumably.

I think converting LocalStorage to use temporary storage system would solve this.
It's been our long term plan for some time.

See my comment at:
https://openwebapps.uservoice.com/forums/258478-open-web-apps/suggestions/6268041-a-quote-api-for-local-storage-and-indexeddb

Comment 9

[George Mauer]

I can testify that I'm having issues with this.

We have a client who runs different instances of an application - latestcommit/staging/production - so at least three for each of their clients, all of these are on subdomains of their site. Each instance uses localStorage fairly moderately - some caching and the [persistState widget](https://github.com/togakangaroo/persistState) so less than 500KB per instance. Yet with 3 instances per client, some consultants are now hitting up against limits and there's no simple fix!

Even worse, `localStorage.clear()` seems to only work on the subdomain making this a huge "who the hell thought this was a good idea" problem.

Comment 10

[kaldari]

Yeah, the main reason this bug is frustrating is because there's no easy way to work around it. Clearing your LocalStorage won't fix it (since that only clears one subdomain), and if you use dozens of different subdomains under one site it can take forever to figure out which ones you have to manually clear to get another one to work. It's like playing whack-a-mole. If Firefox is going to share LocalStorage over all the subdomains, it should also provide a mechanism for clearing the LocalStorage from all of them at once. The other solution is Bug 461684, but that's been marked as WONTFIX.

Comment 11

[George Mauer]

So this is the workaround that I've been able to invent. It's so convoluted that you might not even consider it a workaround but it will do in a pinch.

1) On each site have a route that returns a blank page that calls localStorage.clear(); This should not be behind a login
2) Have (or gather into a db) a master list of all subdomains a particular user might visit.
3) If the error condition is suspected, for each other subdomain create a 1px iframe with src="subdomain.yoursite.com/clear-localstorage"

Might make you vomit to do it, but it will work.

Comment 12

[kaldari]

FYI, due to this bug (and the WONTFIX status on bug 461684) we (the Wikimedia Foundation) will be degrading Wikipedia performance for Firefox specifically. Unfortunately, we were not able to come up with any other workaround. The code change is https://gerrit.wikimedia.org/r/#/c/263417/1/resources/src/mediawiki/mediawiki.js.

Comment 13

[Jan Varga [:janv]]

We can't increase the 5 MB limit since we want to discourage developers from using localStorage for larger apps. The problem with Wikipedia is that there are multiple subdomains like en.wikipedia.org, sk.wikipedia.org, etc. The current 5 MB limit applies to eTLD+1 (wikipedia.org) to prevent fill disk attacks by creating hundreds of subdomains.

We are in the process of moving localStorage to quota manager and temporary storage. Once that's done, there won't be the 5 MB limit for eTLD+1, it will only apply for each origin. However localStorage will still be limited by temporary storage limits.

So that should fix the problem besides other things.

Comment 14

[Andre Klapper]

(In reply to Jan Varga [:janv] from comment #13)
> We are in the process of moving localStorage to quota manager and temporary
> storage. Once that's done, there won't be the 5 MB limit for eTLD+1, it will
> only apply for each origin.

Jan: Which bug report should I CC on to follow that process?

Comment 15

[Jan Varga [:janv]]

(In reply to Andre Klapper from comment #14)
> (In reply to Jan Varga [:janv] from comment #13)
> > We are in the process of moving localStorage to quota manager and temporary
> > storage. Once that's done, there won't be the 5 MB limit for eTLD+1, it will
> > only apply for each origin.
> 
> Jan: Which bug report should I CC on to follow that process?

bug 742822

Comment 16

[Maciej Jaros]

> The current 5 MB limit applies to eTLD+1 (wikipedia.org) to prevent fill disk attacks by creating hundreds of subdomains.

So currently for URL `https://client.application-group.company.pl/application/` the quota is shared for whole `company.pl` or for `application-group.company.pl`?