Open Bug 1064466 Opened 6 years ago Updated 3 months ago

Firefox shares LocalStorage quota across all sub-domains of a domain name

Categories

(Core :: Storage: localStorage & sessionStorage, defect, P3)

Product:
Core
Component:
Storage: localStorage & sessionStorage
Platform:
x86
macOS
Type:
defect

Tracking

()

People

(Reporter: kaldari, Unassigned)

References

(Depends on 2 open bugs, Blocks 1 open bug)

Details

Unlike other web browsers, Firefox's LocalStorage quota is set per domain name, not per host. So, for example, de.wikipedia.org and en.wikipedia.org both share a single 5 MB LocalStorage quota. This means that if you use several different language Wikipedias, your LocalStorage quota will quickly be maxed out.

Ideally, the quota should be per host, not per domain name.

See the following thread for another report of the same problem:
https://support.mozilla.org/en-US/questions/963825
See Also: → 461684
This is purposeful, to avoid people "working around" the quota with sockpuppet subdomains...
Boris: Couldn't you just enact a 'same-origin' policy? In other words, only allow setting LocalStorage for a particular host from that same host?
Does "domain" here use the same definition as the cookie system? https://publicsuffix.org/ (e.g. co.uk is not one domain).

Not sure that should be applied to Storage as hosted services such as:

wordpress.com
stackexchange.com
wikia.com

.. then share one quota for all their instances.

There are plenty of site-specific user interfaces that use local storage (e.g. search suggestions). And larger objects saved by an individual site's admin panel (e.g. saving unsaved drafts in case of browser crash or when offline).

With a practically unlimited number of instances on these sites, there is no reasonable way they can use local storage in Firefox due to having a shared quota (visit enough sites and you'll hit it).

Since 5MB is considered reasonable for a single site, a few admin features together may use that up. And if you're an admin on multiple WordPress blogs, too bad. Quota reached...

And what about hosts like GitHub Pages (github.io), 5minfork.com, your typical No-IP subdomain, temporary tunnel service, etc.

I get the suckpuppet issue, but that only raises the bar on investment a little. One can just get more domains and escalate that way (even for free domains with dot.tk).

--

PS:

Continuing under the assumption that the Public Suffix list is indeed used for this...

Looks like github.io, appspot.com and blogspot.com are actually given special treatment by having an entry on that list. Makes sense security-wise, but may not work well with storage. E.g. Wikipedia, Wikia, WordPress, StackExchange etc. want and do use wildcard cookies for Single sign-on.

Plus, it negatively impacts other companies as their clients end up getting crippled access to local storage. I assume you trust larger parties like github.io and appspot.com to prevent abuse by monitoring their services (and handle abuse reports). But that means smaller organisations are at a disadvantage and have to get special permission first.

For cookies the list makes sense (it restricts the domain, not enable it). But for storage it'd be an enabling feature that essentially requires permission from Mozilla (net neutrality?, j/k *bait*).
(In reply to kaldari from comment #2)
> (In reply to Boris Zbarsky [:bz] from comment #1)
> > This is purposeful, to avoid people "working around" the quota with
> > sockpuppet subdomains...
> Boris: Couldn't you just enact a 'same-origin' policy? In other words, only
> allow setting LocalStorage for a particular host from that same host?

I think that's in place already and not what Boris meant.

He means this is to avoid abuse where a single domain is able to store infinite amount of space (well, as much as the browser's internal storage is limited to). E.g. the owner of example.org would set up *.example.org to point at 5mb-pump.example.org, then open lots of Math.random() .example.org's and they keep increasing occupied storage space.
http://feross.org/fill-disk/ was in the news last year.
It looks like Chrome rejected adopting Firefox's solution (see comments at https://code.google.com/p/chromium/issues/detail?id=178980) and instead proposed the following strategy:
1) take a (temporary) protective action to avoid too aggressive localStorage usage from a renderer, which could be triggered with a reasonable threshold
2) purge memory for areas that are not used, or do so more aggressively when the usage is getting high, and
3) have proper Quota-like mechanism for managing the total amount of on-disk storage.
> Does "domain" here use the same definition as the cookie system? https://publicsuffix.org/

Pretty sure it does, though I haven't checked the code recently.

> 3) have proper Quota-like mechanism for managing the total amount of on-disk storage.

This may be the desired end state, yes.  Once we get that figured out, we can reduce the existing restriction, presumably.
(In reply to Boris Zbarsky [:bz] from comment #7)
> > Does "domain" here use the same definition as the cookie system? https://publicsuffix.org/
> 
> Pretty sure it does, though I haven't checked the code recently.
> 
> > 3) have proper Quota-like mechanism for managing the total amount of on-disk storage.
> 
> This may be the desired end state, yes.  Once we get that figured out, we
> can reduce the existing restriction, presumably.

I think converting LocalStorage to use temporary storage system would solve this.
It's been our long term plan for some time.

See my comment at:
https://openwebapps.uservoice.com/forums/258478-open-web-apps/suggestions/6268041-a-quote-api-for-local-storage-and-indexeddb
I can testify that I'm having issues with this.

We have a client who runs different instances of an application - latestcommit/staging/production - so at least three for each of their clients, all of these are on subdomains of their site. Each instance uses localStorage fairly moderately - some caching and the [persistState widget](https://github.com/togakangaroo/persistState) so less than 500KB per instance. Yet with 3 instances per client, some consultants are now hitting up against limits and there's no simple fix!

Even worse, `localStorage.clear()` seems to only work on the subdomain making this a huge "who the hell thought this was a good idea" problem.
Yeah, the main reason this bug is frustrating is because there's no easy way to work around it. Clearing your LocalStorage won't fix it (since that only clears one subdomain), and if you use dozens of different subdomains under one site it can take forever to figure out which ones you have to manually clear to get another one to work. It's like playing whack-a-mole. If Firefox is going to share LocalStorage over all the subdomains, it should also provide a mechanism for clearing the LocalStorage from all of them at once. The other solution is Bug 461684, but that's been marked as WONTFIX.
So this is the workaround that I've been able to invent. It's so convoluted that you might not even consider it a workaround but it will do in a pinch.

1) On each site have a route that returns a blank page that calls localStorage.clear(); This should not be behind a login
2) Have (or gather into a db) a master list of all subdomains a particular user might visit.
3) If the error condition is suspected, for each other subdomain create a 1px iframe with src="subdomain.yoursite.com/clear-localstorage"

Might make you vomit to do it, but it will work.
FYI, due to this bug (and the WONTFIX status on bug 461684) we (the Wikimedia Foundation) will be degrading Wikipedia performance for Firefox specifically. Unfortunately, we were not able to come up with any other workaround. The code change is https://gerrit.wikimedia.org/r/#/c/263417/1/resources/src/mediawiki/mediawiki.js.
We can't increase the 5 MB limit since we want to discourage developers from using localStorage for larger apps. The problem with Wikipedia is that there are multiple subdomains like en.wikipedia.org, sk.wikipedia.org, etc. The current 5 MB limit applies to eTLD+1 (wikipedia.org) to prevent fill disk attacks by creating hundreds of subdomains.

We are in the process of moving localStorage to quota manager and temporary storage. Once that's done, there won't be the 5 MB limit for eTLD+1, it will only apply for each origin. However localStorage will still be limited by temporary storage limits.

So that should fix the problem besides other things.
(In reply to Jan Varga [:janv] from comment #13)
> We are in the process of moving localStorage to quota manager and temporary
> storage. Once that's done, there won't be the 5 MB limit for eTLD+1, it will
> only apply for each origin.

Jan: Which bug report should I CC on to follow that process?
(In reply to Andre Klapper from comment #14)
> (In reply to Jan Varga [:janv] from comment #13)
> > We are in the process of moving localStorage to quota manager and temporary
> > storage. Once that's done, there won't be the 5 MB limit for eTLD+1, it will
> > only apply for each origin.
> 
> Jan: Which bug report should I CC on to follow that process?

bug 742822
Depends on: 742822
> The current 5 MB limit applies to eTLD+1 (wikipedia.org) to prevent fill disk attacks by creating hundreds of subdomains.

So currently for URL `https://client.application-group.company.pl/application/` the quota is shared for whole `company.pl` or for `application-group.company.pl`?
Depends on: 1305665
Priority: -- → P3
Depends on: 1286798
Blocks: 1504142
Component: DOM → DOM: Core & HTML

This is starting to become an issue for other major sites as well, such as Reddit, which has quite a number of subdomains nowadays and all subdomains (new/old/mod/etc.) share the same LocalStorage quota.

We are going to ship a new LocalStorage implementation in FF 68 or 69 which doesn't have this problem.
See bug 1539835.

(In reply to Jan Varga [:janv] from comment #18)

We are going to ship a new LocalStorage implementation in FF 68 or 69 which doesn't have this problem.
See bug 1539835.

I'm running Aurora/DevEdition 68.0b5 (64-bit) with dom.storage.next_gen set to true, though.

What does that mean ?
Does it work well for you in 68.0b5 ?
You should have 2GB of quota for entire eTDL+1 domain.

(In reply to Jan Varga [:janv] from comment #20)

What does that mean ?
Does it work well for you in 68.0b5 ?
You should have 2GB of quota for entire eTlD+1 domain.

Then it doesn't seem to be working in 68.0b5. LocalStorage on reddit reports being capable of writing a total of 5243043 bytes (±5MB)

There's still the 5MB limit for each origin. However, if there are multiple subdomains they are no longer limited by the 5MB limit as a group.
You can verify that on http://www.filldisk.com/

(In reply to Jan Varga [:janv] from comment #22)

There's still the 5MB limit for each origin. However, if there are multiple subdomains they are no longer limited by the 5MB limit as a group.
You can verify that on http://www.filldisk.com/

There seems to be something not entirely right with that though. here is what I saw yesterday running this testscript I made with the same firefox version (68.0b5, though it is possible it has since updated of course) and the same flag enabled.

LS now shares quota with IndexedDB and DOM cache, so you may have too much data stored using these two APIs.

You can check "Cookies and Site Data" in preferences to see how much space is used by your origin.

You can also use the estimate() method to see how much space is available.
See https://storage.spec.whatwg.org/

(In reply to Jan Varga [:janv] from comment #24)

LS now shares quota with IndexedDB and DOM cache, so you may have too much data stored using these two APIs.

You can check "Cookies and Site Data" in preferences to see how much space is used by your origin.

You can also use the estimate() method to see how much space is available.
See https://storage.spec.whatwg.org/

Executing

navigator.storage.estimate().then(info => { 
   console.log(info) 
});

gives me

quota: 2147483648
​usage: 3073184

checking firefox settings shows me this.

Then, it's a bug. Are you able to reproduce it, or it happened only once ?

Pretty consistently, in fact rctgamer3 was having the same issue today which is how I arrived here as he was talking with me about it as I helped him troubleshoot it.

Do you have time to create a test case for this ?
Or provide steps to reproduce.

I'll try to write them up, what I meant is that I have had it happen consistently but the steps itself aren't precise as it basically amounts to browsing reddit.com from a variety of domains for a little while. I'll see if I can make it a bit less abstract and more useful.

Component: DOM: Core & HTML → DOM: Web Storage

Btw, does it happen with private browsing only ?

Also, does it happen if you set the pref "dom.storage.snapshot_reusing" to false ?

I have had it happen one time today, unfortunately making reliable reproduction steps is proofing to be more difficult than I thought. I'll continue to try.

(In reply to Jan Varga [:janv] from comment #30)

Btw, does it happen with private browsing only ?

It happens with regular browsing, no private browsing involved.

(In reply to Jan Varga [:janv] from comment #31)

Also, does it happen if you set the pref "dom.storage.snapshot_reusing" to false ?

I'll have to try that out.

(In reply to Jan Varga [:janv] from comment #31)

Also, does it happen if you set the pref "dom.storage.snapshot_reusing" to false ?

It appears that it doesn't. I have been using this flag at home for a week now and did not experience the issue. My firefox installation at work does not have the flag set I did run into the same issue today under otherwise identical circumstances.

Ok, that narrows down it a bit. Thank you!

What is the status of this? It looks like it was enabled in Firefox Beta briefly, but then reverted. Is that correct?

IIUC, https://bugzilla.mozilla.org/show_bug.cgi?id=1064466#c18 implies this was fixed, then reverted in https://bugzilla.mozilla.org/show_bug.cgi?id=1592136

Do Mozilla engineers have any plans to approach this task again rather sooner than later?

See Also: https://bugzilla.wikimedia.org/show_bug.cgi?id=64721 → https://phabricator.wikimedia.org/T66721

LocalStorage NextGen (aka LSNG, which mitigates the problem) is still present in the code-base and it's a high priority goal to re-enable it and have that ride the release trains to release. The issue is that LocalStorage NextGen depends on the QuotaManager subsystem that also underpins IndexedDB/Cache API/ServiceWorkers and we encountered high rates of breakage in the wild for a variety of reasons. The breakage has been greatly reduced through a series of fixes and enhancements, but we're being conservative about re-enabling until our telemetry indicates that the problem has been dealt with.

Once LSNG has been released for a few cycles we will also undertake the eviction of old LocalStorage data from origins that are no longer used.

And as an aside about the use-case of caching JS in LocalStorage, we'd advise only using the Cache API or just letting the Firefox HTTP cache store JS, since those mechanisms allow the JS engine to cache the parsed bytecode and won't jank cold pageloads for the site, etc. Obviously, for users whose QuotaManager storage is broken, this won't work, but as those issues are corrected, it will start working. (Also, users can "fix" their installation by going to about:support at any time and choosing "Refresh Firefox...")

You need to log in before you can comment on or make changes to this bug.