1064466 - Firefox shares LocalStorage quota across all sub-domains of a domain name

Status: RESOLVED FIXED

(Core :: Storage: localStorage & sessionStorage, defect, P3)

Description

[kaldari]

Unlike other web browsers, Firefox's LocalStorage quota is set per domain name, not per host. So, for example, de.wikipedia.org and en.wikipedia.org both share a single 5 MB LocalStorage quota. This means that if you use several different language Wikipedias, your LocalStorage quota will quickly be maxed out.

Ideally, the quota should be per host, not per domain name.

See the following thread for another report of the same problem:
https://support.mozilla.org/en-US/questions/963825

Comment 1

[Boris Zbarsky [:bzbarsky]]

This is purposeful, to avoid people "working around" the quota with sockpuppet subdomains...

Comment 2

[kaldari]

Boris: Couldn't you just enact a 'same-origin' policy? In other words, only allow setting LocalStorage for a particular host from that same host?

Comment 3

[Timo Tijhof]

Does "domain" here use the same definition as the cookie system? https://publicsuffix.org/ (e.g. co.uk is not one domain).

Not sure that should be applied to Storage as hosted services such as:

wordpress.com
stackexchange.com
wikia.com

.. then share one quota for all their instances.

There are plenty of site-specific user interfaces that use local storage (e.g. search suggestions). And larger objects saved by an individual site's admin panel (e.g. saving unsaved drafts in case of browser crash or when offline).

With a practically unlimited number of instances on these sites, there is no reasonable way they can use local storage in Firefox due to having a shared quota (visit enough sites and you'll hit it).

Since 5MB is considered reasonable for a single site, a few admin features together may use that up. And if you're an admin on multiple WordPress blogs, too bad. Quota reached...

And what about hosts like GitHub Pages (github.io), 5minfork.com, your typical No-IP subdomain, temporary tunnel service, etc.

I get the suckpuppet issue, but that only raises the bar on investment a little. One can just get more domains and escalate that way (even for free domains with dot.tk).

--

PS:

Continuing under the assumption that the Public Suffix list is indeed used for this...

Looks like github.io, appspot.com and blogspot.com are actually given special treatment by having an entry on that list. Makes sense security-wise, but may not work well with storage. E.g. Wikipedia, Wikia, WordPress, StackExchange etc. want and do use wildcard cookies for Single sign-on.

Plus, it negatively impacts other companies as their clients end up getting crippled access to local storage. I assume you trust larger parties like github.io and appspot.com to prevent abuse by monitoring their services (and handle abuse reports). But that means smaller organisations are at a disadvantage and have to get special permission first.

For cookies the list makes sense (it restricts the domain, not enable it). But for storage it'd be an enabling feature that essentially requires permission from Mozilla (net neutrality?, j/k *bait*).

Comment 4

[Timo Tijhof]

(In reply to kaldari from comment #2)
> (In reply to Boris Zbarsky [:bz] from comment #1)
> > This is purposeful, to avoid people "working around" the quota with
> > sockpuppet subdomains...
> Boris: Couldn't you just enact a 'same-origin' policy? In other words, only
> allow setting LocalStorage for a particular host from that same host?

I think that's in place already and not what Boris meant.

He means this is to avoid abuse where a single domain is able to store infinite amount of space (well, as much as the browser's internal storage is limited to). E.g. the owner of example.org would set up *.example.org to point at 5mb-pump.example.org, then open lots of Math.random() .example.org's and they keep increasing occupied storage space.

Comment 5

[Olli Pettay [:smaug][[e-mail]]]

http://feross.org/fill-disk/ was in the news last year.

Comment 6

[kaldari]

It looks like Chrome rejected adopting Firefox's solution (see comments at https://code.google.com/p/chromium/issues/detail?id=178980) and instead proposed the following strategy:
1) take a (temporary) protective action to avoid too aggressive localStorage usage from a renderer, which could be triggered with a reasonable threshold
2) purge memory for areas that are not used, or do so more aggressively when the usage is getting high, and
3) have proper Quota-like mechanism for managing the total amount of on-disk storage.

Comment 7

[Boris Zbarsky [:bzbarsky]]

> Does "domain" here use the same definition as the cookie system? https://publicsuffix.org/

Pretty sure it does, though I haven't checked the code recently.

> 3) have proper Quota-like mechanism for managing the total amount of on-disk storage.

This may be the desired end state, yes.  Once we get that figured out, we can reduce the existing restriction, presumably.

Comment 8

[Jan Varga [:janv]]

(In reply to Boris Zbarsky [:bz] from comment #7)
> > Does "domain" here use the same definition as the cookie system? https://publicsuffix.org/
> 
> Pretty sure it does, though I haven't checked the code recently.
> 
> > 3) have proper Quota-like mechanism for managing the total amount of on-disk storage.
> 
> This may be the desired end state, yes.  Once we get that figured out, we
> can reduce the existing restriction, presumably.

I think converting LocalStorage to use temporary storage system would solve this.
It's been our long term plan for some time.

See my comment at:
https://openwebapps.uservoice.com/forums/258478-open-web-apps/suggestions/6268041-a-quote-api-for-local-storage-and-indexeddb

Comment 9

[George Mauer]

I can testify that I'm having issues with this.

We have a client who runs different instances of an application - latestcommit/staging/production - so at least three for each of their clients, all of these are on subdomains of their site. Each instance uses localStorage fairly moderately - some caching and the [persistState widget](https://github.com/togakangaroo/persistState) so less than 500KB per instance. Yet with 3 instances per client, some consultants are now hitting up against limits and there's no simple fix!

Even worse, `localStorage.clear()` seems to only work on the subdomain making this a huge "who the hell thought this was a good idea" problem.

Comment 10

[kaldari]

Yeah, the main reason this bug is frustrating is because there's no easy way to work around it. Clearing your LocalStorage won't fix it (since that only clears one subdomain), and if you use dozens of different subdomains under one site it can take forever to figure out which ones you have to manually clear to get another one to work. It's like playing whack-a-mole. If Firefox is going to share LocalStorage over all the subdomains, it should also provide a mechanism for clearing the LocalStorage from all of them at once. The other solution is Bug 461684, but that's been marked as WONTFIX.

Comment 11

[George Mauer]

So this is the workaround that I've been able to invent. It's so convoluted that you might not even consider it a workaround but it will do in a pinch.

1) On each site have a route that returns a blank page that calls localStorage.clear(); This should not be behind a login
2) Have (or gather into a db) a master list of all subdomains a particular user might visit.
3) If the error condition is suspected, for each other subdomain create a 1px iframe with src="subdomain.yoursite.com/clear-localstorage"

Might make you vomit to do it, but it will work.

Comment 12

[kaldari]

FYI, due to this bug (and the WONTFIX status on bug 461684) we (the Wikimedia Foundation) will be degrading Wikipedia performance for Firefox specifically. Unfortunately, we were not able to come up with any other workaround. The code change is https://gerrit.wikimedia.org/r/#/c/263417/1/resources/src/mediawiki/mediawiki.js.

Comment 13

[Jan Varga [:janv]]

We can't increase the 5 MB limit since we want to discourage developers from using localStorage for larger apps. The problem with Wikipedia is that there are multiple subdomains like en.wikipedia.org, sk.wikipedia.org, etc. The current 5 MB limit applies to eTLD+1 (wikipedia.org) to prevent fill disk attacks by creating hundreds of subdomains.

We are in the process of moving localStorage to quota manager and temporary storage. Once that's done, there won't be the 5 MB limit for eTLD+1, it will only apply for each origin. However localStorage will still be limited by temporary storage limits.

So that should fix the problem besides other things.

Comment 14

[Andre Klapper]

(In reply to Jan Varga [:janv] from comment #13)
> We are in the process of moving localStorage to quota manager and temporary
> storage. Once that's done, there won't be the 5 MB limit for eTLD+1, it will
> only apply for each origin.

Jan: Which bug report should I CC on to follow that process?

Comment 15

[Jan Varga [:janv]]

(In reply to Andre Klapper from comment #14)
> (In reply to Jan Varga [:janv] from comment #13)
> > We are in the process of moving localStorage to quota manager and temporary
> > storage. Once that's done, there won't be the 5 MB limit for eTLD+1, it will
> > only apply for each origin.
> 
> Jan: Which bug report should I CC on to follow that process?

bug 742822

Comment 16

[Maciej Jaros]

> The current 5 MB limit applies to eTLD+1 (wikipedia.org) to prevent fill disk attacks by creating hundreds of subdomains.

So currently for URL `https://client.application-group.company.pl/application/` the quota is shared for whole `company.pl` or for `application-group.company.pl`?

Comment 17

[rctgamer3]

This is starting to become an issue for other major sites as well, such as Reddit, which has quite a number of subdomains nowadays and all subdomains (new/old/mod/etc.) share the same LocalStorage quota.

Comment 18

[Jan Varga [:janv]]

We are going to ship a new LocalStorage implementation in FF 68 or 69 which doesn't have this problem.
See bug 1539835.

Comment 19

[rctgamer3]

(In reply to Jan Varga [:janv] from comment #18)

We are going to ship a new LocalStorage implementation in FF 68 or 69 which doesn't have this problem.
See bug 1539835.

I'm running Aurora/DevEdition 68.0b5 (64-bit) with dom.storage.next_gen set to true, though.

Comment 20

[Jan Varga [:janv]]

What does that mean ?
Does it work well for you in 68.0b5 ?
You should have 2GB of quota for entire eTDL+1 domain.

Comment 21

[rctgamer3]

(In reply to Jan Varga [:janv] from comment #20)

What does that mean ?
Does it work well for you in 68.0b5 ?
You should have 2GB of quota for entire eTlD+1 domain.

Then it doesn't seem to be working in 68.0b5. LocalStorage on reddit reports being capable of writing a total of 5243043 bytes (±5MB)

Comment 22

[Jan Varga [:janv]]

There's still the 5MB limit for each origin. However, if there are multiple subdomains they are no longer limited by the 5MB limit as a group.
You can verify that on http://www.filldisk.com/

Comment 23

[creesch.r]

(In reply to Jan Varga [:janv] from comment #22)

There's still the 5MB limit for each origin. However, if there are multiple subdomains they are no longer limited by the 5MB limit as a group.
You can verify that on http://www.filldisk.com/

There seems to be something not entirely right with that though. here is what I saw yesterday running this testscript I made with the same firefox version (68.0b5, though it is possible it has since updated of course) and the same flag enabled.

Comment 24

[Jan Varga [:janv]]

LS now shares quota with IndexedDB and DOM cache, so you may have too much data stored using these two APIs.

You can check "Cookies and Site Data" in preferences to see how much space is used by your origin.

You can also use the estimate() method to see how much space is available.
See https://storage.spec.whatwg.org/

Comment 25

[creesch.r]

(In reply to Jan Varga [:janv] from comment #24)

LS now shares quota with IndexedDB and DOM cache, so you may have too much data stored using these two APIs.

You can check "Cookies and Site Data" in preferences to see how much space is used by your origin.

You can also use the estimate() method to see how much space is available.
See https://storage.spec.whatwg.org/

Executing

navigator.storage.estimate().then(info => { 
   console.log(info) 
});

gives me

quota: 2147483648
​usage: 3073184

checking firefox settings shows me this.

Comment 26

[Jan Varga [:janv]]

Then, it's a bug. Are you able to reproduce it, or it happened only once ?

Comment 27

[creesch.r]

Pretty consistently, in fact rctgamer3 was having the same issue today which is how I arrived here as he was talking with me about it as I helped him troubleshoot it.

Comment 28

[Jan Varga [:janv]]

Do you have time to create a test case for this ?
Or provide steps to reproduce.

Comment 29

[creesch.r]

I'll try to write them up, what I meant is that I have had it happen consistently but the steps itself aren't precise as it basically amounts to browsing reddit.com from a variety of domains for a little while. I'll see if I can make it a bit less abstract and more useful.

Comment 30

[Jan Varga [:janv]]

Btw, does it happen with private browsing only ?

Comment 31

[Jan Varga [:janv]]

Also, does it happen if you set the pref "dom.storage.snapshot_reusing" to false ?

Comment 32

[creesch.r]

I have had it happen one time today, unfortunately making reliable reproduction steps is proofing to be more difficult than I thought. I'll continue to try.

(In reply to Jan Varga [:janv] from comment #30)

Btw, does it happen with private browsing only ?

It happens with regular browsing, no private browsing involved.

(In reply to Jan Varga [:janv] from comment #31)

Also, does it happen if you set the pref "dom.storage.snapshot_reusing" to false ?

I'll have to try that out.

Comment 33

[creesch.r]

(In reply to Jan Varga [:janv] from comment #31)

Also, does it happen if you set the pref "dom.storage.snapshot_reusing" to false ?

It appears that it doesn't. I have been using this flag at home for a week now and did not experience the issue. My firefox installation at work does not have the flag set I did run into the same issue today under otherwise identical circumstances.

Comment 34

[Jan Varga [:janv]]

Ok, that narrows down it a bit. Thank you!

Comment 35

[Timo Tijhof]

What is the status of this? It looks like it was enabled in Firefox Beta briefly, but then reverted. Is that correct?

Comment 36

[Andre Klapper]

IIUC, https://bugzilla.mozilla.org/show_bug.cgi?id=1064466#c18 implies this was fixed, then reverted in https://bugzilla.mozilla.org/show_bug.cgi?id=1592136

Do Mozilla engineers have any plans to approach this task again rather sooner than later?

Comment 37

[Andrew Sutherland [:asuth] (he/him)]

LocalStorage NextGen (aka LSNG, which mitigates the problem) is still present in the code-base and it's a high priority goal to re-enable it and have that ride the release trains to release. The issue is that LocalStorage NextGen depends on the QuotaManager subsystem that also underpins IndexedDB/Cache API/ServiceWorkers and we encountered high rates of breakage in the wild for a variety of reasons. The breakage has been greatly reduced through a series of fixes and enhancements, but we're being conservative about re-enabling until our telemetry indicates that the problem has been dealt with.

Once LSNG has been released for a few cycles we will also undertake the eviction of old LocalStorage data from origins that are no longer used.

And as an aside about the use-case of caching JS in LocalStorage, we'd advise only using the Cache API or just letting the Firefox HTTP cache store JS, since those mechanisms allow the JS engine to cache the parsed bytecode and won't jank cold pageloads for the site, etc. Obviously, for users whose QuotaManager storage is broken, this won't work, but as those issues are corrected, it will start working. (Also, users can "fix" their installation by going to about:support at any time and choosing "Refresh Firefox...")

Comment 38

[[e-mail]]

This is a big problem for our company as every web app we use runs on the same main domain. Even the openshift web "console" fills up the default 5 MB quota, leaving no space for other apps. And we cant say our users to change their FF configuration.

Comment 39

[hossein.mohammadi.1201]

(In reply to [e-mail] from comment #38)

This is a big problem for our company as every web app we use runs on the same main domain. Even the openshift web "console" fills up the default 5 MB quota, leaving no space for other apps. And we cant say our users to change their FF configuration.

W have the same problem. For example in addition to main site and domain, we have three subdomains for email, test and jira. So it's a really big annoying bug that all of them grouped as one origin and can't work properly.

Comment 40

[Timo Tijhof]

This has been resolved as of Firefox 92, https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Releases/92.

(In reply to Jan Varga [:janv] from bug 742822 comment #46)

LSNG has been enabled in bug 1599979, so this can be finally resolved as fixed.

Here are some LSNG features relevant for this bug:

• Quota checks and usage reporting for LS is shared with IDB and Cache API (and will be shared with any future quota clients like OPFS)
• There’s no 5MB for entire eTLD+1 group as in the old implementation, the group limit is now 10GB (given/managed by QM).
• The size of LS data is included in navigator.storage.estimate()
• The logical size of LS data is tracked (this is different from all other quota clients like IDB and Cache API which track the physical size of files on disk)
• LS data is evicted along with IDB and Cache API (the new caching behavior with LRU eviction, no prompts asking for a permission are required)
• Alll data is stored under $PROFILE/storage (under the same directory hierarchy)

Commits:
93a: https://hg.mozilla.org/mozilla-central/rev/e165d93f022ce3ff7a26f700098cfa64430f710e
92.0 beta/stable backport: https://hg.mozilla.org/releases/mozilla-beta/rev/098ba31fd111

Comment 41

[Andre Klapper]

Boldly resolving per Krinkle's previous comment.