847913 - eTLD+1 grouping for permanent storage

Status: RESOLVED WONTFIX

(Core :: Storage: IndexedDB, defect, P5)

Comment 1

[Jan Varga [:janv]]

Attachment: backup

Comment 2

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

I haven't looked at what policy the current patch implements, but figured I should jolt down some thoughts.

I don't think we should use the same eTLD+1 policy for permanent storage as for temporary storage.

The goal of the eTLD+1 policy for temporary storage is to ensure that a single website doesn't take over your whole temporary storage quota.

However there is no such concern for permanent storage since we put up a prompt. I.e. the user knows that he/she authorized a website to use X MB or storage and it doesn't really how the website uses that. It won't affect any other websites and it won't affect the user more than the user authorized.

Instead I think the more important concern for permanent storage is if something like https://myblog.tumblr.com is authorized 50MB of permanent storage from the user, and then website https://funnycatpictures.tumblr.com uses all of that up.

I think an awesome solution would be if https://myblog.tumblr.com could request 50MB of storage, and when it does so, indicate if that storage should be granted to https://myblog.tumblr.com, or *.tumblr.com or *.myblog.tumblr.com etc. This would be really cool if it was part of the quota API.

In the meantime, I think we should simply grant permanent storage on a per-origin basis. I.e. when https://myblog.tumblr.com starts using permanent storage through indexedDB we'll put up a prompt, and if the user accepts that prompt we'll grant storage to only the https://myblog.tumblr.com origin.

Let me know what you think.

Comment 3

[Jan Varga [:janv]]

Yeah, that completely makes sense.

Comment 4

[Florian Bender]

I sent this mail to Jan Varga in March regarding this topic, I think it fits here now: 


#### snip ####

Hi Jan, 

I was wondering about the Quota limit for (Sub)Domains (in the following, WebApp means any website requiring local storage, app cache, iDB etc.): 

The current implementation (with scope eTLD+1, whatever that means*) allows for (e. g.) 5MB for google.com including docs.google.com and mail.google.com, iirc. If google.com needs 5MB, there's no storage left for *.google.com. That's both good (see recent storage-fill-up attack) and bad (WebApps cannot share one domain). 

* I guess an eTLD+1 is domain.com and domain.co.uk but not s.domain.com? If so, when I say "domain" I mean eTLD+1 and when I say "subdomain" I mean eTLD+2 (or even higher). 

What I'd like to see is: 
1. Each individual domain (google.com) and subdomain (docs.google.com, mail.google.com) has a quota of 5MB.
2. But each domain (group) has a total limit of e. g. 30MB for both the domain (google.com) and subdomains (docs|mail.google.com). This effectively allows for 5 WebApps on different subdomains as well as one on the (root) domain or 6 WebApps on different subdomains.
3. If a WebApp gets higher storage privileges (however this is achieved – e. g. a prompt if it reaches its quota, or installing a WebApp from the Marketplace), its quota is excluded from the group quota and gets its own 30MB (50? 100? In several steps each with a prompt?) quota. Thus docs.google.com can store e. g. 30MB, while mail.google.com and google.com each store 5MB, leaving 20MB for other subdomains. 
4. There should be a watchdog monitoring if the quota is reached on several subdomains (of the same domain) and possibly even across domains in a short timeframe (within tens of seconds, escalating to minutes with an exponential algorithm) to prevent the storage-fill-up attack and any non-optimal behavior of apps spanning across multiple subdomains. 
5. All those mechanisms need to be documented clearly with a high-level overview and maybe more in-depth information, possibly involving PR explaining the "clever" combination of mechanisms when (if) it's released. 


Is any of this implemented, or can be implemented by an add-on (however I'd like to see this as a standard functionality)? Either way, what do you think about my ideas?

Comment 5

[Jan Varga [:janv]]

(In reply to Jonas Sicking (:sicking) from comment #2)
> Instead I think the more important concern for permanent storage is if
> something like https://myblog.tumblr.com is authorized 50MB of permanent
> storage from the user, and then website https://funnycatpictures.tumblr.com
> uses all of that up.

hm, https://myblog.tumblr.com and https://funnycatpictures.tumblr.com are separate origins, no ?
so they have separate 50 MB limit too

Comment 6

[Jan Varga [:janv]]

Florian, we plan to implement a quota API, but it is a lower priority at the moment.
See https://wiki.mozilla.org/WebAPI/Storage2013

Comment 7

[Jan Varga [:janv]]

(In reply to Jan Varga [:janv] from comment #5)
> (In reply to Jonas Sicking (:sicking) from comment #2)
> > Instead I think the more important concern for permanent storage is if
> > something like https://myblog.tumblr.com is authorized 50MB of permanent
> > storage from the user, and then website https://funnycatpictures.tumblr.com
> > uses all of that up.
> 
> hm, https://myblog.tumblr.com and https://funnycatpictures.tumblr.com are
> separate origins, no ?
> so they have separate 50 MB limit too

or you mean that we will change our implementation for permanent storage and give 50 MB for entire eTLD+1 domain, and then we would have the problem you described ?

Comment 8

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

(In reply to Jan Varga [:janv] from comment #5)
> (In reply to Jonas Sicking (:sicking) from comment #2)
> > Instead I think the more important concern for permanent storage is if
> > something like https://myblog.tumblr.com is authorized 50MB of permanent
> > storage from the user, and then website https://funnycatpictures.tumblr.com
> > uses all of that up.
> 
> hm, https://myblog.tumblr.com and https://funnycatpictures.tumblr.com are
> separate origins, no ?
> so they have separate 50 MB limit too

Yes, they are separate origins, but they are not separate eTLD+1s.

What I'm saying is that I think that when a page on https://myblog.tumblr.com uses permanent storage, we should only grant the origin "https://myblog.tumblr.com" 50MB of storage.

If later, a page from https://funnycatpictures.tumblr.com tries to use permanent storage, we should not let it use space from the 50MB that the first page was granted. Instead we should put a prompt to the user. If the user says "yes", then we should grant the origin "https://funnycatpictures.tumblr.com" 50MB of storage.

Once we have a quota API then it might make sense to enable the page that is asking for permanent storage to define which origins or some such that it wants to share quota with, but that's something we can look at when we do the quota API.

Makes sense?

Comment 9

[Jan Varga [:janv]]

Yeah, makes sense, thanks

Comment 10

[Florian Bender]

Jan, what's the current status and plan here, now?

Comment 11

[Jan Varga [:janv]]

Well, we haven't started working on public quota API yet. Quota manager is currently being refactored to work on PBackground and to provide simpler internal API for clients (IDB, asm.js caching, LocalStorage) plugged into it.

Comment 12

[Florian Bender]

How does this bug depend on a *public* quota API? IMHO, this should be transparent and just work automatically. Waiting for refactored internal APIs is good, though.

Comment 13

[Jan Varga [:janv]]

It seems that we will need to implement [1] for this. See requestPersistentQuota() method.

[1] http://www.w3.org/TR/quota-api/

Comment 14

[Florian Bender]

(In reply to Jan Varga [:janv] from comment #13)
> It seems that we will need to implement [1] for this. […]
> 
> [1] http://www.w3.org/TR/quota-api/

Is there a bug on file for this? Haven't found one.

Comment 15

[Jan Varga [:janv]]

Quota API has been deprecated, we implemented the Storage standard instead:
https://storage.spec.whatwg.org/

The spec mentions "schemeless origin group" which is something like eTLD+1, but the method for persisting data always work with given origin.

For now, there are no plans to have persist permissions for entire groups like it was suggested in comment 2.