1281416 - Give all sites a large storage quota

Status: RESOLVED INVALID

(Core :: DOM: Core & HTML, defect)

Description

[Anne (:annevk)]

Instead of giving all sites a small piece of the pie, we should give everyone the opportunity to take a big piece of the pie.

In the end, most sites will probably need a little bit of storage, but the current strategy prevents those needing a lot of storage from getting it. (That is, I strongly suspect storage distribution to be a power law of sorts.)

If lots of sites end up using a lot of storage (and we should probably measure all the things here to the extent we can) that basically means the LRU algorithm will be often hit to clear things.

I think we should also look into the LRU algorithm and see whether something like "frecency" makes more sense. That should probably be a separate bug.

The way I was thinking about this, if the user has 100% storage, and we reserve 50% of that, we should give sites at least 2% and maybe even a tenth (5%).

This would improve the storage situation vastly without us having to do UX work or implement new APIs.

Comment 1

[Jan Varga [:janv]]

(In reply to Anne (:annevk) from comment #0)
> The way I was thinking about this, if the user has 100% storage, and we
> reserve 50% of that, we should give sites at least 2% and maybe even a tenth
> (5%).

The current implementation reserves 50% and we give sites (eTLD+1) 20% of that / a fifth (and we cap it to 2 GB)

Here's code for that:
https://dxr.mozilla.org/mozilla-central/source/dom/quota/ActorsParent.cpp#4564

Comment 2

[Anne (:annevk)]

Okay, so I was under the mistaken impression we had been super conservative so far. Luke, Dale, is the 2 GB cap the problem for games?

The other thing we talked about in London was going from 50% to 80-90% and also enforcing that upper limit for persistent storage.

Maybe we can use this bug for that discussion (if there is one to be had) for now and then file new bugs once we have something more concrete.

Comment 3

[Jan Varga [:janv]]

(In reply to Anne (:annevk) from comment #2)
> The other thing we talked about in London was going from 50% to 80-90% and
> also enforcing that upper limit for persistent storage.

Yeah, going from 50% to 80-90% should be trivial to do.
The upper limit for persistent storage is more complex, both should be separate bugs.

> 
> Maybe we can use this bug for that discussion (if there is one to be had)
> for now and then file new bugs once we have something more concrete.

Agreed.

Comment 4

[Anne (:annevk)]

Luke, Dale, could you please reply to comment 2? Thanks!

Comment 5

[Luke Wagner [:luke]]

(Sorry, back from PTO)  Yes, this more-recent relaxation helped a lot, as I recall, and addressed the problems we were hearing at the time.  2GiB as an upper bound when the user has plenty of free storage seems reasonable.

It's also important to consider the low-free-space cases, which we hear from flash gamedevs is common.  If a user has, say, 1gb free, that leaves a given site with 100mb which feels like kindof a lot.  It'd be nice to be more aggressive (given that the space is free), but that risks frustrating the user when non-FF applications start failing unexpectedly (w/ no obvious cause).  If we (1) actively monitored free space while FF was open and (2) had a way to be notified by the OS when disk space was running low (when FF was closed), that could perhaps allow us to be more aggressive.  That's a lot more work though.

It'd be useful to know what Flash does here; does anyone know?

Comment 6

[Dale Harvey (:daleharvey)]

> Okay, so I was under the mistaken impression we had been super conservative 
> so far. Luke, Dale, is the 2 GB cap the problem for games?

I havent seen any PouchDB users complain about the storage limits in Firefox (2GB), they are pretty much all hitting the far stricter limits in iOS, IE and to a lesser extend Android. In terms of the wider ecosystem I think we do a reasonably good job here although I wouldnt be against expanding it. 

I think frecency would be a nice but relatively minor improvement to the LRU eviction algorithm however I think the important thing is to give users a way to indicate that storage is not to be deleted automatically (ie installed web apps are auto granted permanent). I have seen a large number of complaints about evicted storage and developers look to be 100% against triggering any UI that the user needs to deal with before using their application (ie requesting permanent storage)

Comment 7

[Anne (:annevk)]

Reading https://www.macromedia.com/support/documentation/en/flashplayer/help/help02.html I suspect Flash doesn't do anything sophisticated.

So suggestions so far:

* Integrate better with the OS to manage "out of storage" scenarios.
* Increase the total quota to 80-90%.
* Make sure persistent storage is bound by the total quota.
* Use frecency rather than LRU
* Once we have "add to New Tab" UI or something similar, we should look into combining that with persistent storage.

Comment 8

[Anne (:annevk)]

I filed bug 1296591 (suggestion 3) and bug 1296592 (suggestion 1+2). That's probably enough for the foreseeable future. Resolving this as INVALID per comment 1 and making it no longer block the meta bug.