1156671 - indexedDB limits and eviction bugs on MDN

Status: RESOLVED FIXED

(Developer Documentation Graveyard :: API: IndexedDB, defect)

Description

[Szmozsánszky István [:flaki]]

User Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:40.0) Gecko/20100101 Firefox/40.0
Build ID: 20150418030216

Steps to reproduce:

According to http://www.raymondcamden.com/2015/04/17/indexeddb-and-limits there are several inconsistencies and errors in our indexedDB docs on MDN that would be nice if could be remedied.

Comment 1

[Szmozsánszky István [:flaki]]

According to Teoli, Chris you should be the one in the know about this one. :)

Comment 2

[Chris Mills [:cmills]]

Yes, that's me all right ;-)

I have added a ddn keyword to this so it will be picked up in our triages and I can deal with it in due course.

Comment 3

[Ben Kelly [:bkelly, not reviewing]]

I think bent added the ddn keyword to the bug that caused the change in the code.  Bug 1083927.

Comment 4

[Chris Mills [:cmills]]

I'm working on this bug. Currently I've collected the following information form a few sources, but it needs more detail and checking:

Persistent versus temporary

Persistent storage versus temporary storage ... ?

Persistent - will only be evicted if chosen by user

temporary - will be evicted under a LRU policy - see below.

LRU

When the available disk space for IndexedDB (is this under the same quota as all browser storage?) is filled up, Chrome and Firefox will start clearing out IDB data based on a LRU policy — the least recently used database will be deleted first, then the next one, until the browser is no longer over the limit.

Entire databases are deleted at a time — there is no trimming effect put in place.

Maximum storage

The maximum browser storage space is dynamic — it is based on your hard drive size. The maximum browser storage is 10% of your hdd space, and no one origin can take up more than 20% of this.

So if your hdd is 500GB, then the total storage for a browser is 50GB, and one origin can take up a maximum of 10GB.

If this is exceeded, the broser will throw a QuoteExceededError, and the LRU policy will come into effect on temporary data.

UI and controls

A prompt is shown if you request persistent storage, but not for anything else.

You can control the quota ... HOW? IS THIS POSSIBLE?

Comment 5

[Jan Varga [:janv]]

(In reply to Chris Mills (Mozilla, MDN editor) [:cmills] from comment #4)
> I'm working on this bug. Currently I've collected the following information
> form a few sources, but it needs more detail and checking:
> 
> Persistent versus temporary
> 
> Persistent storage versus temporary storage ... ?

There's now a third type of storage :)
It's called "default" storage, this is what you get if you don't use the special parameter of indexedDB.open(), for example:
var request = indexedDB.open("myDatabase", { version: 1, storage: "persistent|temporary" });

The default storage behaves as persistent storage for installed apps (principals with appStatus==APP_STATUS_NOT_INSTALLED) and as temporary storage for everything else.

Each storage type represents a separate repository, here's actual mapping to directories under user's profile:

<profile>/storage - the main top level directory for storages maintained by quota manager
<profile>/storage/permanent - persistent repository
<profile>/storage/temporary - temporary repository
<profile>/storage/default - default repository

Note that there's the "permanent" vs "persistent", we had to rename it to keep upgrading/migration simple.

There were no changes in behavior of temporary storage, however persistent storage now behaves a bit differently, especially prompts and limits are different.
A bit of history, long time ago, we had the first prompt for persistent storage. So before any data could be stored, the user was prompted to allow storing data for given origin. The permission was then stored and used in subsequent requests. Another prompt popped up when 50 MB of raw data (as on the disk) was reached. The user was asked to allow so called unlimited storage for given origin. After the unlimited storage was permitted we actually stopped tracking quota since we literally allowed unlimited storage for given origin. So all in all two prompts which was quite safe for the user, but a bit annoying. The first prompt was actually identified as a showstopper for unprefixing IndexedDB (indexedDB.open vs mozIndexedDB.open). So we decided to temporarily disable the first prompt. The code and UI for it was still there, it was just disabled.
After the default storage landed along with data migration from <profile>/storage/persistent to <profile>/storage/default, we could introduce "explicit" persistent storage. Someone has to use this syntax to get it:
indexedDB.open("myDatabase", { version: 1, storage: "persistent" }
This will trigger the first prompt which was disabled in the past, but after that there are no prompts.
Quota is not tracked for persistent storage from the beginning.
Explicit persistent storage is not allowed on mobile/Firefox OS for normal content, apps can use it.


> 
> Persistent - will only be evicted if chosen by user

If you mean the "clear storage" button in page info dialog, then yes.
Otherwise there's no automatic eviction for persistent storage.

> 
> temporary - will be evicted under a LRU policy - see below.
> 
> LRU
> 
> When the available disk space for IndexedDB (is this under the same quota as
> all browser storage?) is filled up, Chrome and Firefox will start clearing
> out IDB data based on a LRU policy — the least recently used database will
> be deleted first, then the next one, until the browser is no longer over the
> limit.
> 
> Entire databases are deleted at a time — there is no trimming effect put in
> place.

This is not correct (I know it's not your fault, there's no documentation for it right now).
First thing, we evict/delete entire origins. Deleting one database of an origin could cause
problems with inconsistency. We track last access time for each origin which participates in
temporary storage. Note that in this context, temporary storage means a policy, not the
repository (<profile>/storage/temporary). The default storage also participates in temporary
storage.
So once the global limit for temporary storage is reached (more on the limit later), we try
to find all currently unused origins (there are no tabs/apps that keep open databases).
These are then sorted according to last access time. The least recently used origins are then
deleted until there's enough space to fulfill the request that triggered this origin eviction.

There's also another limit called "group" limit, it's defined as 20% of the global limit.
Each origin is part of a group (group of origins). There's one group for each eTLD+1 domain.
For example:
mozilla.org - group1, origin1
www.mozilla.org - group1, origin2
joe.blogs.mozilla.org - group1, origin3
firefox.com - group2, origin4

So mozilla.org, www.mozilla.org, joe.blogs.mozilla.org can aggregately use 20% of the global limit at most.
The group limit is so called "hard limit", it doesn't trigger origin eviction. The global limit is a "soft limit" since there's a chance that some space will be freed and the operation can continue.

> 
> Maximum storage
> 
> The maximum browser storage space is dynamic — it is based on your hard
> drive size. The maximum browser storage is 10% of your hdd space, and no one
> origin can take up more than 20% of this.

This is not corrent. The global limit is calculated as 50% of free disk space.

> 
> So if your hdd is 500GB, then the total storage for a browser is 50GB, and
> one origin can take up a maximum of 10GB.
> 
> If this is exceeded, the broser will throw a QuoteExceededError, and the LRU
> policy will come into effect on temporary data.

QuoteExceededError will only be thrown if the group limit was reached or if origin eviction couldn't free enough space.

> 
> UI and controls
> 
> A prompt is shown if you request persistent storage, but not for anything
> else.

Right, currently there's just one prompt if you request persistent storage.
Everything else is promptless which was our intention.

> 
> You can control the quota ... HOW? IS THIS POSSIBLE?

Unfortunately, there's not way to control or get info about the quota currently.

One more thing ... :)
Quota manager has currently three clients: IndexedDB, AsmJS caching and Cache API
It means that last access time of origins is updated when any of these are activated/deactivated.
And origin eviction will delete data for all these quota clients.

Feel free to ask questions if there's anything clear and sorry that it took so long.

Comment 6

[Chris Mills [:cmills]]

(In reply to Jan Varga [:janv] from comment #5)

..lots...

Thanks so much for the details Jan! I have arranged your info into a document that I think seems fairly coherent. I think presenting this as a standalone doc is good, as it is relevant to much more than just IDB, say.

https://docs.google.com/document/d/1UneDcGP8pBhKBnNfGS8y1sWXMlQ2O7Tjz6qMnQ8vcsk/edit#

Can you have a read and let me know if this sounds ok? I think this is a good — albeit Firefox-centric — guide to browser data storage.

Once this is published I might reach out to other browser vendors and ask them if they want to add their own browser's details to it.

Comment 7

[Jan Varga [:janv]]

I added some suggestions, will add more.

Comment 8

[Chris Mills [:cmills]]

(In reply to Jan Varga [:janv] from comment #7)
> I added some suggestions, will add more.

Cool, thanks - I've accepted the suggestions and combined them with the main document text.

Also see my question at the end of the document.

Comment 9

[Chris Mills [:cmills]]

Documentation on this published at :

https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Browser_storage_limits_and_eviction_criteria

This could definitely use a tech review.

When we are good to go, I will reach out to other browser vendors and ask them to start adding more info to cover their products too.