687566 - Change localStorage/sessionStorage back-end to LevelDB

Status: RESOLVED WONTFIX

(Core :: DOM: Core & HTML, defect)

Description

[Honza Bambas (:mayhemer)]

We suffer from performance issues, complex bugs and overall code complexity in our DOM storage impl.  Sqlite also cause I/O hangs on the main thread we need to avoid.

I'd like to try to move the implementation back end to LevelDB.

Operations we need the back end is capable to handle in a scalable way:
- monitor usage quota for domain and/or subdomains (DOM_QUOTA error)
- evict keys created/modified after some date (history clear)
- keep data just in the memory (sessionStorage and PB mode)
- clone a range from disk and then operate on a mirror in memory (session only cookies mode)
- clear all domain and/or subdomain data


LevelDB can query ranges of keys.  Usage quota can be measured with GetApproximateSizes on a range.  There is a setting to let a LevelDB instance work all in memory.  We can clone just by copying key-by-key (as we do now).


My idea is to store keys approximately like this ("key name" -> "key value"):
"moc.alizzom.www.:http:80#name" -> "value"
"moc.alizzom.www.:http:80@2011-09-19 20:54:30 UTC" -> "name"

This represents localStorage["key"] = "value" invoked on http://www.mozilla.com/ page right now.

This way we may get and delete all keys for "mozilla.com" domain or "http:www.mozilla.com:80" origin, delete all keys that are newer then a particular date, get usage for "mozilla.com" domain and optionally its subdomains.

I'd like to start coding a simple prototype for localStorage first and see the performance benefit, if any.

This is with high probability preconditioned by removing support for globalStorage.

Comment 1

[Honza Bambas (:mayhemer)]

(In reply to Honza Bambas (:mayhemer) from comment #0)
> "moc.alizzom.www.:http:80@2011-09-19 20:54:30 UTC" -> "name"

This should more likely be "2011-09-19 20:54:[e-mail].:http:80" -> "name" to get all keys added after some date.  There is no UI to delete just data stored by a site since some date.

Comment 2

[Matthew McPherrin]

(In reply to Honza Bambas (:mayhemer) from comment #0) 
> LevelDB can query ranges of keys.  Usage quota can be measured with
> GetApproximateSizes on a range.  There is a setting to let a LevelDB
> instance work all in memory.

How accurate do quotas need to be?  GetApproximateSizes measures filesystem usage and may not include recent changes, so it could under or over estimate usage.  Especially with compression.

It would probably be a better idea to have explicit accounting stored in the database.

Comment 3

[Honza Bambas (:mayhemer)]

(In reply to Matthew McPherrin [:mcpherrin] from comment #2)
> How accurate do quotas need to be?  GetApproximateSizes measures filesystem
> usage and may not include recent changes, so it could under or over estimate
> usage.  Especially with compression.
> 
> It would probably be a better idea to have explicit accounting stored in the
> database.

We want to prevent DoS and wast of disk space, nothing more.  This doesn't need to be precise that much.  Now we are byte-precise and have nothing then performance problems thanks that (DoS'ing our self).

It is a big win if compression allows more actual data be stored in less space, don't you think?

Comment 4

[Josh Aas]

We aren't planning to integrate leveldb any more so marking this as WONTFIX. We can re-open if things change.