1513937 - LSNG: Compress specific values

Status: RESOLVED FIXED

(Core :: Storage: localStorage & sessionStorage, enhancement, P2)

Description

[Jan Varga [:janv]]

IndexedDB compresses every structured clone before storing it in the database. It uses snappy for that which provides good compression ratio and very fast compression/decompression.

We could do something similar for LS too with few modifications:
1. Compress values only if their length exceeds given threshold.

2. If compressed length is close to original length, store it uncompressed.

3. Do compression/decompression in the content process to mitigate the overhead that must be sent across IPC and do less CPU heavy work in the main process.
This can also be used for "storage" event notifications.

Hopefully, compression would help with issues like bug 1475218.

Comment 1

[Andrew Sutherland [:asuth] (he/him)]

(In reply to Jan Varga [:janv] from comment #0)
> 2. If compressed length is close to original length, store it uncompressed.

For this, :bkelly's comment at https://bugzilla.mozilla.org/show_bug.cgi?id=1132041#c5 is probably relevant, wherein he passes on the wisdom that Snappy should not meaningfully bloat the data if it doesn't compress well.  So from a code complexity perspective, it might be simplest to just always use Snappy compression.

Comment 2

[Jan Varga [:janv]]

(In reply to Andrew Sutherland [:asuth] from comment #1)
> (In reply to Jan Varga [:janv] from comment #0)
> > 2. If compressed length is close to original length, store it uncompressed.
> 
> For this, :bkelly's comment at
> https://bugzilla.mozilla.org/show_bug.cgi?id=1132041#c5 is probably
> relevant, wherein he passes on the wisdom that Snappy should not
> meaningfully bloat the data if it doesn't compress well.  So from a code
> complexity perspective, it might be simplest to just always use Snappy
> compression.

Ok, good to know. Thanks for the pointer.

Comment 3

[Jan Varga [:janv]]

I have a patch for this, will send it soon.

Comment 4

[Jan Varga [:janv]]

My data.sqlite for twitter.com shrank from ~900K to ~200K.

Comment 5

[Jan Varga [:janv]]

Attachment: Bug 1513937 - LSNG: Compress specific values; r=asuth

Comment 6

[Jan Varga [:janv]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=e39a44dc2d88ba1b8332c4bff30a60b82179f501

Comment 7

[Jan Varga [:janv]]

Attachment: Bug 1513937 - LSNG: Compress specific values; r=asuth

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/d1bd31177b14
LSNG: Compress specific values; r=asuth

Comment 9

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/d1bd31177b14

Comment 10

[BugBot [:suhaib / :marco/ :calixte]]

Since the status are different for nightly and release, what's the status for beta?
For more information, please visit auto_nag documentation.

Comment 11

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/df3eadfa74a8
Backed out 8 changesets (bug 1513937, bug 1546310, bug 1548788, bug 1547688, bug 1547452, bug 1540777, bug 1542669, bug 1547454) for causing bug 1549362.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/6f45208b6065
LSNG: Compress specific values; r=asuth

Comment 13

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/6f45208b6065

Comment 14

[Jan Varga [:janv]]

(In reply to Jan Varga [:janv] from comment #4)

My data.sqlite for twitter.com shrank from ~900K to ~200K.

www.linkedin.com shrank from 1583104 to 745472 bytes on disk.