1277612 - [kinto] Migrate browser.storage.local to use kinto collections

Status: RESOLVED WONTFIX

(WebExtensions :: Storage, defect, P3)

Description

[Bob Silverberg [:bsilverberg]]

This was discussed in bug 1253740. storage.sync is going to use Kinto collections to store data locally, and this could also be used for storage.local. Sharing this implementation will result in simpler code for storage in general.

In addition to changing the implementation of storage.local to use Kinto, this bug also needs to include a facility that will automatically migrate any data currently stored in storage.local's json files to Kinto collections.

Comment 1

[Wladimir Palant]

Does this mean that storage.local entries will be separate records? I am working on data migration for Adblock Plus and was horrified to discover that all data in storage.local is being saved as a single JSON blob - for my Adblock Plus config that's currently 26 MB, and only part of that data really needs to be loaded every time.

Comment 2

[u595893]

https://issues.adblockplus.org/ticket/5284

is caused by this too and makes the browser jank/hang even with simple tasks,Latest version causing severe lag on page scrolls etc and constant severe freezes... Issue solved by reverting back to ABP 2.8.2 (or disabling 2.9).
Could you guys please look into this because it looks like processing storage.js isn't  efficient when compared to previous way.
Ghostery is also affected by this.

Comment 3

[Andy McKay]

Perhaps it would make more sense for that much of data to go into indexeddb. 

According to the Chrome docs https://developer.chrome.com/extensions/storage has a limit of 5,242,880 (~5MB). The Chrome storage docs also point out that storing a large amount of data there isn't a great idea. It wouldn't be able to store the 26MB mentioned in comment 1 without asking for unlimited storage permission (on Chrome).

Comment 4

[(no longer active)]

shellye, please avoid needinfoing random people on bugs.  I don't think there's anything I can help with here, but it looks like you're in good hands.  :-)

Comment 5

[u595893]

(In reply to Andy McKay [:andym] from comment #3)
> Perhaps it would make more sense for that much of data to go into indexeddb. 
> 
> According to the Chrome docs https://developer.chrome.com/extensions/storage
> has a limit of 5,242,880 (~5MB). The Chrome storage docs also point out that
> storing a large amount of data there isn't a great idea. It wouldn't be able
> to store the 26MB mentioned in comment 1 without asking for unlimited
> storage permission (on Chrome).

Current implementation in 52/53 is to store in sqlite or pattern file can't that be used moving forward?

It's faster & pref can be saved .js file.
So only the data needed is loaded when needed aka lazyfy.

Are there any plans to sort this in 57?
currently most addons using webext are slowing the browser down :(

Comment 6

[Andy McKay]

You are making an assumption that the goal to moving to kinto.collections has anything to do with performance. That wasn't the case when filing this bug. Will this improve performance? Do you have any information about that, would be very interested in that.

Comment 7

[Wladimir Palant]

(In reply to Andy McKay [:andym] from comment #6)
> You are making an assumption that the goal to moving to kinto.collections
> has anything to do with performance. That wasn't the case when filing this
> bug. Will this improve performance?

I strongly suspect that it will. We are currently getting numerous reports about a performance regression in Adblock Plus 2.9 (see bug 1369655). It has been identified now that the code saving our data got very inefficient now that browser.storage API is being used. While I am implementing a change to make saving data less frequent (https://issues.adblockplus.org/ticket/5284), the root issue remains. Problem is, Adblock Plus is storing lots of data - for me the storage.js file is currently 63 MB large (unlikely representative for the average user but still). Much of this data are automatic backups, the assumption was always that individual storage entries would be stored separately and so having data in unused entries wouldn't degrade performance elsewhere. Yet, my profiling currently shows multiple seconds being spent serializing JSON data, as well as some considerable time spent sanitizing it (yes, bug 1320186) - all of that on the main thread, meaning that the browser UI is hanging.

As I indicated in comment 1, I assume that with kinto there will no longer be a single JSON blob but rather a proper collection of storage entries that can be read and written independently. That would improve performance quite considerably in our case.

Comment 8

[u601173]

see also Bug 1371255

Comment 9

[David Durst [:ddurst]]

WONTFIX in favor of 1371255 (and 1406181, which is underway).