Note: There are a few cases of duplicates in user autocompletion which are being worked on.

[kinto] Migrate browser.storage.local to use kinto collections

ASSIGNED
Assigned to

Status

()

Toolkit
WebExtensions: Untriaged
P3
normal
ASSIGNED
a year ago
2 months ago

People

(Reporter: bsilverberg, Assigned: glasserc)

Tracking

(Blocks: 2 bugs)

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [storage]triaged)

(Reporter)

Description

a year ago
This was discussed in bug 1253740. storage.sync is going to use Kinto collections to store data locally, and this could also be used for storage.local. Sharing this implementation will result in simpler code for storage in general.

In addition to changing the implementation of storage.local to use Kinto, this bug also needs to include a facility that will automatically migrate any data currently stored in storage.local's json files to Kinto collections.
(Reporter)

Updated

a year ago
Assignee: nobody → eglassercamp
Status: NEW → ASSIGNED
Priority: -- → P3
Summary: Migrate browser.storage.local to use Kinto collections → [kinto] Migrate browser.storage.local to use kinto collections
Whiteboard: [storage]triaged
(Assignee)

Updated

9 months ago
Blocks: 1311710
No longer blocks: 1253740

Updated

4 months ago
webextensions: --- → ?

Updated

4 months ago
webextensions: ? → ---

Comment 1

4 months ago
Does this mean that storage.local entries will be separate records? I am working on data migration for Adblock Plus and was horrified to discover that all data in storage.local is being saved as a single JSON blob - for my Adblock Plus config that's currently 26 MB, and only part of that data really needs to be loaded every time.
Blocks: 1226547

Comment 2

2 months ago
https://issues.adblockplus.org/ticket/5284

is caused by this too and makes the browser jank/hang even with simple tasks,Latest version causing severe lag on page scrolls etc and constant severe freezes... Issue solved by reverting back to ABP 2.8.2 (or disabling 2.9).
Could you guys please look into this because it looks like processing storage.js isn't  efficient when compared to previous way.
Ghostery is also affected by this.
Flags: needinfo?(kmaglione+bmo)
Flags: needinfo?(ehsan)
Flags: needinfo?(eglassercamp)
Flags: needinfo?(amckay)

Comment 3

2 months ago
Perhaps it would make more sense for that much of data to go into indexeddb. 

According to the Chrome docs https://developer.chrome.com/extensions/storage has a limit of 5,242,880 (~5MB). The Chrome storage docs also point out that storing a large amount of data there isn't a great idea. It wouldn't be able to store the 26MB mentioned in comment 1 without asking for unlimited storage permission (on Chrome).
Flags: needinfo?(amckay)

Updated

2 months ago
See Also: → bug 1320186
shellye, please avoid needinfoing random people on bugs.  I don't think there's anything I can help with here, but it looks like you're in good hands.  :-)
Flags: needinfo?(ehsan)

Comment 5

2 months ago
(In reply to Andy McKay [:andym] from comment #3)
> Perhaps it would make more sense for that much of data to go into indexeddb. 
> 
> According to the Chrome docs https://developer.chrome.com/extensions/storage
> has a limit of 5,242,880 (~5MB). The Chrome storage docs also point out that
> storing a large amount of data there isn't a great idea. It wouldn't be able
> to store the 26MB mentioned in comment 1 without asking for unlimited
> storage permission (on Chrome).

Current implementation in 52/53 is to store in sqlite or pattern file can't that be used moving forward?

It's faster & pref can be saved .js file.
So only the data needed is loaded when needed aka lazyfy.

Are there any plans to sort this in 57?
currently most addons using webext are slowing the browser down :(
Flags: needinfo?(amckay)

Comment 6

2 months ago
You are making an assumption that the goal to moving to kinto.collections has anything to do with performance. That wasn't the case when filing this bug. Will this improve performance? Do you have any information about that, would be very interested in that.
Flags: needinfo?(kmaglione+bmo)
Flags: needinfo?(eglassercamp)
Flags: needinfo?(amckay)

Comment 7

2 months ago
(In reply to Andy McKay [:andym] from comment #6)
> You are making an assumption that the goal to moving to kinto.collections
> has anything to do with performance. That wasn't the case when filing this
> bug. Will this improve performance?

I strongly suspect that it will. We are currently getting numerous reports about a performance regression in Adblock Plus 2.9 (see bug 1369655). It has been identified now that the code saving our data got very inefficient now that browser.storage API is being used. While I am implementing a change to make saving data less frequent (https://issues.adblockplus.org/ticket/5284), the root issue remains. Problem is, Adblock Plus is storing lots of data - for me the storage.js file is currently 63 MB large (unlikely representative for the average user but still). Much of this data are automatic backups, the assumption was always that individual storage entries would be stored separately and so having data in unused entries wouldn't degrade performance elsewhere. Yet, my profiling currently shows multiple seconds being spent serializing JSON data, as well as some considerable time spent sanitizing it (yes, bug 1320186) - all of that on the main thread, meaning that the browser UI is hanging.

As I indicated in comment 1, I assume that with kinto there will no longer be a single JSON blob but rather a proper collection of storage entries that can be read and written independently. That would improve performance quite considerably in our case.
You need to log in before you can comment on or make changes to this bug.