853176 - Deleting simple-storage data freezes ui

Status: RESOLVED INVALID

(Add-on SDK Graveyard :: General, defect, P2)

Description

[Dethe Elza]

Using SDK version 1.13.2 and master, when I receive an OverQuota event from simple-storage, I delete old data (kept in array, removed using shift()) until storage is down to 90%. During this time, Firefox shows the spinning beachball and appears to have crashed. Once the data is cleared, it resumes working normally.

There are several things I can (and will) do to improve this. I can use splice instead of shift(). I may be able to move the operation to a worker thread (no idea if workers can access the same simple-storage, but worth a try).

The main bug is that browser shouldn't become unresponsive from a single add-on, and that if there were tools for managing simple-storage asynchronously I could easily avoid this.

Comment 1

[Erik Vold [:erikvold] (please needinfo? me)]

this sounds important

Comment 2

[Jordan Santell [:jsantell] [@jsantell]]

Dethe, are you just looping through, removing elements from the array until it's under quota? The simple storage saves every 5 minutes, or something else if the user changes their config, so this may just be synchronous code hanging the thread, regardless of SS writing to disk.

Since this data is just discarded from the front of the array, could you start discarding it slower, earlier? Depending on the size of the array, and array methods being synchronous without callbacks, even using setTimeout could cause backup in your thread.

Comment 3

[Dethe Elza]

Sorry, thought I'd answered this awhile back. Maybe I only replied by email.

I was initially looping through the array. I now figure out what 10% of the number of entries are and Array.splice() them out of the array in one go, which is faster and reduces how often the quota is exceeded. There is still a delay, but it is much less of a problem now. If there is a way to move this off the main thread, I'd be happy to test that out.

Comment 4

[Jordan Santell [:jsantell] [@jsantell]]

Dethe, unloading to a web worker would still involve serializing the large storage you have currently stored in memory, do operations there, and have to reserialize it back. I don't think removing large chunks of data synchronously is a good strategy for staying under quota (if it's being removed, is it arbitrary? on every new data entry, maybe remove an oldest entry?) -- also, check out the indexed-db storage solution, it's a bit more robust and can handle data manipulation better:
https://addons.mozilla.org/en-US/developers/docs/sdk/latest/modules/sdk/indexed-db.html

Comment 5

[Jordan Santell [:jsantell] [@jsantell]]

Is this still an issue?

Comment 6

[Dethe Elza]

Not for Collusion, we've implemented a variety of workarounds and have much less dependence on simpleStorage.

Comment 7

[Jordan Santell [:jsantell] [@jsantell]]

Awesome, good to hear. Running that many consecutive sync operations on anything will hold up anything. Let me know if you run to any other issues!