Create some optional pref or extension permission to increase storage.sync quota
Categories
(WebExtensions :: Storage, enhancement, P3)
Tracking
(Not tracked)
People
(Reporter: aminomancer, Unassigned)
Details
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:81.0) Gecko/20100101 Firefox/81.0
Steps to reproduce:
A recent update broke a lot of extensions which rely on storing significant amounts of data with storage.sync: https://blog.mozilla.org/addons/2020/07/09/changes-to-storage-sync-in-firefox-79/
Some extensions can get around this by 'cleaning' obsolete keys from the database to get back under quota, etc. But there are whole archetypes of extensions that rely on permanently storing data. The reason I first noticed this problem was because I use a webextension to turn my new tab page into a notes input page. I originally based it on this extension ( https://github.com/wildskyf/tab-notes ) which used storage.sync so that notes would be synced between devices. This was a very useful feature.
Actual results:
But the quota was set SO low that you can't even store more than a couple pages of notes. I don't get this at all. I haven't even been able to find a discussion where someone offers a justification. We're only talking about text, really tiny amounts of data.
So this change broke the core functionality of my NTP extension fork, it wouldn't update the database at all so notes could not be saved. I had to revise the script to migrate from storage.sync to storage.local. The original author of that extension has since done the same thing, by the way. As far as I know, all NTP notes extensions now use local storage, though I guess in theory one could require the user to make an account and host their private notes on a remote server.
Although that restores the basic functionality, it completely eliminates the syncing function. Notes are no longer synced between devices, all because your notes amount to more than 100KB or maybe your entries are larger than 8KB.
Expected results:
I feel like these quotas are comically low. These numbers look like quotas from the late 90s, idk. If someone made an effort to explain the reasoning behind it, then I might be able to stomach it. Is this to reduce firefox sync's bandwidth usage? But even if that's the case, surely there has to be some kind of compromise that will allow these kinds of extensions to still function.
It's not just notes, it could be all sorts of things. Even if the majority of users and extensions don't need more storage, it would be great if there was at least some measure of support for us, like a preference, or some kind of google drive integration so we could simply host our own databases, and make them 10TB for all Mozilla cares. Many other third party apps have a way to back up their configurations/data to google drive and restore from this remote storage.
Obviously that's not Mozilla's responsibility, and webextensions developers could set up their own remote storage rather than Firefox having some convoluted google drive integration. But most users don't examine their extensions' source code. They just have to decide whether to trust an extension. And an extension that says it directly reads and writes to your google drive account doesn't sound secure at all.
From the user's point of view, I imagine it sounds a lot more trustworthy for an extension to simply read/write to its own abstract storage.sync area, while the user only needs to grant permission to Firefox itself for some internal module to read/write to their google drive account. I'm sure there are many ways to implement it such that the extension itself need not be aware of the google drive account.
And of course it makes developing the extensions easier. Personally I don't have the first clue how to make a webextension use a db on the user's google drive account. But I do know how to use the web storage API, and I know how to give users instructions to enable some sort of preference or link their google drive account.
Thanks
|
||
Comment 1•5 years ago
|
||
Hi!
Thanks so much for your contribuition.I will move this over to a component so developers can take a look over this enhancement.
Have a good day.
Best regards, Clara.
|
||
Comment 2•5 years ago
•
|
||
Hi there, thank you for the thorough report! I agree that the low limits are a stumbling block to using this API, and it’s the kind of thing that extensions might not be prepared to handle.
The low quota limits are the same as Chrome’s implementation. I think our worry with having a higher limit was that extensions coded against Firefox wouldn’t work when used in Chrome, but I wonder if there’s a middle ground here. Firefox Sync does have quota limits for other data types (bookmarks, history pages, tabs, saved passwords), too, for the reason you mentioned—reducing bandwidth and storage costs—but they’re higher than the storage.sync limits. On the one hand, requiring extensions to use their own backend when they want to store more than 100k of data (or 8k per value) feels like a big increase in effort and complexity. On the other, I think those extensions already have to do that for Chrome, today.
Before 79, we only had a server-side quota, though it wasn’t enforced by Firefox—so storage.sync.set would appear to succeed, and even save the data locally. In some ways, this was better, because extensions that used storage.sync for local storage (IOW, treating it exactly like storage.local) would continue to work. But it was worse overall, because that data wouldn’t sync, and neither the extension nor the user had any way to know that. Since the old Kinto limits were (I think) per-user, one extension that stored too much data could prevent all of them from syncing. There’s some more details about that in bug 1422673 and this GitHub issue.
So I’m not sure if we can bump the limits, introduce a Firefox-specific “more storage” permission, or if this will end up being a WONTFIX, but thanks for taking the time to file this!
|
Reporter | |
Comment 3•5 years ago
|
||
I see what you mean about chrome. I suppose I will look into the possibility of using google drive API to store users' databases. I definitely don't think mozilla is obligated to host all our data or anything, I just don't actually know how to recreate the same functionality for all users without paying for hosting and so on. At least not yet, but maybe google drive will be able to solve this issue. I have some android apps that let you back up your configuration to google drive from within the app so I assume they have some javascript methods I can use in webextensions. Although I guess that's not a perfect solution either, since I assume one of the primary reasons for many users to choose firefox is because they trust mozilla with their data more than they trust google.
|
|
Reporter | |
Comment 4•5 years ago
|
||
By the way though, I see what you mean that if firefox's storage.sync quotas are higher than chrome's, then some firefox extensions may break in chrome. But it doesn't really make sense to preemptively break all those firefox extensions just to pressure developers into coding their extensions in a way that doesn't break in chrome. Like that means mozilla is intentionally breaking firefox extensions with the intention of keeping extensions from breaking in chrome, a competitor. It just doesn't really make sense, particularly because how are developers supposed to respond to this? How can they write their code in such a way that it won't break if the program relies on storage.sync and the developer can't afford a domain and webserver? Isn't it better for those extensions to at least work in firefox but not in chrome than for mozilla to preemptively decide they won't work in any modern browser?
Also, isn't it better for firefox to distinguish itself from chrome? I think most people would agree that mozilla is not going to beat google on google's terms. People who use firefox are using it because it has always had some relatively obscure capabilities that other browsers did not have. The average user may have no need for these things but firefox has lasted as long as it has because there are enough people who care about these things to create a niche for firefox. I get that firefox has to evolve, and I didn't complain about jetpack or webextensions or e10s even though XPCOM enabled a lot of things I really cared about. I did complain a bit when XBL was removed because it enabled a userchrome.js implementation that I used to access some of those pre-webextensions methods. But then I found an autoconfig implementation so that's all good. I generally like the direction firefox is going but I do feel like there's a pattern emerging of firefox removing a lot of things that previously distinguished it from chrome.
I don't think every firefox extension needs to work identically in chrome. For one, a lot of firefox extension developers do not even bother to add the browser polyfill. But those that do could certainly take a little extra trouble to add some browser checks and disable any functionality that relies on storage of large DBs. Those that are right on the fence could potentially break up the items into chunks to maximize the potential storage, and then just warn the user as they approach the quota. There are ways for developers to make the extension effectively unlimited in firefox, but limited in chrome. And alternatively, developers could easily set it up so that the firefox version uses storage.sync and the chrome version uses storage.local.
Firefox supporting sync for this type of extension would be a clear advantage for firefox, wouldn't it? I don't know all the nuances of mozilla's business model, but I assume that more users would be beneficial to mozilla. Anyone who relies on these extensions would have a greater incentive to use firefox than to use chrome, so maybe mozilla wouldn't need to lay off as many hard-working employees.
And personally I like the idea of a "more storage.sync" permission, just like there's already an unlimited storage permission. Extensions that don't need it can just ignore it, but extensions that do need it would have life breathed into them as they're effectively dead right now.
Anyways thanks so much for the detailed response.
|
||
Comment 5•5 years ago
|
||
We're unlikely to unilaterally extend these limits as Lina mentions it might create compatibility issues.
Description
•