Open Bug 1643969 Opened 6 years ago Updated 1 year ago

Dire Local Storage situation

Categories

(Core :: Storage: IndexedDB, task, P3)

Product:
Core
Component:
Storage: IndexedDB
Version:
77 Branch
Type:
task

Tracking

()

Status:
UNCONFIRMED

People

(Reporter: aros, Unassigned)

Details

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0

Steps to reproduce:

I'd like Mozilla to reconsider the decision to lump all local storage together, and I mean addons storage and websites storage stored in the same folder: $PROFILE/storage/default

  1. These are very different storage types with wildly different lifetimes. Websites local storage is mostly temporary while extensions data is pretty much permanent.
  2. I would like to see an option to have websites local storage to be stored only in RAM because
    2.1 Some websites have started misusing it, e.g. I've already found some video hostings similar to NetFlix which are using Local Web Storage for video caching! Recently I've discovered a local storage folder which weighed over 1GB!
    2.2 Some websites create an enormous amount of disk writes, e.g. Discourse based forums where a single page refresh/new URL request creates up to 20MB of disk writes.
    2.3 Most websites still use local storage only as temporary which means it probably should be treated differently than cookies.

What I propose:

  1. Extensions and websites should have their storage folders completely separated - this is a must imho.
  2. An option to specify websites storage folder (some users might want to store this data on a RAM disk) - this is again a must and should be relatively easy to implement.
  3. An option to specify websites storage as temporary and store it only in RAM while the browser is running - this would be extremely nice to have (without necessarily switching into private mode).

Under no circumstances I want to deal with hundreds of folders under $PROFILE/storage/default which contain almost exclusively temporary data.

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Component: Untriaged → Storage
Product: Firefox → WebExtensions
Component: Storage → Storage: IndexedDB
Product: WebExtensions → Core
Severity: -- → S3
Type: defect → task
Priority: -- → P3

(In reply to Artem S. Tashkinov from comment #0)

  1. These are very different storage types with wildly different lifetimes. Websites local storage is mostly temporary while extensions data is pretty much permanent.

Websites and WebExtensions can currently both request persistent storage with both subject to clearing due to storage pressure. For a while we were automatically granting WebExtensions the persistence permission by default, but now it's an explicit permission they need to request.

  1. I would like to see an option to have websites local storage to be stored only in RAM because
    2.1 Some websites have started misusing it, e.g. I've already found some video hostings similar to NetFlix which are using Local Web Storage for video caching! Recently I've discovered a local storage folder which weighed over 1GB!

You can set the dom.quotaManager.temporaryStorage.fixedLimit preference via about:config to the maximum amount of disk space that you'd like to be made available to WebExtensions and Websites without the persistent storage permission. It's a single pool, with each site limited to at most 20% of that pool. See https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Browser_storage_limits_and_eviction_criteria for more info.

2.2 Some websites create an enormous amount of disk writes, e.g. Discourse based forums where a single page refresh/new URL request creates up to 20MB of disk writes.

If you want to block specific sites from using storage, see https://support.mozilla.org/en-US/kb/storage#w_allow-or-block-websites-from-storing-information.

We are considering options to reduce disk churn for LocalStorage keys that have a high rate of churn and currently buffer LocalStorage writes to every 5 seconds. But in general if there are sites that are doing things you don't like, the options tend towards providing the sites feedback about their usage, not using the site, disabling the site's access to storage, and/or applying site-specific mitigations using specific WebExtensions or extensions like Greasemonkey/Tampermonkey that allow site-specific scripts to alter site behavior.

2.3 Most websites still use local storage only as temporary which means it probably should be treated differently than cookies.

It's my understanding that our user research has shown that user mental models lump all site storage together with "cookies", and view clearing their cookies as also wiping site data. There was a point where different clearing options would clear cookies separately from IndexedDB and users found this surprising and not desirable. This is why we now treat "Cookies and Site Data" as the same thing by default.

What I propose:

  1. Extensions and websites should have their storage folders completely separated - this is a must imho.

The current structure names the directories based on the origin including the scheme, so extension directories can be distinguished by possessing the prefix moz-extension+++, whereas http sites will have the prefix http+++ and https sites https+++. This is an outgrowth of the core of the QuotaManager logic treating all origins/principals the same with the differences between schemes/etc. being policy decisions made at higher levels or based on properties of the origins/principals.

Can you elaborate on the motivation behind this request?

Note that in general it's a non-goal to allow manual filesystem management of the profile directory. The supported mechanisms and UI affordances are the Manage Data UI in about:preferences and the WebExtensions browsingData API for more advanced management of this data.

It is an explicit goal, however, to enable users to have control over site storage, which is why there are features to:

  1. An option to specify websites storage folder (some users might want to store this data on a RAM disk) - this is again a must and should be relatively easy to implement.

Being an extremely expert level feature that can be locally implemented by users using symlinks, this doesn't seem like a feature that we could get product management to sign off on. I suspect this is related to your separate directory request above. Is the idea that you could symlink websites to a RAM disk if they had their own directory but leave WebExtensions on persistent storage? Is the primary concern about write cycles on the underlying hardware, or something else? Our hardware survey at https://data.firefox.com/dashboard/hardware suggests that very few users have sufficient RAM to consider a RAM disk without degrading system performance.

Right now it should be possible to mount a ramdisk at the storage directory (either directly or via a symlink) and transfer any durable data across prior to launching Firefox, then moving any data leftover after the "Delete cookies and site data when Firefox is closed" setting has cleared site data back.

  1. An option to specify websites storage as temporary and store it only in RAM while the browser is running - this would be extremely nice to have (without necessarily switching into private mode).

I don't think this is feasible given the system memory available as indicated by https://data.firefox.com/dashboard/hardware and our desire to use the same codebase on mobile (and expressed user concerns about browser memory usage) where devices have less memory available and many more things that want the memory and where it's expected that apps or tabs will be closed on memory pressure and reloaded as needed and using only RAM-backed storage would result in data-loss for things the user is actively doing.

A general goal of the storage APIs is to support creating offline web sites that are able to effectively compete with dedicated apps (on both desktop and mobile) so that users don't need to install site-specific apps that the user has significantly less control over. It sounds like this is at odds with your use-cases and willingness to allow sites persistent storage, but does drive our underlying engineering decisions along with the crash reports from the field that indicate out-of-memory errors are unfortunately still too common.

It's probably also worth noting that we're currently attempting to enable Private Browsing to have access to the same storage abilities as normal web browsing via encrypted storage (where the keys are never written to disk) that's purged when the private-browsing is closed. We've had many requests to make PrivateBrowsing be able to do everything normal browsing is capable of doing, both to enable explicit use-cases like playing web games in PrivateBrowsing, as well as to make it harder for sites to refuse to operate in private browsing mode.

It's all nice and logical except again, some websites totally abuse local storage and generate hundreds of megabytes of worthless writes which specially for SSD disks are not good at all.

Try browsing e.g. https://forums.developer.nvidia.com/c/gpu-unix-graphics/linux/148 and see for yourself. There are hundreds of websites using Discourse and this forum software is just utterly broken in terms of using and abusing local storage. I can probably file a bug report against Discourse and wait a few months for it get resolved but what about thousands of other websites whose programmers are out of reach?

It would be great if Mozilla addressed this issue by implementing at least some of the proposed solutions and one of them is separating the storage for websites and extensions, e.g. having them in different directories. Having them under the same directory has never made any sense anyways.

It's really appalling no one cares as if you guys are so rich you're OK with replacing your SSD disks every six months. And perhaps you're backing up your every action on your PCs which is not what most people routinely do.

You need to log in before you can comment on or make changes to this bug.