1647017 - Thousands of `*.final` files, hundreds MB in storage `/morgue` directory seems to be never removed

Status: RESOLVED WORKSFORME

(Core :: Storage: Quota Manager, defect, P5)

Description

[gwarser]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0

Steps to reproduce:

I visit app.slack.com text chat every other day. Some time ago I noticed my Firefox profile backup size was increased. I noticed that profile/storage/default/https+++app.slack.com^firstPartyDomain=slack.com/cache/morgue/ subdirectories contains thousands of *.final files. First time it was over half gigabyte, I removed https+++app.slack.com^firstPartyDomain=slack.com completely, week later this was over 120 megabytes, ~week later, today this is now around 300 megabytes within over 3000 files.

Actual results:

app.slack.com is hoarding 150MB of files in storage morgue directory per week.

Expected results:

It should not be allowed for web page to keep so many data in my profile.
This files should be regularly removed.

Comment 1

[Olli Pettay [:smaug][[e-mail]]]

(To Storage::foo)

Comment 2

[Tom Tung [:tt, :ttung]]

A few things might be related to this:

• Is the storage for that "https://app.slack.com^firstPartyDomain=slack.com" persisted?

Best-effort storage can be transformed into persisted storage by navigator.storage.persist().

To check if the origin is persisted, you can check the result of navigator.storage.persisted() or click the lock symbol in the URL bar and see if there is "Store Data in Persistent Storage" and "Allowed" under "Permissions" label.

Once the storage is persisted, it is guaranteed not to be removed by the eviction mechanism when your usage is close to full.

• When does the best-effort get removed?

Firefox evicts old storage based on LRU policy and the eviction would only happen when the global usage is close to the global limit.

If you prefer to manage storage, you can manually remove storage for selected origins in "about:perferences".

• How much free space do you have on your computer?

Firefox can use up to 50% of free space (note it's a snapshot value during startup, FF doesn't calculated that from time to time) and the quota for a site (group) can be up to 2G by default.

See more details in "Storage limits" section in
https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Browser_storage_limits_and_eviction_criteria

• How to check the usage / quota for a site?

You can use navigator.storage.estimate() to how much usage/quota for the site.

Comment 3

[gwarser]

I did not granted any permission. Nothing allowed under lock icon. Calling navigator.storage.persist() returns promise and asks for permission (dialog).

Storage usage is correctly calculated and shown in about:preferences#privacy (302MB right now).

navigator.storage.estimate().then(console.log)
Promise { <state>: "pending" }
Object { quota: 2147483648, usage: 317727453 }

mega.nz uses 24MB, Gmail/Whatsapp ~10MB each. I will expect lot more for these domains than for Slack.

Comment 4

[Tom Tung [:tt, :ttung]]

(In reply to gwarser from comment #3)

I did not granted any permission. Nothing allowed under lock icon. Calling navigator.storage.persist() returns promise and asks for permission (dialog).

Storage usage is correctly calculated and shown in about:preferences#privacy (302MB right now).

navigator.storage.estimate().then(console.log)
Promise { <state>: "pending" }
Object { quota: 2147483648, usage: 317727453 }

I see so that means the storage for that site is best-effort which can be up to 2G. And the result of quota: 2147483648 proves that.

mega.nz uses 24MB, Gmail/Whatsapp ~10MB each. I will expect lot more for these domains than for Slack.

I guess that it depends on how do these sites/origins manage their storage.

As mentioned in comment #2, Firefox would only evict old storage when the global usage (the overall usages across sites) is close to the global limit (which can be 50% of the free space in your disk).

Comment 5

[Simon Giesecke [:sg] [he/him]]

As discussed, this seems to be the expected behavior, so I am resolving this bug for now. Maybe report this to Slack? Maybe it's by design, maybe it's a defect on their side. It might still be a Firefox defect though, if Slack were requesting deletion of cache items, but these were not correctly honored by Firefox. But if that's the case, we would need further details on how to reproduce this to investigate.

Comment 6

[TGOS]

The reporter complained about morgue, which is like some kind of trash, it is not where persistent data is stored, so all your replies to the reporter are about a completely different thing. Data which is still in active use is not found under morgue.

See also this comment:

This "morgue" folder seems to be like Trash/Recycle bin - everything deleted is moved there, but there is no way to empty it manually (CAD cannot touch this folder). It's only emptied if you fill your disk with lot of files so there is not much space left, then it will be removed on next Firefox start(shutdown?)(It will probably kept there forever if you don't use Firefox often). And of course you can clean it with other data by cleaning cookies using browser settings, but this is rather overkill.

Source: https://github.com/Cookie-AutoDelete/Cookie-AutoDelete/issues/1257#issuecomment-957869015

The problem is that tools that use the official APIs to delete cache, IndexDB, Localstorage, ServiceWorkers, etc. cannot really delete anything as everything gets moved to morgue instead of really being deleted. What's the purpose of doing so?

Comment 7

[gwarser]

Discussed more here: https://bugzilla.mozilla.org/show_bug.cgi?id=1745178 and something about deleting cache is fixed here: https://bugzilla.mozilla.org/show_bug.cgi?id=1784700