1593365 - QuotaManager storage v4

Status: NEW

(Core :: Storage: Quota Manager, enhancement, P3)

Description

[Jan Varga [:janv]]

After more thinking about storage initialization failures and unknown stuff in /storage, I've come to a conclusion that we should get rid of directory traversals for good. We should keep a list of origins and their mapping to directories on disk in a database. Consistency between the database and stuff on disk would be ensured by marker files. Client directories and stuff in some client directories would be managed the same way.
This would solve many problems (including very long file names) and it would open the door for new functionality and optimizations (like getting rid of .metadata-v2 and origin parser, replace flat repository directory structure with multi level repository directory structure).
We wouldn't have to worry about unknown stuff and performance wouldn't be badly affected by too many unknown files that need to be checked during storage initialization.

Comment 1

[Jan Varga [:janv]]

The new functionality would include for example better support for private browsing and buckets.

Comment 2

[Jan Varga [:janv]]

(In reply to Jan Varga [:janv] from comment #1)

The new functionality would include for example better support for private browsing and buckets.

and new SessionStorage (for Fission) which needs to support multiple origin directories on disk for the same origin

Comment 3

[Jan Varga [:janv]]

I originally thought that everything would follow this schema:
<profile>/storage2/
UUID1/
UUID2/
UUID3/

However, asuth suggested that it might be good to have a special case for private browsing:
<profile>/storage2/
UUID1/
UUID2/
UUID3/

<profile>/storage2/pb/
UUID4/
UUID5/
UUID6/

This way, it would be easy to delete private browsing data without loading mappings from storage.sqlite. This can be especially useful when we implement a separate daemon for deleting private browsing data that would be launched after Firefox shutdown.

Comment 4

[Jan Varga [:janv]]

Actually the final layout would be a bit different, we need multilevel structure like:
<profile>/storage2/00/
UUID1
UUID2
UUID3

<profile>/storage2/01/
UUID4
UUID5
UUID6

<profile>/storage2/02/
UUID7
UUID8
UUID9

Comment 5

[Jan Varga [:janv]]

Bumping this to v4 since we can't use v3 because we already used v3 in the past (and then downgraded to 2.1).

Comment 6

[Jan Varga [:janv]]

Attachment: Bug 1593365 - Define new tables

Comment 7

[Jan Varga [:janv]]

It seems there will only be multiple patches.

Comment 8

[Jan Varga [:janv]]

The dependency was messy, so I cleaned it up a bit. In general, the main meta bug for addressing storage initialization failures is bug 1482662.
This bug is about QuotaManager storage v4 which is a sub project of bug 1482662.

Comment 9

[Jan Varga [:janv]]

We are now focusing on asynchronous storage initialization instead, bug 1671932.
QMv4 might be addressed in H2 of 2021.