1801495 - Immutable noLock VFS

Status: RESOLVED FIXED

(Core :: SQLite and Embedded Database Bindings, defect)

Description

[Marco Bonardo [:mak]]

VFS on top of the -none ones to implement immutable behavior.
More details to follow.

Comment 1

[Marco Bonardo [:mak]]

Attachment: Bug 1801495 - Add a shim no-lock VFS that implements immutable behavior. r=asuth

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

I presume this is low enough level in whatever use case it's for that mozStorage won't be the right place for canonical docs, but it would be great to make sure any docs here can chain upwards to wherever describes:

• The mechanism to make sure successor versions[1] of the database file can be cleanly moved into place. (Like does our updater use A/B locations?)
• The mechanism to make sure obsolete versions of the database files are removed and don't clutter the user's disk. (Like if the mechanism to avoid contention is that every new revision of the database has a new filename, how do the old files get cleaned up?)

1: I'm assuming that the high level intent here is to use SQLite databases like pre-computed look-up tables which themselves are not mutated by Firefox at runtime, so they are individually immutable, but that there is a pipeline that produces new immutable successor databases.

Comment 3

[Marco Bonardo [:mak]]

Hi, sorry I didn't find more time to explain this yesterday. The use-case is opening third party (other browsers) files to import data, we already introduced the noLock VFS for that (unix-none and win32-none) some time ago to be able to open a file when it's in use, we made those only supported in read-only mode, because if you use a noLock VFS to write, you are 99% of the times corrupting your data.

The remaining problem, that we just discovered, is this: opening a WAL database in readonly mode doesn't work, unless the -wal and -shm exist. This is actually documented at https://sqlite.org/wal.html:

It is not possible to open read-only WAL databases. The opening process must have write privileges for "-shm" wal-index shared memory file associated with the database, if that file exists, or else write access on the directory containing the database file if the "-shm" file does not exist. Beginning with version 3.22.0 (2018-01-22), a read-only WAL-mode database file can be opened if the -shm and -wal files already exists or those files can be created or the database is immutable.

These files are sometimes in places where we can't write to, other times we must copy the db to a different position, and it's much easier to just copy the main db. Ideally if it's in a position where we can create files, it should automatically create them and allow to open (the documentation says "or those files can be created") though it should really not be necessary to create a journal for a read-only access. Plus we really don't want to create files in other apps folders.

So the ideal solution is immutable, but it can only be passed with fileuris (as immutable=1 option), there's no corresponding SQLITE_OPEN_* flag. We may ask for it, but as you know we're a bit blocked on Sqlite upgrades, and it's not a given.
Since we basically only need this in the noLock case, I thought I could just write a shim VFS on top of the -none ones that simulates a read-only device. This solves the problem indeed, as expected.
This is only supposed to be used in read only mode, indeed there's an assert in place for that, and the only way to opt-in to this is to use the ignoreLockingMode option of the connection (that enforces read-only mode).

Comment 4

[Andrew Sutherland [:asuth] (he/him)]

Aha! Interesting! Thanks for the context!

Comment 5

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/a69845ef7e6e