1913238 - [meta] Recover corrupt Skv databases

Status: NEW

(Core :: SQLite and Embedded Database Bindings, task, P3)

Description

[Lina Butler [:lina]]

Detecting and recovering from database file corruption is a challenge for every storage system. Let's see if we can make things a bit easier for Skv consumers!

Broadly, there are three kinds of corruption that we're worried about:

1. Failing to open the database file.
2. Successfully opening a database file, but encountering a bad page during a read or write.
3. Incoherent or inconsistent data—a malformed value, a foreign key constraint that isn't upheld—in an otherwise accessible database file.

(1) is the easiest to recover from: we can move the old database file aside, then open a fresh one, the first time it's accessed. Application Services and Places do this today.

(3) needs an application-level maintenance task—like PlacesDBUtils.sys.mjs on Desktop, or StorageMaintenanceWorker on Android—to check for and fix up coherence issues. This is where we'd run PRAGMA integrity_check, PRAGMA foreign_key_check, REINDEX, etc.

(2) feels like it's the trickiest, but still doable. We can have Skv trigger maintenance as soon as it witnesses any operation failing with "database disk image malformed". If maintenance succeeds, then maybe we witnessed a temporary I/O error, and we can let the consumer reopen the database file? If it fails, then we treat it the same as case (1).

Skv's coordinator makes it possible to close database connections, even if the consumer still has a reference to an open nsIKeyValueDatabase. This would let consumers who want to try and recover from a corrupt database without restarting Firefox do so, with a bit of extra work. Consumers that are OK with waiting until the next restart to recover would also work just fine in this world.

The other big question I have is how to recover from corruption. SQLite provides a "Recovery API" extension, but we don't currently ship that. The built-in "Online Backup API" and VACUUM INTO can produce a duplicate database file, which we could do periodically—but I'd feel more comfortable about storing backups in an alternative, and ideally textual, format. Marco also pointed me at the "fileio extension", which we could use to export and import the contents of the database as JSON.

This ticket might turn into its own metabug, with blockers for handling each of these cases, but I wanted to jot down some rough ideas first.