1919655 - Support importing rkv databases from different storage directories into Skv

Status: RESOLVED FIXED

(Core :: SQLite and Embedded Database Bindings, enhancement, P3)

Description

[Lina Butler [:lina]]

The Skv importer assumes that it'll be importing rkv databases in the same storage directory—there's currently no way to ask it to import keys and values from databases in different storage directories. (Separating databases into different directories was necessary to work around race conditions in rkv's safe mode).

In bug 1919530, add-ons would like to consolidate their databases, from 4 different storage directories into just one directory. We'll need to extend the importer to support this.

Comment 1

[Lina Butler [:lina]]

The Rust internals already support this, but we'll need to make a few changes to the API, and call into the internals differently.

Here's a sketch of how it could work:

let importer = SQLiteKeyValueService.createImporter(
    SQLiteKeyValueService.Importer.RKV_SAFE_MODE,
    destinationDir,
);

// Copy all keys and values from the `foo` rkv database in `sourceDir`
// into the `foo` Skv database in `destinationDir`, with the default
// conflict and cleanup policies.
importer.from(sourceDir)
    .add("foo");

// Copy all keys and values from the `bar` rkv database in `sourceDir2`
// into the `qux` Skv database in `destinationDir`, with custom
// conflict and cleanup policies.
importer.from(sourceDir2)
    .add("bar")
    .renameTo("qux")
    .setConflictPolicy(KeyValueImporter.ConflictPolicy.IGNORE)
    .setCleanupPolicy(KeyValueImporter.CleanupPolicy.KEEP);

// Copy all keys and values from all rkv databases in `destinationDir` into
// Skv databases of the same name in `destinationDir`, with the
// default conflict and cleanup policies.
importer.from(destinationDir)
    .addAll();

// Do the import.
await importer.import();

This would support all the use cases that I can think of. (Technically, renameTo isn't needed; all the extensions databases that are in separate directories have unique names—but it's easy to add!)

Comment 2

[Lina Butler [:lina]]

Attachment: Bug 1919655 - Encapsulate `kvstore::fs::canonicalize` into `WidePathBuf`. r?nanj!

kvstore uses "wide" ns{A}Strings (potentially ill-formed UTF-16
strings) to represent paths in its XPCOM interface methods.
ns{A}String doesn't implement Hash or the built-in comparison
traits, so a wide path can't currently be used as a map key.

This commit introduces a WidePathBuf wrapper type, with the same
semantics as an ns{A}String path, but usable as a map key.

Comment 3

[Lina Butler [:lina]]

Attachment: Bug 1919655 - Support importing `kvstore` databases from multiple directories. r?nanj!,robwu

This commit extends the Skv importer to allow consolidating multiple
Rkv databases in different storage directories, which add-ons would
like to do when they migrate (bug 1919530).

nsIKeyValueImporter is now a fluent builder interface. Callers can
use importer.add{Database, AllDatabases}() to configure the importer
to import Rkv databases in the same storage directory, and use
importer.withSourcePath().add{Database, AllDatabases()} to configure
importing databases from other directories. The reworked import()
method does the actual import.

Depends on D223228

Comment 4

[Lina Butler [:lina]]

https://treeherder.mozilla.org/jobs?repo=try&revision=9ac1e3dcbec8e04afb538cdef3efc9105b874c7e

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/92cc663ef661
Encapsulate `kvstore::fs::canonicalize` into `WidePathBuf`. r=nanj
https://hg.mozilla.org/integration/autoland/rev/eca8c06d72f1
Support importing `kvstore` databases from multiple directories. r=nanj

Comment 6

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/92cc663ef661
https://hg.mozilla.org/mozilla-central/rev/eca8c06d72f1