1596322 - Add bindings for implementing Sync engines in Rust

Assignee

Description

•

5 years ago

As part of shipping Remerge (generic sync) on Desktop, we'll need to figure out how to integrate it with the existing Sync machinery. In a-s, this is handled entirely by the sync15 crate, which manages authentication with the token server, setting up the server, fetching and uploading BSOs, and record crypto.

It also provides a Store abstraction, which all our syncable data stores implement, as well as top-level sync functions for kicking the whole process off.

Shipping sync15 on Desktop will be tricky to do in its current state, because it pulls in networking libraries, and also (IIUC) depends on the a-s copy of NSS for crypto. Long-term, we'll definitely consider writing a Viaduct backend for HTTP requests, and linking to Gecko. Additionally, Desktop already has logic for talking to the token server, setting up storage, and downloading and uploading records. Having two state machines run as part of the same sync also duplicates work, and introduces more opportunities for things to go wrong.

In the meantime, we can make incremental progress by using our existing Sync code, and having the Rust code do all the heavy lifting for merging and storage. This would also bring us a step closer to landing Rust components on Desktop, without boiling the ocean.

Let's use this bug to brainstorm and prototype ideas.

Lina Butler [:lina]

Assignee

Comment 1

•

5 years ago

I'll go first, to get things going! 😁

I wonder, can we use Rust XPCOM bindings to make a "native engine", with the guts of the storage and merging logic encapsulated entirely in Rust, but leaning on the Sync JS code for networking and crypto? I'm imagining a minimal interface that looks something like this:

interface mozIBridgedSyncEngineCallback {
    void handleSuccess(in nsIVariant result);
    void handleError(in AString message);
};

interface mozIBridgedSyncEngineSyncCallback : mozIBridgedSyncEngineCallback {
    void queueForUpload(in AString collectionName,
                        in Array<AString> recordsAsJSON);
};

interface mozIBridgedSyncEngine {
    attribute mozISyncedBookmarksMirrorLogger logger;

    void getLastSync(in AString collectionName,
                     in mozIBridgedSyncEngineCallback callback);
    void setLastSync(in AString collectionName,
                     in mozIBridgedSyncEngineCallback callback);

    void withIncomingRecords(in AString collectionName,
                             in Array<AString> recordsAsJSON);

    mozIPlacesPendingOperation sync(in mozIBridgedSyncEngineCallback callback);

    void syncFinished(in AString collectionName,
                      in Array<AString> uploadedIds);
};

Since we already get a JSON string when we decrypt incoming records, we can pass them directly over to Rust, instead of parsing them and inflating them into records on the JS side. The JS side would also take care of fetching from the data and metadata collections (if we keep using multiple collections), and the Rust code would stitch it together.

A Rust implementation of mozIBridgedSyncEngine would also manage a background thread, which we can use for merging and running synchronous storage operations. Gecko has a thread pool designed for this kind of thing, which we can lean on. We'll also need to figure out initializing the database connection, how we run storage operations, whether we use Rusqlite or mozStorage—but one step at a time. sync returns an interruptible operation, and there's a logger that we can use to integrate with Sync logging, like we do for bookmarks now...but, other than that, merging is completely opaque to the Sync JS code.

On success, Remerge can hand us records to upload, which we'd queue on the JS side until handleSuccess is called. We'd encrypt these directly, without parsing them first, and upload them to the data and meta collections.

There's also some boilerplate-y methods for managing the last sync time, which the JS side needs to know to fetch and upload records, and syncFinished, to tell Remerge to write everything that just got uploaded back to its mirror.

There isn't nearly enough detail in this—and, when it comes to Sync, the 😈 is in the details—but something to get us started.

Lina Butler [:lina]

Assignee

Bug 1596322 - Add XPCOM bindings for Rust Sync engines. r?markh!,tcsc!,LougeniaBailey! 5 years ago Lina Butler [:lina] 47 bytes, text/x-phabricator-request		Details \| Review
Bug 1596322 - [DO NOT LAND] Add an example form autofill storage implementation. 5 years ago Lina Butler [:lina] 47 bytes, text/x-phabricator-request		Details \| Review
Bug 1596322 - [DO NOT LAND] Add an example extension sync engine implementation. 5 years ago Lina Butler [:lina] 47 bytes, text/x-phabricator-request		Details \| Review