1328670 - Expose SQLite to Rust code in tree

Status: NEW

(Firefox Build System :: General, defect)

Description

[Nick Alexander :nalexander [he/him]]

This ticket tracks connecting Rust code in the tree to our existing custom compiled SQLite library.

It would be really nice to use *some* existing Rust SQLite crate, although I'm not sure there's a consensus "best of breed" yet.  I'm not sure how tricky it will be to use our custom compiled SQLite library with such a crate, but we should find out now and get this working for future consumers.

My potential first consumer: the Rust replacement for the Datomish data storage layer that joewalker, rnewman, me, and others are working on.

Comment 1

[Ralph Giles (:rillian)]

Generally the way this works is that there's a foo-sys crate which wraps the "system" shared library, and then a more rustful api wrapping that. In this case the sqlite3-sys crate seems to include its own build from source. We'd need to tweak our vendored copy to link our build instead. https://github.com/stainless-steel/sqlite3-sys

Comment 2

[(not currently active) Ted Mielczarek]

Seems like a fairly inoffensive build option we could upstream, no?

Comment 3

[John Gallagher]

There are at least two "sqlite-sys" crates that were each made to support independent Rust SQLite wrappers; sqlite3-sys linked above (the low level wrapper for https://crates.io/crates/sqlite) and https://crates.io/crates/libsqlite3-sys (the low level wrapper for https://crates.io/crates/rusqlite). I can't speak to the former but I maintain the latter.

As of pretty recently it supports building SQLite from source and linking against that (by default it tries to find an already-installed SQLite lib). A snapshot of the SQLite source is included, but I suspect it would not be a lot of work to either configure that to match how you build SQLite or provide a way for users to supply their own amalgamated source file.

Comment 4

[Ralph Giles (:rillian)]

(In reply to John Gallagher from comment #3)

> A snapshot of the SQLite source is included, but I suspect it would not be a
> lot of work to either configure that to match how you build SQLite or
> provide a way for users to supply their own amalgamated source file.

Do you do binding generation? We already have a copy of the sqlite source in-tree (two copies infact...) but we're already building it are part of the C++ side. What we'd like to do is to leave resolution of the sqlite C implementation symbols unresolved when the rlib is built, so when it's finally linked (together with the the sqlite build) into the final executable (actually a shared library) the rust crate's references get resolved to the same implementation as everything else.

Comment 5

[John Gallagher]

(In reply to Ralph Giles (:rillian) | needinfo me from comment #4)

> > A snapshot of the SQLite source is included, but I suspect it would not be a
> > lot of work to either configure that to match how you build SQLite or
> > provide a way for users to supply their own amalgamated source file.
> 
> Do you do binding generation?

If by "binding" here you mean the conversion of the SQLite header(s) into unsafe Rust functions (https://github.com/jgallagher/rusqlite/blob/master/libsqlite3-sys/src/bindgen.rs) - that isn't currently done at build time, no; we manually ran rust-bindgen on the version of SQLite that we ship with the library and commit that file. Early in the life of both rusqlite and rust-bindgen some manual massaging was necessary; I'm not sure if it still is. If it's not (or if the massaging is relatively simple) it would probably be straightforward to run rust-bindgen at build time instead.

> We already have a copy of the sqlite source
> in-tree (two copies infact...) but we're already building it are part of the
> C++ side. What we'd like to do is to leave resolution of the sqlite C
> implementation symbols unresolved when the rlib is built, so when it's
> finally linked (together with the the sqlite build) into the final
> executable (actually a shared library) the rust crate's references get
> resolved to the same implementation as everything else.

I'm _far_ from an expert on rlibs, but I think this is how rlibs already work. I just tried building a test binary both against a shared sqlite lib and a static one, and in both cases `nm` on librusqlite.rlib lists all the sqlite3 symbols as undefined. (The test binary itself listed them as undefined in the shared case and as text in the static case, as expected.)