1766045 - Add support for UniFFI JS bindings

Status: RESOLVED FIXED

(Toolkit :: UniFFI Bindings, enhancement, P2)

Description

[Ben Dean-Kawamura [:bdk]]

The UniFFI tool has been developed by several Mozilla teams to provide Swift, Kotlin, and Python bindings for Rust crates. Right now, it's mainly used for the mobile browsers.

It would be good to add support for JS bindings for FF desktop. We could then replace some of our hand-written FFI code including tab syncing and the webext-storage-bridge. It would also open the door to incorporating more Rust components into desktop, for example the application-services storage/sync components could potentially be shared by Desktop, iOS, and Fenix.

Comment 1

[Ben Dean-Kawamura [:bdk]]

Attachment: WIP: Bug 1766045 - Add support for generating JS bindings

Currently supported:

• Toplevel functions
• Records
• Strings
• Primitives
• Sequence, Maps, and Optional
• Enums (both flat and complex)
• Errors (both flat and complex)
• Objects

Still Todo:

• CallbackInterfaces
• Custom and external types
• Datetime and TimeInterval

Comment 2

[Ben Dean-Kawamura [:bdk]]

Attachment: Bug 1766045 - Vendoring in Rust code for uniffi-bindgen-gecko-js

• Added --enable-uniffi-fixtures flag. When set, we will compile in
  the UniFFI test fixtures into our shared Rust crate and eventually
  into libxul.
• Vendoring in the Rust crates needed for uniffi-bindgen-gecko-js
• Updated vendor_rust.py now that there are new duplicate Rust crates.
  Since uniffi-bindgen-gecko-js is only used as a build-time tool, I
  don't think this will result in wasted space in libxul, but it would
  be good to reduce in any case. Here's what I can figure out about the
  deps:
  • paste: Gecko is on an older version (0.1 vs 1.0). The version comes
    from the rkv dependecy.
  • semver: Gecko is on an older version (0.9 vs 0.11). The version
    comes from the packed_simd dependency, which depends on
    rustc_version 0.2.3, which depends on semver 0.9
  • semver-parser: Extra fallout from the semver issue
  • toml: Gecko is on an older version (0.4 vs 0.4). The version comes
    from the neqo-crypto dependency.
  • nom: Gecko is on an older version (5.x vs 7.x). There are several
    dependencies at play here.

Depends on D144466

Comment 3

[Ben Dean-Kawamura [:bdk]]

Attachment: Bug 1766045 - Adding mach command to generate UniFFI files

Added mach uniffi generate which uses uniffi-bindgen-gecko-js to
generate JS bindings. It's unfortunate that we need to check these files
in, but I couldn't figure out a way to auto-generate them as part of the
build process.

Adding #include "nsIContent.h" to dom/base/nsINodeList.h. I think
that should have been present before, but things built okay because of
the way things got combined in the uniffied .cpp files. Adding these new
webIDL files bumped NodeListBinding.cpp to a new uniffied .cpp file
which caused the build to fail.

Depends on D144467

Comment 4

[Ben Dean-Kawamura [:bdk]]

Attachment: Bug 1766045 - Adding xpcshell tests for our UniFFI fixtures

To test the bindings generation, we generate JS bindings for a set of test
Rust crates, then run xpcshell tests against those bindings.

Currently these tests only run when the --enable-uniffi-fixtures
option is set, so they need to be run manually be a dev. I'd love to
create a system where these can optionally run in CI, but I'm not sure
how to do that.

Depends on D144468

Comment 5

[Ben Dean-Kawamura [:bdk]]

Attachment: Bug 1766045 - Document generating JS bindings with UniFFI

Depends on D144469

Comment 6

[Ben Dean-Kawamura [:bdk]]

Attachment: WIP: Bug 1766045 - Get `./mach lint` mostly working

Depends on D144470

Comment 7

[Ben Dean-Kawamura [:bdk]]

Attachment: Bug 1766045 - Initial work for UniFFI JS bindings

At this point, it can generate a WebIDL interface to handle the
scaffolding layer, which is a low-level C-ABI layer. The next step is
to generate a JS wrapper on top of this that presents a nice API for
devs.

There's support for both synchronus and async functions, but we haven't
decided the how we want this to be configured. In practice, almost all
functions will need to be async, so for now we're just forcing all
functions to be.

The uniffi-bindgen-gecko-js crate implements a UniFFI external
bindings generator and will generate the JS bindings. The render
module contains submodules that handling rendering different languages.
The basic system is that each module defines extension traits on the
ComponentInterface types (Type, Argument, Field, Function, etc.). The
traits are then used by the template code to render things specific to
that language, for example how a function should be named in the WebIDL
file. The extend library adds some syntactic sugar to allow this
without too much boilerplate.

The uniffi-js directory which contains shared code for
uniffi-bindgen-gecko-js. Right now this is the OwnedRustBuffer
class, which is responsible for tracking ownership and freeing our
RustBuffer instances if needed.

UniFFIRustCallResult is a simply WebIDL dictionary that stores the
result of a scaffolding function call.

Comment 8

[Ben Dean-Kawamura [:bdk]]

Attachment: WIP: Bug 1766045 - Fix geckodriver warnings after upgrade to clap 3.1

Commmand is the new name for App.

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::JavaScript Engine' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 10

[Ben Dean-Kawamura [:bdk]]

Attachment: Bug 1766045 - Fixes to make the hybrid builds work

This is a repsonse to the build errors I noticed with
build-macosx64-hybrid/plain, build-win64-hybrid/plain, and
build-linux64-hybrid/plain.

I'm not exactly sure why those build had errors but others didn't, but
my guess is that it was a combination of:

• A clang++ producing different warnings/errors.
• The uniffied builds being arranged slightly differently, which
  surfaced errors based on missing include statements.

Depends on D144470

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cc77fd5ec7a8
Initial work for UniFFI JS bindings r=nika
https://hg.mozilla.org/integration/autoland/rev/5708672d0e83
Vendoring in Rust code for uniffi-bindgen-gecko-js r=glandium,supply-chain-reviewers

Comment 12

[Cristina Cozmuta (:CrissCozmuta)]

Backed out for causing Bp-hybrid failures on UniFFIGeneratedScaffolding.cpp
Backout link
Push with failures
Link to failure log
Failure line :
/builds/worker/checkouts/gecko/toolkit/components/uniffi-js/UniFFIGeneratedScaffolding.cpp:44:10: error: 'return' will never be executed [-Werror,-Wunreachable-code-return]

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4d5874e16302
Initial work for UniFFI JS bindings r=nika
https://hg.mozilla.org/integration/autoland/rev/2d33e41ac507
Vendoring in Rust code for uniffi-bindgen-gecko-js r=glandium,supply-chain-reviewers
https://hg.mozilla.org/integration/autoland/rev/e8bc2ec40cff
Adding mach command to generate UniFFI files r=glandium
https://hg.mozilla.org/integration/autoland/rev/d86ad195a9a0
Adding xpcshell tests for our UniFFI fixtures r=teshaq,markh
https://hg.mozilla.org/integration/autoland/rev/f137315bb6af
Document generating JS bindings with UniFFI r=teshaq,markh
https://hg.mozilla.org/integration/autoland/rev/113593df6272
Fixes to make the hybrid builds work r=markh

Comment 14

[Atila Butkovits]

Backed out for causing build bustages

Backout link: https://hg.mozilla.org/integration/autoland/rev/7d17c1f11050f689cf57d90729e91078f3c5b9b5

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=WJwMLVO9Tk2ei8j28UK-Ww.0&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=113593df627273c8e66f58bf735c38ad7e937f38

Failure log: https://treeherder.mozilla.org/logviewer?job_id=386273457&repo=autoland&lineNumber=27434

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cc78aaf0481c
Initial work for UniFFI JS bindings r=nika
https://hg.mozilla.org/integration/autoland/rev/40c7a52560d2
Vendoring in Rust code for uniffi-bindgen-gecko-js r=glandium,supply-chain-reviewers
https://hg.mozilla.org/integration/autoland/rev/9e28580c4505
Adding mach command to generate UniFFI files r=glandium
https://hg.mozilla.org/integration/autoland/rev/6fb10be3b37a
Adding xpcshell tests for our UniFFI fixtures r=teshaq,markh
https://hg.mozilla.org/integration/autoland/rev/7c864c151572
Document generating JS bindings with UniFFI r=teshaq,markh
https://hg.mozilla.org/integration/autoland/rev/a3cb50eab998
Fixes to make the hybrid builds work r=markh,teshaq

Comment 16

[Ben Dean-Kawamura [:bdk]]

The first bustage was caused by me not using Lando right and not including all the commits.

The second one was caused by an compiler error that happened in build-linux64-base-toolchains/debug, but not most other builds.

I'm hoping the third try is the charm.

Comment 17

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/cc78aaf0481c
https://hg.mozilla.org/mozilla-central/rev/40c7a52560d2
https://hg.mozilla.org/mozilla-central/rev/9e28580c4505
https://hg.mozilla.org/mozilla-central/rev/6fb10be3b37a
https://hg.mozilla.org/mozilla-central/rev/7c864c151572
https://hg.mozilla.org/mozilla-central/rev/a3cb50eab998