1654840 - Investigate rewriting glean_parser in Rust

Status: RESOLVED WONTFIX

(Data Platform and Tools :: Glean: SDK, defect, P4)

Description

[Michael Droettboom [:mdroettboom]]

Having glean_parser written in Python has some downsides:

• Ensuring there is a useful Python environment around is hard

  • For Android we install a local miniconda (some 300MB, hard to manage), or use a provided Python for offline builds
  • For iOS, we use a system-wide install of Python, but the one that ships with macOS by default doesn't support https
  • For FOG, we have to vendor everything in the mach Python environment, which is full of its own quirks

• Integrating into the build

  • For Android, the Glean Gradle plugin has to deal with installing a whole Python environment. (This was a ton of work -- though seemingly doesn't have a lot of ongoing work).
  • For other platforms, we have the complexity of installing glean_parser into a virtual environment for use in build-time scripts
  • Though we don't use it now, using the standard build.rs pattern isn't (directly) available to us -- it would require a wrapper at the commandline level, probably
  • [Not sure if it's harder than the alternatives within mach which is already Python based, but chutten has the battle scars there]

• Integrating Glean into largish Python applications requires resolving a complex dependency space where Glean/glean_parser's dependencies overlap with the applications (see issues with mozregression in particular)

On the plus side, there are some instances where glean_parser being in Python integrates naturally:

• probe_scraper just uses glean_parser as a Python library
• The Glean Python bindings use glean_parser as a library to parse the metrics at runtime

The obvious alternative is to rewrite glean_parser in Rust. This would have some obvious upsides:

• General Rust safety guarantees / fewer classes of bugs possible
• Distributing a standalone Rust app doesn't have to concern itself with complexities of Python virtual environments and having a Python environment around
• Probably a slightly higher familiarity with Rust over Python on the team overall (and that's definitely where the team expertise is trending)

However, we should make sure this doesn't make our life worse:

• Is building Rust binaries for all supported platforms and shipping those to developers for use at build time easier than what we've built for Python? (Note our developers are not expected / required to have a rustup/cargo environment).
• For offline builds, will running prebuilt binaries at build time be considered acceptable from a security perspective? (Not a showstopper as we could require building it from source locally for the offline use case if necessary).
• For integration with probe_scraper, the Rust tool could probably output what it needs as JSON. This is pretty close to how probe_scraper interacts with glean_parser as an API now.
• For the Glean Python bindings, we would probably compiled the Rust version of glean_parser into the Rust shared object being used, and use the same JSON-based API proposed above to interface with it.
• Is there a Rust templating system compatible with Jinja2 templates such that we don't have to rewrite all of the templates? The highest cost of glean_parser is probably in the templates.

Comment 1

[Chris H-C :chutten|PTO (back July 13)]

Pointing slightly at https://bugzilla.mozilla.org/show_bug.cgi?id=1635260#c6 to note that GNOME embedders of SpiderMonkey might also not be best pleased to have to build or run Rust as part of their JS build. Or it might be okay. Hard to say, and since this would be generating a no-op API anyway, perhaps we put no-op generation in its own project.

Not a firm requirement one way or another at this stage.

Comment 2

[Jan-Erik Rediger [:janerik]]

Not a full answer, but just a couple of drive-by comments.
A full answer might need some more thinking on my side.

(In reply to Michael Droettboom [:mdroettboom] from comment #0)

• Is building Rust binaries for all supported platforms and shipping those to developers for use at build time easier than what we've built for
  Python? (Note our developers are not expected / required to have a rustup/cargo environment).

Not hard to build, we already solved it for mdbook-dtmo.
m-c also expects some Rust tools to be available (cbindgen) and can install it for you if wanted (though there they do expect to have a Rust env around).
Installing it is a bit annoying for users, but if we wrap that into some shell script it's solvable.

• For offline builds, will running prebuilt binaries at build time be considered acceptable from a security perspective? (Not a showstopper as we could require building it from source locally for the offline use case if necessary).

Given that's how Gradle & the miniconda environment works I would be surprised if that's not ok. We should still check.

[...]

• Is there a Rust templating system compatible with Jinja2 templates such that we don't have to rewrite all of the templates? The highest cost of glean_parser is probably in the templates.

Yes, askama

Comment 3

[Chris H-C :chutten|PTO (back July 13)]

An additional wrinkle to consider: in mozilla-central, the Python that uses glean_sdk transitively depends on glean_parser for runtime reasons (the Python binding generates APIs from metrics.yaml at runtime) so that mach can send Telemetry... while at the same time Project FOG has written a build script run by mach which is trying to use the vendored glean_parser for buildtime reasons. (see bug 1654663 and bug 1635260).

Will a Rust glean_parser make this better or worse?