1526021 - Refactor mozboot into a (build) context free library

Status: NEW

(Firefox Build System :: Bootstrap Configuration, enhancement)

Description

[Andrew Halberstadt [:ahal]]

Right now mozboot is a glorified script. You invoke |mach bootstrap| and a sequence of setup steps get run. There is a lot of branching that can happen based on environment and platform, but at the end of the day the steps are all hardcoded one after the other.

I've long thought it would be useful if mozboot could be made more generic. I.e, implement the ability to install things various ways (apt-get, dnf, pip, wget tar, etc) and then allowing the consumers of mozboot to define their own bootstrap recipes.

In other words, it would be awesome if mozboot didn't have any build context baked into it. Instead, we would move the current implementation of |mach bootstrap| under python/mozbuild.

We could also create new bootstrap recipes for other parts of the dev workflow. For example to install lint and formatting dependencies when running |mach lint| or |mach clang-format|.

In an ideal world, we could prompt developers to bootstrap only the packages necessary to complete their current task at hand without forcing them to run a separate |mach bootstrap| command. This means we wouldn't need to prompt them to re-run |mach bootstrap| from time to time, and they wouldn't waste time and disk space installing packages they'll never end up using.

I'm filing this bug just to get the idea down in writing somewhere.

Comment 1

[Andrew Halberstadt [:ahal]]

I envision a bootstrap.toml/yml file that can live next to the mach_commands.py or code that needs them. Either the mach command could call an entrypoint in mozboot at runtime, passing in a path to a bootstrap.toml or devs could run ./mach bootstrap <path> to do the process manually.

We would curate a list of 'default' bootstrap configs that get invoked when running ./mach bootstrap without arguemnts (e.g build dependencies, lint dependencies, etc).

Comment 2

[Nick Alexander :nalexander [he/him]]

I think this is a pretty great idea, 'cuz over in Bug 1543247 I'm doing exactly this kind of "task-specific bootstrapping". In some ways, fetching artifacts for artifact builds feels similar, too -- I want that kind of cache management for bootstrapping. This would be great!

Comment 3

[Māris Fogels [:mars] (please needinfo)]

It's worth mentioning that, should we adopt a Starlark-based build tool in the future like Bazel, support for task-specific dependency bootstrapping may come built in and we could leverage it.

For example, I did a quick experiment in Bazel where I created a mach-google build target as a proxy for the mach google command. Running my new mach-google tool with bazel run //tools:mach-google can find or fetch the dependencies for a python toolchain, find or fetch the dependencies for the mach tool, build them, then execute the tool itself. And Bazel has full local and remote dependency cache support.

It could be worth exploring. Perhaps we could take the experiment further and move the complexity of per-command dependency discovery and execution into the build tool: try mach as a wrapper around bazel run and bazel build. Maybe the mozboot code for per-OS dependency discovery and installation could move into the build tool too, or at least be driven by it.