There's a newish feature wherein mach commands can define a custom virtualenv, e.g like this. This allows them to define custom dependencies in the corresponding file under
python/sites, for example. These dependencies can be from pypi, the vendored dir or even just add a directory to the PYTHONPATH.
All commands inherit the packages in the
mach.txt site. This is supposed to be the set of packages that are needed for Mach core itself. However it also includes all of the vendored packages under
third_party/python. I am pretty sure we did this because most Mach commands just assume these packages are available in their import scope, and it was easier to maintain a backwards compatible migration path if this was still the case in the new system.
However! The problem now is that all sites must remain compatible with the
mach site. For example, if a site wants to use the latest version of
requests, it can't because the vendored
requests is pinned to an older version and listed in
So to allow commands to use newer versions of vendored packages and reduce the number of compatibility errors in general, we should remove as much as possible from the
mach.txt site, such that it contains only the packages necessary to run
There are two approaches we could take here, the easy one and the one that will minimize conflicts even further.
The Easy Approach
- First we move all vendor and pth dependencies out of the
mach site (unless it is actually needed by mach), and into the
common site (which is currently empty).
- Then, for every command that needs to access the vendored packages, we ensure that it either has
virtualenv_name=common in the Mach command definition, OR if it already is using a site, we add:
The Fewer Conflicts Approach
The easy approach should work, but might still result in conflicts. Commands need to either be compatible with all vendors, or not use them at all with no in-between. So instead of putting everything in the
common site, we could go through each command one by one, determine exactly which vendors (if any) are needed, and then create a custom site just for that command which lists exactly the vendored libraries that it needs. Of course, related commands could still share a site if their set of dependencies are very similar.
In addition to having fewer conflicts, this approach would force us to explicitly register which packages are being used. While it might be a bit more work, I think this is very valuable when trying to determine which vendors are being used by which commands, and what needs to be tested when updating them!
Given all of the above, I'd recommend we first implement the easy approach just to get the ball rolling and get all commands onto the new "sites" system. Once we have that, I'd recommend we spend the effort going through all commands and figuring out exactly what is needed for each one. I think this will pay big dividends over the long run.