## Background There's a newish feature wherein mach commands can define a custom virtualenv, e.g [like this](https://searchfox.org/mozilla-central/rev/d6a131ceb435c03ccab2592578f6e2ebf12c1644/python/mach_commands.py#92). This allows them to define custom dependencies in the corresponding file under `python/sites`, [for example](https://searchfox.org/mozilla-central/source/python/sites/python-test.txt). These dependencies can be from pypi, the vendored dir or even just add a directory to the PYTHONPATH. All commands inherit the packages in the `mach.txt` site. This is supposed to be the set of packages that are needed for Mach core itself. However it also includes all of the vendored packages under `third_party/python`. I am pretty sure we did this because most Mach commands just assume these packages are available in their import scope, and it was easier to maintain a backwards compatible migration path if this was still the case in the new system. However! The problem now is that all sites must remain compatible with the `mach` site. For example, if a site wants to use the latest version of `requests`, it can't because the vendored `requests` is pinned to an older version and listed in `python/sites/mach.txt`. So to allow commands to use newer versions of vendored packages and reduce the number of compatibility errors in general, we should remove as much as possible from the `mach.txt` site, such that it contains *only* the packages necessary to run `mach` itself. There are two approaches we could take here, the easy one and the one that will minimize conflicts even further. ## The Easy Approach 1. First we move all vendor and pth dependencies out of the `mach` site (unless it is actually needed by mach), and into the `common` site (which is currently empty). 2. Then, for every command that needs to access the vendored packages, we ensure that it either has `virtualenv_name=common` in the Mach command definition, OR if it already is using a site, we add: ``` packagest.txt:python/sites/common.txt ``` to it. ## The Fewer Conflicts Approach The easy approach should work, but might still result in conflicts. Commands need to either be compatible with all vendors, or not use them at all with no in-between. So instead of putting everything in the `common` site, we could go through each command one by one, determine exactly which vendors (if any) are needed, and then create a custom site just for that command which lists exactly the vendored libraries that it needs. Of course, related commands could still share a site if their set of dependencies are very similar. In addition to having fewer conflicts, this approach would force us to explicitly register which packages are being used. While it might be a bit more work, I think this is very valuable when trying to determine which vendors are being used by which commands, and what needs to be tested when updating them! ## Recommendation Given all of the above, I'd recommend we first implement the easy approach just to get the ball rolling and get all commands onto the new "sites" system. Once we have that, I'd recommend we spend the effort going through all commands and figuring out exactly what is needed for each one. I think this will pay big dividends over the long run.
Bug 1808732 Comment 0 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
## Background There's a newish feature wherein mach commands can define a custom virtualenv, e.g [like this](https://searchfox.org/mozilla-central/rev/d6a131ceb435c03ccab2592578f6e2ebf12c1644/python/mach_commands.py#92). This allows them to define custom dependencies in the corresponding file under `python/sites`, [for example](https://searchfox.org/mozilla-central/source/python/sites/python-test.txt). These dependencies can be from pypi, the vendored dir or even just add a directory to the PYTHONPATH. All commands inherit the packages in the `mach.txt` site. This is supposed to be the set of packages that are needed for Mach core itself. However it also includes all of the vendored packages under `third_party/python`. I am pretty sure we did this because most Mach commands just assume these packages are available in their import scope, and it was easier to maintain a backwards compatible migration path if this was still the case in the new system. However! The problem now is that all sites must remain compatible with the `mach` site. For example, if a site wants to use the latest version of `requests`, it can't because the vendored `requests` is pinned to an older version and listed in `python/sites/mach.txt`. So to allow commands to use newer versions of vendored packages and reduce the number of compatibility errors in general, we should remove as much as possible from the `mach.txt` site, such that it contains *only* the packages necessary to run `mach` itself. There are two approaches we could take here, the easy one and the one that will minimize conflicts even further. ## The Easy Approach 1. First we move all vendor and pth dependencies out of the `mach` site (unless it is actually needed by mach), and into the `common` site (which is currently empty). 2. Then, for every command that needs to access the vendored packages, we ensure that it either has `virtualenv_name=common` in the Mach command definition, OR if it already is using a site, we add: ``` packages.txt:python/sites/common.txt ``` to it. ## The Fewer Conflicts Approach The easy approach should work, but might still result in conflicts. Commands need to either be compatible with all vendors, or not use them at all with no in-between. So instead of putting everything in the `common` site, we could go through each command one by one, determine exactly which vendors (if any) are needed, and then create a custom site just for that command which lists exactly the vendored libraries that it needs. Of course, related commands could still share a site if their set of dependencies are very similar. In addition to having fewer conflicts, this approach would force us to explicitly register which packages are being used. While it might be a bit more work, I think this is very valuable when trying to determine which vendors are being used by which commands, and what needs to be tested when updating them! ## Recommendation Given all of the above, I'd recommend we first implement the easy approach just to get the ball rolling and get all commands onto the new "sites" system. Once we have that, I'd recommend we spend the effort going through all commands and figuring out exactly what is needed for each one. I think this will pay big dividends over the long run.