bugzilla.mozilla.org has resumed normal operation. Attachments prior to 2014 will be unavailable for a few days. This is tracked in Bug 1475801.
Please report any other irregularities here.

Remove the vendor directory and list all packages there in the requirements files instead



Tree Management
3 years ago
3 years ago


(Reporter: emorley, Assigned: emorley)




(1 attachment)



3 years ago
Once bug 1146184 lands, it becomes trivial to maintain the packages in use locally/in production in a deterministic manner. As such, we can move all of the packages currently listed in checked-in.txt (which is only used to generate the vendor directory) into common.txt (which gets peep installed into the virtualenv) and then remove the vendor directory entirely.

This will:
* Make grepping the repo (or using the github web search) return fewer false positives.
* Avoid the "update checked-in.txt -> run the vendor directory creation script on a platform that actually works -> have to manually change a few things because the script is broken -> check in the result" dance, which has to be explained to contributors wanting to add/update packages.
* Increase consistency, since at the moment we're neither sticking to one approach nor another throughout (not that we could check everything into vendor/ if we wanted, due to some of our packages having compiled components).

Comment 1

3 years ago
Created attachment 8593369 [details] [review]
Remove the vendor directory and use requirements files for all packages

Not urgent review - bug bug 1146184 will need to land / have some testing in production before I land this anyway :-)
Attachment #8593369 - Flags: review?(cdawson)
Be sure that all the packages in here have up-to-date releases on pypi before landing this. I'm pretty sure treeherder-client hasn't been updated on pypi since my timeout change (bug 1144138), for example.

Comment 3

3 years ago
(In reply to William Lachance (:wlach) from comment #2)
> Be sure that all the packages in here have up-to-date releases on pypi
> before landing this. I'm pretty sure treeherder-client hasn't been updated
> on pypi since my timeout change (bug 1144138), for example.

This PR just moves the requirements files lines from checked-in.txt to common.txt. As such, if the packages didn't exist, we wouldn't have been able to generate them in vendor to start with. For packages that are not released to pypi, we already use the URL format:

Comment 4

3 years ago
The SHA for treeherder-client was updated as part of bug 1144138, so that's fine:


3 years ago
Attachment #8593369 - Flags: review?(cdawson) → review+

Comment 5

3 years ago
Commits pushed to master at https://github.com/mozilla/treeherder

Bug 1155160 - Move packages in checked-in.txt to common.txt

Now that we're using virtualenvs and peep to manage packages in
production, there's no need to use an in-repo vendor directory. As
such, all packages that were in checked-in.txt have been moved to
common.txt, so they will now be peep installed during deployment/testing
and also during the provision of the Vagrant environment.

Bug 1155160 - Remove duplicate entry for the 'requests' package

Previously, the requests package had to be listed in dev.txt even
though it was in the vendor directory, since it was used by conftest.py
before the vendor directory was added to the Python path. Now that the
packages in checked-in.txt have been moved to common.txt, 'requests' is
listed in two requirements files that are peep installed, so we can
remove the dupe.

Bug 1155160 - Stop referring to compiled vs pure packages

This differentiation was only useful when explaining which packages
could be listed in which requirements file (since compiled packages
could not be added to checked-in.txt). Now that all packages are peep
installed, common.txt contains both pure and compiled packages.

Bug 1155160 - Remove the vendor directory from the Python path

All packages in the vendor directory are now installed in the virtualenv
site-packages, so there's no need to add the in-repo vendor directory to
the Python path.

Bug 1155160 - Remove script for generating the vendor directory

We're no longer using the vendor directory & this script wasn't entirely
reliable anyway, so let's remove it. The virtualenv package can be
removed from dev.txt, since virtualenv is installed globally, and
nothing inside our virtualenv (which is where the packages in dev.txt
end up) needs a local installation of it.

Bug 1155160 - Remove the now unused vendor directory


3 years ago
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.