1155160 - Remove the vendor directory and list all packages there in the requirements files instead

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P3)

Description

[Ed Morley [:emorley]]

Once bug 1146184 lands, it becomes trivial to maintain the packages in use locally/in production in a deterministic manner. As such, we can move all of the packages currently listed in checked-in.txt (which is only used to generate the vendor directory) into common.txt (which gets peep installed into the virtualenv) and then remove the vendor directory entirely.

This will:
* Make grepping the repo (or using the github web search) return fewer false positives.
* Avoid the "update checked-in.txt -> run the vendor directory creation script on a platform that actually works -> have to manually change a few things because the script is broken -> check in the result" dance, which has to be explained to contributors wanting to add/update packages.
* Increase consistency, since at the moment we're neither sticking to one approach nor another throughout (not that we could check everything into vendor/ if we wanted, due to some of our packages having compiled components).

Comment 1

[Ed Morley [:emorley]]

Attachment: Remove the vendor directory and use requirements files for all packages

Not urgent review - bug bug 1146184 will need to land / have some testing in production before I land this anyway :-)

Comment 2

[William Lachance (:wlach)]

Be sure that all the packages in here have up-to-date releases on pypi before landing this. I'm pretty sure treeherder-client hasn't been updated on pypi since my timeout change (bug 1144138), for example.

Comment 3

[Ed Morley [:emorley]]

(In reply to William Lachance (:wlach) from comment #2)
> Be sure that all the packages in here have up-to-date releases on pypi
> before landing this. I'm pretty sure treeherder-client hasn't been updated
> on pypi since my timeout change (bug 1144138), for example.

This PR just moves the requirements files lines from checked-in.txt to common.txt. As such, if the packages didn't exist, we wouldn't have been able to generate them in vendor to start with. For packages that are not released to pypi, we already use the URL format:
https://github.com/mozilla/treeherder-service/commit/07cea7aedac46f3c4c56c8d89d5b3fb19fbba190#diff-b80ca39e5ba09203c741fcc8e103bd16R75

Comment 4

[Ed Morley [:emorley]]

The SHA for treeherder-client was updated as part of bug 1144138, so that's fine:
https://github.com/mozilla/treeherder-service/commit/bb2e7c435613f83b7b8cfdbdf89b091da93588ed

Comment 5

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/a5295870a8d7baf1d307e696e7a43508967e2a11
Bug 1155160 - Move packages in checked-in.txt to common.txt

Now that we're using virtualenvs and peep to manage packages in
production, there's no need to use an in-repo vendor directory. As
such, all packages that were in checked-in.txt have been moved to
common.txt, so they will now be peep installed during deployment/testing
and also during the provision of the Vagrant environment.

https://github.com/mozilla/treeherder/commit/8cdc06ec26b761d1eaeffca895b82f909adbd14f
Bug 1155160 - Remove duplicate entry for the 'requests' package

Previously, the requests package had to be listed in dev.txt even
though it was in the vendor directory, since it was used by conftest.py
before the vendor directory was added to the Python path. Now that the
packages in checked-in.txt have been moved to common.txt, 'requests' is
listed in two requirements files that are peep installed, so we can
remove the dupe.

https://github.com/mozilla/treeherder/commit/0e188d4d9b15f26591b995611ba76c61642fa190
Bug 1155160 - Stop referring to compiled vs pure packages

This differentiation was only useful when explaining which packages
could be listed in which requirements file (since compiled packages
could not be added to checked-in.txt). Now that all packages are peep
installed, common.txt contains both pure and compiled packages.

https://github.com/mozilla/treeherder/commit/878262b358655da5b31df066da3a61f4bcf60bfe
Bug 1155160 - Remove the vendor directory from the Python path

All packages in the vendor directory are now installed in the virtualenv
site-packages, so there's no need to add the in-repo vendor directory to
the Python path.

https://github.com/mozilla/treeherder/commit/eaf0c2a79274419c123f12534d9f1c081e477248
Bug 1155160 - Remove script for generating the vendor directory

We're no longer using the vendor directory & this script wasn't entirely
reliable anyway, so let's remove it. The virtualenv package can be
removed from dev.txt, since virtualenv is installed globally, and
nothing inside our virtualenv (which is where the packages in dev.txt
end up) needs a local installation of it.

https://github.com/mozilla/treeherder/commit/461e28b3fbd0e9fa2f7475edea566eb8b2b99557
Bug 1155160 - Remove the now unused vendor directory