635007 - puppet support for installing virtualenvs full of packages

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

This will be a new Puppet module, 'python', capable of setting up arbitrary virtualenvs and populating them with desired packages.  It should do so with no external network access, instead using Puppet to distribute tarballs of the packages we wish to install.

It would be nice to also have this module install Python executables, but we don't need this at the moment.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: m635007-puppet-manifests-r1.patch

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

see https://wiki.mozilla.org/ReleaseEngineering/How_To/Install_a_Python_Virtualenv_with_Puppet, too

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 513277 [details] [diff] [review]
m635007-puppet-manifests-r1.patch

So the use of un-scoped global variables doesn't work.  In particular, these

+$package_dir = "/tools/python-packages"
+$package_dir_source = "puppet:///$level/python-packages"

basically, those variables only get set in the global scope if puppetmasterd happens to decide to re-parse the file when the client requests a catalog.  The puppetmaster will only re-parse the file if it has changed - it remembers classes and defines, but forgets variables.

It's a pretty ugly language, but at least if you know what to avoid you can work around it.

The workaround is to create a settings class, and use scoped variables.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: m635007-puppet-manifests-r2.patch

Impressively, the error only caused every *second* invocation of puppetd to fail, which I assumed, for a while, was due to some weird caching problems.  At any rate, this is the new, recommended syntax (by Volcane on IRC, at any rate)

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 513353 [details] [diff] [review]
m635007-puppet-manifests-r2.patch

This relies on getting python-devel installed on fed64, and possibly other stuff, so I'll hold off on the review until I've tested it all in staging.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: m635007-puppet-manifests-r3.patch

OK, for real this time.  This has installed virtualenvs on several mac and linux staging slaves now, so I'm fairly confident in its efficacy.

This installs python packages by using pip to read from the puppet http root - packages are not stored on the slaves, and only the packages required are downloaded.  It uses pip-check.py to check for a package in the virutalenv, and only tries to install if that package is not found -- so no action is taken in the normal (already-installed) case.

I believe this can support installing the same packages in multiple virtualenvs, although I haven't verified that.  It's not required at this point anyway.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 515228 [details] [diff] [review]
m635007-puppet-manifests-r3.patch

829ee80177dc

(note that this has no effect yet, since it's not used by any nodes)