puppet support for installing virtualenvs full of packages

RESOLVED FIXED

Status

Release Engineering
General
P2
normal
RESOLVED FIXED
7 years ago
4 years ago

People

(Reporter: dustin, Assigned: dustin)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [puppet])

Attachments

(1 attachment, 2 obsolete attachments)

(Assignee)

Description

7 years ago
This will be a new Puppet module, 'python', capable of setting up arbitrary virtualenvs and populating them with desired packages.  It should do so with no external network access, instead using Puppet to distribute tarballs of the packages we wish to install.

It would be nice to also have this module install Python executables, but we don't need this at the moment.

Updated

7 years ago
Whiteboard: [puppet]
(Assignee)

Comment 1

7 years ago
Created attachment 513277 [details] [diff] [review]
m635007-puppet-manifests-r1.patch
Attachment #513277 - Flags: review?(bhearsum)
(Assignee)

Comment 2

7 years ago
see https://wiki.mozilla.org/ReleaseEngineering/How_To/Install_a_Python_Virtualenv_with_Puppet, too
(Assignee)

Comment 3

7 years ago
Comment on attachment 513277 [details] [diff] [review]
m635007-puppet-manifests-r1.patch

So the use of un-scoped global variables doesn't work.  In particular, these

+$package_dir = "/tools/python-packages"
+$package_dir_source = "puppet:///$level/python-packages"

basically, those variables only get set in the global scope if puppetmasterd happens to decide to re-parse the file when the client requests a catalog.  The puppetmaster will only re-parse the file if it has changed - it remembers classes and defines, but forgets variables.

It's a pretty ugly language, but at least if you know what to avoid you can work around it.

The workaround is to create a settings class, and use scoped variables.
Attachment #513277 - Flags: review?(bhearsum)
(Assignee)

Comment 4

7 years ago
Created attachment 513353 [details] [diff] [review]
m635007-puppet-manifests-r2.patch

Impressively, the error only caused every *second* invocation of puppetd to fail, which I assumed, for a while, was due to some weird caching problems.  At any rate, this is the new, recommended syntax (by Volcane on IRC, at any rate)
Attachment #513277 - Attachment is obsolete: true
Attachment #513353 - Flags: review?(bhearsum)
(Assignee)

Comment 5

7 years ago
Comment on attachment 513353 [details] [diff] [review]
m635007-puppet-manifests-r2.patch

This relies on getting python-devel installed on fed64, and possibly other stuff, so I'll hold off on the review until I've tested it all in staging.
Attachment #513353 - Flags: review?(bhearsum)
(Assignee)

Comment 6

7 years ago
Created attachment 515228 [details] [diff] [review]
m635007-puppet-manifests-r3.patch

OK, for real this time.  This has installed virtualenvs on several mac and linux staging slaves now, so I'm fairly confident in its efficacy.

This installs python packages by using pip to read from the puppet http root - packages are not stored on the slaves, and only the packages required are downloaded.  It uses pip-check.py to check for a package in the virutalenv, and only tries to install if that package is not found -- so no action is taken in the normal (already-installed) case.

I believe this can support installing the same packages in multiple virtualenvs, although I haven't verified that.  It's not required at this point anyway.
Attachment #513353 - Attachment is obsolete: true
Attachment #515228 - Flags: review?(bhearsum)
Attachment #515228 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 7

7 years ago
Comment on attachment 515228 [details] [diff] [review]
m635007-puppet-manifests-r3.patch

829ee80177dc

(note that this has no effect yet, since it's not used by any nodes)
Attachment #515228 - Flags: checked-in+
(Assignee)

Updated

7 years ago
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
Depends on: 637577
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.