843426 - utilize pip for objdir virtualenv package installation

Status: NEW

(Firefox Build System :: General, defect)

Description

[Jeff Hammel]

currently we use http://mxr.mozilla.org/mozilla-central/source/build/virtualenv/populate_virtualenv.py to install python modules into the objdir virtualenv.  While this works currently, I would propose that if we do have to extend what we have to do with packages much (and, ideally, even if we don't, time-constraints aside) we move to pip to do as much of this as possible: http://www.pip-installer.org/en/latest/ .

We would need pip to handle most of http://mxr.mozilla.org/mozilla-central/source/build/virtualenv/packages.txt as possible;  for instance, it currently does not install as .pth files.  `copy` -> site-packages would be another nice to have for pip.  We'd also probably have to interpolate packages.txt to make it wrt the topsrcdir, but that is easy. Talking here and there to the pip maintainers, I would guess they'd be open to such functionality.

I'm guessing we'd have to front-end pip in any case, but using a ad hoc standard off-the-shelf tool IMHO gives us not only a solid piece of work to build on vs maintaining yet-another python installation thing (this can also all be done with zc.buildout but, um, yeah) and give us both community exposure and hopefully help in that front.

I would say that priority depends on what more we might need for populate_virtualenv.py.  If we need something big, I would move priority up.  If what we have now is fine, it is low priority.

Comment 1

[Jeff Hammel]

The first thing to do is probably ticket the issues on github pypa/pip; e.g. adding .pth support.  This can gauge reception there.  The fix probably isn't hard; it also probably isn't something that will get done without intervention.

Comment 2

[Gregory Szorc [:gps]]

Tarek is involved in Python packaging and he might have some suggestions for us.

Tarek: For additional context, see the latter comments in bug 841713. tl;dr we've sort of rolled a custom package manager (populate_virtualenv.py) and we think it is silly for us to invent and maintain a one-off.

Comment 3

[Jeff Hammel]

Might as well loop Ian in since he's the original author though I have no idea if he touches the code anymore.

Comment 4

[Ian Bicking (:ianbicking)]

There's a bunch of parts of this that I don't exactly understand.  Can .pth files actually be static, instead of being generated by populate_virtualenv.py?  For installing other packages, "pip install path/to/foo" will call "python path/to/foo/setup.py install" which is maybe the other thing populate_virtualenv.py does?  I feel like there's something I'm missing here that is requiring a special solution.