899599 - Package/deploy psutil for virtualenv

Status: RESOLVED INCOMPLETE

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[John Hopkins (:jhopkins)]

In bug 781277, binary package 'psutil' was deployed to the c:\mozilla-build\python27 directory tree but that is not being picked up by the virtualenv created during the mozilla-central build, resulting in attempts to rebuild psutil.  Rebuilding locally is undesirable because psutil needs to be built using the same compiler that built the python binary (MSVS9) but we have deprecated that compiler.
We need to build/install psutil (on ix-mn-w0864-001) in such a way that the mozilla-build virtualenv includes that package and does not try and rebuild it.

Comment 1

[John Hopkins (:jhopkins)]

I copied the psutil package files to mozilla-build\buildbotve, activated the virtualenv, and ran into the following problem:

C:\mozilla-build\buildbotve\scripts>activate
(buildbotve) C:\mozilla-build\buildbotve\scripts>python
Python 2.6.5 (r265:79096, Mar 19 2010, 21:48:26) [MSC v.1500 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\mozilla-build\buildbotve\lib\site-packages\psutil\__init__.py", line
77, in <module>
    import psutil._psmswindows as _psplatform
  File "C:\mozilla-build\buildbotve\lib\site-packages\psutil\_psmswindows.py", l
ine 15, in <module>
    import _psutil_mswindows
ImportError: Module use of python27.dll conflicts with this version of Python.

I'll try and rebuild psutil on a w64-ix build slave with the virtualenv activated and see if that resolves this problem.

Comment 2

[Mark Cornmesser [:markco]]

Is there any update on this? I am wondering because this is currently blocking https://bugzilla.mozilla.org/show_bug.cgi?id=781277.

Comment 3

[John Hopkins (:jhopkins)]

There is a separate virtualenv that is created at build time which also wants psutil to be installed.  It does not make use of the \mozillabuild\buildbotve install of psutil.  We need to find a way to get psutil installed into this build-time virtualenv _without_ rebuilding psutil at build time.  It may be as simple as specifying --site-packages at virtualenv creation time, but I have not had a chance to investigate.  One complicating factor is that (afaik) 'pip' does not support installing pre-built binary packages.

gps, your thoughts?

Comment 4

[Gregory Szorc [:gps]]

If we need to hack the build system to find a binary psutil package outside of the tree, we can do that.

Erik: Do you have any tips for Python packaging to find and install pre-built binary packages? We essentially want to install/uncompress a pre-built binary package into a virtualenv. How can we do that? I consider .pth files a sufficient solution. I just don't know how to solve the discovery problem.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

You should be able to install this from a bdist package, right?  I know pip doesn't find bdists, but python::virtualenv could probably be modified to support them.

Once that's done, if you build the build-time virtualenv with --site-packages and using the buildbotve python, you should have access to the package.

The other option is to rewrite the functionality you need from psutil using ctypes.  I don't know how tricky that would be.

Comment 6

[Gregory Szorc [:gps]]

We are currently constructing the build system virtualenv with --site-packages, but we have bug 907902 open to change that. We explicitly don't want to pull in the system's site-packages because we have no clue what unwanted content may be present there. We want isolation and repeatability.

I'm willing to carve an exception for psutil or any other binary packages. Perhaps we should just check in a precompiled bdist archive into m-c and modify the build system virtualenv population logic to look for those before attempting source builds. But that only solves the problem for the build - mozharness is now attempting to load psutil, so we'll need a separate solution for automation.

Comment 7

[Erik Rose [:erik][:erikrose]]

gps: pip doesn't support binary packages at all, but easy_install does. It doesn't look like there's a Windows-compiled version of psutil on PyPI, but you could set up your own index server and then call easy_install for just the binary packages. Of course, the server would have to be public if you wanted to let the public run the build system, so that's a no-go. Alternatively, you could check a built psutil egg into your source tree and then see if you can get easy_install to install that. (You'd have to get easy_install in there somehow as a prerequisite.) I'm no expert on compiled packages on platforms that have no ready compiler, but I don't think there's any easy way now to do it; I think you're stuck duct-taping something like the above together.

In the Great Gleaming Future, we'll have Wheels, which are binary packages that can be installed simply by unpacking them into the right directory. Nobody offers wheels of their packages yet, so...

Aha! Okay, how about this: build a wheel of psutil for Windows, and ship it with your source tree. Then, use pip install --use-wheel --no-index --find-links=/your/wheels/dir etc. to install it. http://www.pip-installer.org/en/latest/cookbook.html#building-and-installing-wheels has the details. I think that'll solve your problem with a minimum of pain.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Can pip find wheels on a pypi mirror?  E.g., if we put them in http://puppetagain.pub.build.mozilla.org/data/python/packages?  That'd be super duper awesome.

Comment 9

[Erik Rose [:erik][:erikrose]]

Yep! http://www.pip-installer.org/en/latest/usage.html#install-use-wheel

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

So, John, you should just be able to build a wheel and drop it in the usual place.  You may also need to upgrade the version of pip, but that's not a bad idea anyway.  Let me know how it goes!

Comment 11

[John Hopkins (:jhopkins)]

Thanks, gps, dustin, erikrose for your insights!  I will look into this in detail next week.

Comment 12

[Gregory Szorc [:gps]]

I investigated wheel and it is awesome. I highly encourage releng/mozharness to use it heavily as an alternative to the traditional pip install mechanism (which uses setup.py). If nothing else, package installs should be much faster.

Comment 13

[John Hopkins (:jhopkins)]

Hit a failure when trying to import the psutil module which I installed from a wheel file:

(venv) C:\tmp>python
Python 2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\tmp\mozilla-central\python\psutil\psutil\__init__.py", line 85, in <module>
    import psutil._psmswindows as _psplatform
  File "C:\tmp\mozilla-central\python\psutil\psutil\_psmswindows.py", line 56, in <module>
    _psutil_mswindows.MIB_TCP_STATE_ESTAB : CONN_ESTABLISHED,
AttributeError: 'module' object has no attribute 'MIB_TCP_STATE_ESTAB'

Investigating...

Comment 14

[Gregory Szorc [:gps]]

Bug 908296. We can bump the psutil version used by mozharness since IIRC it is still on a 0.x release.

Comment 15

[John Hopkins (:jhopkins)]

Turns out I incorrectly assumed hg-shared was reasonably up-to-date on my build slave and was working with an older copy of mozilla-central.  Pulled in the latest changes, rebuilt my virtualenvs, rebuilt the wheel, and did a test install passed.

Note to self: we'll want to keep an older w64 build slave around with VC9 to do the wheel package building since the new replacements don't have VC9 installed.

Comment 16

[John Hopkins (:jhopkins)]

Next step: upload http://people.mozilla.org/~jhopkins/bug899599/psutil-1.0.1-cp27-none-win32.whl to our pip mirror.

Comment 17

[Gregory Szorc [:gps]]

Any machine with Python 2.7 + VS2008 should be able to produce wheels that can be used by our automation machines. A major goal of wheels is for binary distribution to "just work." So everything should be contained within the produced .whl. You're probably fine designating a saved VM image as the source of wheels.

Comment 18

[John Hopkins (:jhopkins)]

Uploaded psutil wheel file to http://puppetagain.pub.build.mozilla.org/data/python/packages/psutil-1.0.1-cp27-none-win32.whl

Will take up to 30 minutes to sync to the other puppetagain hosts.  --use-wheel is _not_ the default mode of pip so this is a no-op until we make the necessary in-tree changes.  mozharness will also need to point at a newer version of psutil.

Comment 19

[John Hopkins (:jhopkins)]

Will pick this up after the new win64 build slaves are rolled out (bug 781277) as it is not considered a blocker.

Comment 20

[Chris Cooper [:coop] (he/him)]

jhopkins: were you still planning to come back to this?

Comment 21

[John Hopkins (:jhopkins)]

coop: I'm currently focused on bug 886640 (and dependencies).  I have no specific plans to come back to this bug at the present time.