893254 - Make psutil installable throughout automation

Status: RESOLVED WONTFIX

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Gregory Szorc [:gps]]

psutil (https://code.google.com/p/psutil/) is a nifty Python package for getting at detailed system and process information. We need it to record resource usage as part of running automation jobs (bug 859573). Aki tells me it needs to be added to an internal PyPI mirror or something.

What makes psutil special is it is a compiled Python extension (it has C source files). A machine needs a working compiler and the Python development headers to install psutil from source.

Alternatively, we should be able create a pre-built psutil package. However, this may require matching Python distributions everywhere (yay uniformity).

Depending on the timeline for this, a gradual deployment is acceptable. The work in bug 859573 can likely be adapted to make psutil optional. I would like 100% deployment some day, however.

Comment 1

[Aki Sasaki (not active)]

It's available: http://puppetagain.pub.build.mozilla.org/data/python/packages/psutil-0.7.1.tar.gz

However, we may hit compilation issues, as noted above.

Comment 2

[Aki Sasaki (not active)]

Also added http://puppetagain.pub.build.mozilla.org/data/python/packages/mozsystemmonitor-0.0.tar.gz .

Comment 3

[Aki Sasaki (not active)]

From a b2g emulator unittest run:

16:42:03     INFO -  building '_psutil_linux' extension
16:42:03     INFO -  creating build/temp.linux-i686-2.6
16:42:03     INFO -  creating build/temp.linux-i686-2.6/psutil
16:42:03     INFO -  gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i686 -mtune=atom -fasynchronous-unwind-tables -D_GNU_SOURCE -fPIC -fPIC -I/usr/include/python2.6 -c psutil/_psutil_linux.c -o build/temp.linux-i686-2.6/psutil/_psutil_linux.o
16:42:03     INFO -  unable to execute gcc: No such file or directory
16:42:03     INFO -  error: command 'gcc' failed with exit status 1
16:42:03     INFO -  ----------------------------------------
16:42:03     INFO -  Command /home/cltbld/talos-slave/test/build/venv/bin/python -c "import setuptools;__file__='/home/cltbld/talos-slave/test/build/venv/build/psutil/setup.py';execfile(__file__)" install --single-version-externally-managed --record /tmp/pip-LeTY4k-record/install-record.txt --install-headers /home/cltbld/talos-slave/test/build/venv/include/site/python2.6 failed with error code 1
16:42:03     INFO -  Storing complete log in /home/cltbld/.pip/pip.log
16:42:03    ERROR - Return code: 1
16:42:03    FATAL - Unable to install psutil==0.7.1!
16:42:03    FATAL - Exiting -1
16:42:03     INFO - Running post-action listener: _resource_record_post_action
16:42:03     INFO - Running post-action listener: _start_resource_monitoring
16:42:03    ERROR - Exception during post-action for create-virtualenv: Traceback (most recent call last):
16:42:03    ERROR -   File "/home/cltbld/talos-slave/test/scripts/mozharness/base/script.py", line 959, in run_action
16:42:03    ERROR -     method(action, success=success and self.return_code == 0)
16:42:03    ERROR -   File "/home/cltbld/talos-slave/test/scripts/mozharness/base/python.py", line 377, in _start_resource_monitoring
16:42:03    ERROR -     from mozsystemmonitor.resourcemonitor import SystemResourceMonitor
16:42:03    ERROR - ImportError: No module named mozsystemmonitor.resourcemonitor
16:42:03    FATAL - Aborting due to failure in post-action listener.
16:42:03    FATAL - Exiting -1
16:42:03     INFO - Running post-run listener: _resource_record_post_run
program finished with exit code 255

Comment 4

[Gregory Szorc [:gps]]

It looks like psutil fails to install on most everywhere except Linux. And even then it's not working everywhere. OS X problem appears to be missing compiler. Ditto for Windows.

I understand that you may not want compilers installed everywhere, so my guess is we'll soon be talking about a pre-compiled package.

Comment 5

[Gregory Szorc [:gps]]

This doesn't block resource monitoring landing. It just makes it work more places.

Comment 6

[Chris Cooper [:coop] (he/him)]

Can I get someone to itemize (In reply to Gregory Szorc [:gps] from comment #4)
> It looks like psutil fails to install on most everywhere except Linux. And
> even then it's not working everywhere. OS X problem appears to be missing
> compiler. Ditto for Windows.

Does that mean all the build/trybuild systems are already covered since they have compilers?

If so, that would leave the test systems:

Linux (rpm via mock)
talos-linux32-ix/tst-linux32-ec2/tst-linux32-spot
talos-linux64-ix/tst-linux64-spot/tst-linux64-ec2

Mac (dmg via puppet)
t-mavericks-r5
talos-mtnlion-r5
talos-r4-snow

Windows (installers via GPO)
t-w732-ix/t-xp32-ix
t-w864-ix/tst-w64-ec2

That's six packages total if my groupings are legit.

Comment 7

[Chris Cooper [:coop] (he/him)]

Also, how is this bug related to bug 899599 and bug 917346?

Comment 8

[Gregory Szorc [:gps]]

I /think/ bug 899599 tracks getting psutil to work inside the build system.

This and bug 917346 track getting psutil to play nice with mozharness's virtualenv.

There is probably room for duplication.

IMO, we should just use wheels for all Python packaging in automation. The results are guaranteed consistent and we don't have to worry about having the right compiler bits present on the machines.

Comment 9

[Selena Deckelmann :selenamarie :selena]

In bug 1050343 :markco added psutil support for a pkgzip install of psutil into C:\mozilla-build\sitepackages.

Can we reuse that somehow for the virtualenv?

Comment 10

[[e-mail] (:bhearsum)]

I don't think is relevant in the post-tcmigration world.