917346 - Use newer psutil, install from wheel

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect)

Description

[Gregory Szorc [:gps]]

Bug 899599 is packaging a modern psutil as a wheel. We should modify mozharnes to support installing from wheels and we should grab a newer psutil version.

Comment 1

[Gregory Szorc [:gps]]

Attachment: Part 1: Upgrade psutil to 1.0.1

Trivial patch. Just changing psutil version number. Can't land until we
have the 1.0.1 package staged everywhere, preferably as a wheel.

Comment 2

[Gregory Szorc [:gps]]

Attachment: Part 2: Support installing Python packages from wheels

I think this should "just work." If a wheel is available, the virtualenv
populator will use it. This is probably a good idea for global behavior
since wheels rock. If a wheel isn't available, it should fall back to
source install.

Comment 3

[John Hopkins (:jhopkins)]

Comment on attachment 806039 [details] [diff] [review]
Part 2: Support installing Python packages from wheels

In my testing, the psutil wheel package could not install on a mw32-ix-slaveXX host which ESR17 uses.  We may need to adjust this patch to take that into account (eg. detect a NO_PIP_WHEEL env var and set that in buildbot-configs).

Comment 4

[[e-mail] (:bhearsum)]

(In reply to John Hopkins (:jhopkins) from comment #3)
> Comment on attachment 806039 [details] [diff] [review]
> Part 2: Support installing Python packages from wheels
> 
> In my testing, the psutil wheel package could not install on a
> mw32-ix-slaveXX host which ESR17 uses.  We may need to adjust this patch to
> take that into account (eg. detect a NO_PIP_WHEEL env var and set that in
> buildbot-configs).

We can lock this down for ESR17 release automation by adjusting https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla/release-firefox-mozilla-esr17.py.template#L72

Not sure if dep builds depend on it or whether or not we'd be able to lock it down for them.

Comment 5

[Aki Sasaki (not active)]

(In reply to John Hopkins (:jhopkins) from comment #3)
> Comment on attachment 806039 [details] [diff] [review]
> Part 2: Support installing Python packages from wheels
> 
> In my testing, the psutil wheel package could not install on a
> mw32-ix-slaveXX host which ESR17 uses.  We may need to adjust this patch to
> take that into account (eg. detect a NO_PIP_WHEEL env var and set that in
> buildbot-configs).

I don't think esr17 runs anything on mozharness... am I wrong?

Comment 6

[Aki Sasaki (not active)]

Comment on attachment 806039 [details] [diff] [review]
Part 2: Support installing Python packages from wheels

We should be able to see this go green on Cedar (default mozharness branch) before pushing live to the production branch.

Comment 7

[Gregory Szorc [:gps]]

https://hg.mozilla.org/build/mozharness/rev/7f989a1f7ee2
https://hg.mozilla.org/build/mozharness/rev/fc38bbc65c0f

Let's see what happens!

Comment 8

[Aki Sasaki (not active)]

Adding :jhopkins for comment 5, and this one:

Looks like we uploaded a psutil 1.0.1 for win32 in bug 899599.
We'll need at least the base psutil 1.0.1 package (non-wheel) on puppetagain to land these patches without regressing our resource monitoring.

Comment 9

[Kim Moir [:kmoir] ET]

So I'm testing my changes for mozharness talos with the default branch of mozharness and it looks like this broke the pip install for mozpoolclient in the venv on the foopies

12:00:15     INFO - Installing mozpoolclient into virtualenv /builds/panda-0318/test/build/venv
12:00:15     INFO - Running command: ['/builds/panda-0318/test/build/venv/bin/pip', 'install', '--use-wheel', '--download-cache', '/builds/panda-0318/test/build/venv/cache', '--find-links', 'http://repos/python/packages', 'mozpoolclient'] in /builds/panda-0318/test/build
12:00:15     INFO - Copy/paste: /builds/panda-0318/test/build/venv/bin/pip install --use-wheel --download-cache /builds/panda-0318/test/build/venv/cache --find-links http://repos/python/packages mozpoolclient
12:00:15     INFO -  Usage: /builds/panda-0318/test/build/venv/bin/pip install [OPTIONS] PACKAGE_NAMES...
12:00:15     INFO -  /builds/panda-0318/test/build/venv/bin/pip install: error: no such option: --use-wheel

Could this be fixed otherwise it will break things in production for mozharness non-talos tests :-)

Comment 10

[Gregory Szorc [:gps]]

We'll need pip 1.4+ for wheel support. pip likely comes from virtualenv, so we'll need virtualenv 1.10+ everywhere.

Where does virtualenv come from in mozharness land? I don't think it would be the worst idea in the world to check virtualenv into the mozharness repo. Otherwise, a very low-level component of the stack is at the mercy of the system. virtualenv 1.10 and pip 1.4 are relatively new, so I don't think we can trust the system's virtualenv particularly well.

Comment 11

[Kim Moir [:kmoir] ET]

We install it via /tools/misc-python/virtualenv.py which lists it as virtualenv_version = "1.7.1.2"

2:01:44     INFO - Running main action method: create_virtualenv
12:01:44     INFO - Creating virtualenv /builds/panda-0318/test/build/venv
12:01:44     INFO - Running command: ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py', '--no-site-packages', '--distribute', '/builds/panda-0318/test/build/venv'] in /builds/panda-0318/test/build
12:01:44     INFO - Copy/paste: /tools/buildbot/bin/python /tools/misc-python/virtualenv.py --no-site-packages --distribute /builds/panda-0318/test/build/venv
12:01:44     INFO -  The --no-site-packages flag is deprecated; it is now the default behavior.
12:01:44     INFO -  Using real prefix '/tools/python27'
12:01:44     INFO -  New python executable in /builds/panda-0318/test/build/venv/bin/python
12:01:49     INFO -  Installing distribute.............................................................................................................................................................................................done.
12:01:53     INFO -  Installing pip..............done.
12:01:53     INFO - Return code: 0
12:01:53     INFO - Installing mozpoolclient into virtualenv /builds/panda-0318/test/build/venv
12:01:53     INFO - Running command: ['/builds/panda-0318/test/build/venv/bin/pip', 'install', '--use-wheel', '--download-cache', '/builds/panda-0318/test/build/venv/cache', '--find-links', 'http://repos/python/packages', 'mozpoolclient'] in /builds/panda-0318/test/build
12:01:53     INFO - Copy/paste: /builds/panda-0318/test/build/venv/bin/pip install --use-wheel --download-cache /builds/panda-0318/test/build/venv/cache --find-links http://repos/python/packages mozpoolclient
12:01:53     INFO -  Usage: /builds/panda-0318/test/build/venv/bin/pip install [OPTIONS] PACKAGE_NAMES...
12:01:53     INFO -  /builds/panda-0318/test/build/venv/bin/pip install: error: no such option: --use-wheel
12:01:53    ERROR - Return code: 2
12:01:53    FATAL - Unable to install mozpoolclient!

http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20cedar%20talos%20remote-trobopan/builds/9/steps/run_script/logs/stdio

Comment 12

[Gregory Szorc [:gps]]

Attachment: Part 3: Distribute and install virtualenv from repository

The system-provided virtualenv may not be as modern as we require. It is
easier to distribute it as part of mozharness than to rely on
system-provided versions.

--distribute does nothing in virtualenv 1.10.1, so it has been removed.

pip and setuptools are provided by virtualenv, so there is no need to
download them. They have been removed.

Comment 13

[Aki Sasaki (not active)]

Backed out the previous two patches so mtabara can continue working on cedar; we'll need to land all 3 when we're ready.

Comment 14

[Aki Sasaki (not active)]

I can test these three patches on ash when my todo list gets a little less hectic.

Comment 15

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #14)
> I can test these three patches on ash when my todo list gets a little less
> hectic.

https://tbpl.mozilla.org/?tree=Ash&rev=de4a247ddeea

Comment 16

[Aki Sasaki (not active)]

Windows bustage across the board for talos+unittests:
15:05:39    FATAL - caught OS error 8: %1 is not a valid Win32 application while running ['C:\\slave\\test\\scripts\\external_tools\\virtualenv\\virtualenv.py', '--no-site-packages', 'C:\\slave\\test\\build\\venv']
15:05:39    FATAL - Running post_fatal callback...
15:05:39    FATAL - Exiting -1
15:05:39     INFO - Running post-action listener: _resource_record_post_action
15:05:39     INFO - Running post-action listener: _start_resource_monitoring
15:05:39    ERROR - Exception during post-action for create-virtualenv: Traceback (most recent call last):
15:05:39    ERROR -   File "C:\slave\test\scripts\mozharness\base\script.py", line 1065, in run_action
15:05:39    ERROR -     method(action, success=success and self.return_code == 0)
15:05:39    ERROR -   File "C:\slave\test\scripts\mozharness\base\python.py", line 419, in _start_resource_monitoring
15:05:39    ERROR -     self.activate_virtualenv()
15:05:39    ERROR -   File "C:\slave\test\scripts\mozharness\base\python.py", line 391, in activate_virtualenv
15:05:39    ERROR -     execfile(activate, dict(__file__=activate))
15:05:39    ERROR - IOError: [Errno 2] No such file or directory: 'C:\\slave\\test\\build\\venv\\Scripts\\activate_this.py'
15:05:39    FATAL - Aborting due to failure in post-action listener.
15:05:39    FATAL - Running post_fatal callback...
15:05:39    FATAL - Exiting -1

Linux talos bustage: looks like we do everything in /builds/slave/talos-slave/test/build but our venv is in /home/cltbld/talos-slave/test/build/venv for some reason, and doesn't get clobbered.  (We have to clobber the virtualenv for every mozharness job everywhere to avoid bustage with this patch; I also found this in my hg-git conversion working directories.)
14:31:47    ERROR - Return code: 2
14:31:47    FATAL - Unable to install PyYAML!
14:31:47    FATAL - Running post_fatal callback...
14:31:47    FATAL - Exiting -1 f

Linux debug browser-chrome bustage: looks like we have to prepend the python 27 path before the virtualenv path ?
14:59:39     INFO - Copy/paste: /home/cltbld/talos-slave/test/scripts/external_tools/virtualenv/virtualenv.py --no-site-packages /home/cltbld/talos-slave/test/build/venv
14:59:39     INFO -  ERROR: None
14:59:39     INFO -  ERROR: this script requires Python 2.6 or greater.
14:59:39    ERROR - Return code: 101
14:59:39    FATAL - Halting on failure while running ['/home/cltbld/talos-slave/test/scripts/external_tools/virtualenv/virtualenv.py', '--no-site-packages', '/home/cltbld/talos-slave/test/build/venv']
14:59:39    FATAL - Running post_fatal callback...
14:59:39    FATAL - Exiting 101

Emulator unittest bustage: same
15:10:00     INFO - Copy/paste: /home/cltbld/talos-slave/test/scripts/external_tools/virtualenv/virtualenv.py --no-site-packages /home/cltbld/talos-slave/test/build/venv
15:10:01     INFO -  ERROR: None
15:10:01     INFO -  ERROR: this script requires Python 2.6 or greater.
15:10:01    ERROR - Return code: 101
15:10:01    FATAL - Halting on failure while running ['/home/cltbld/talos-slave/test/scripts/external_tools/virtualenv/virtualenv.py', '--no-site-packages', '/home/cltbld/talos-slave/test/build/venv']
15:10:01    FATAL - Running post_fatal callback...
15:10:01    FATAL - Exiting 101
15:10:01     INFO - Running post-action listener: _resource_record_post_action
15:10:01     INFO - Running post-action listener: _start_resource_monitoring
15:10:01    ERROR - Exception during post-action for create-virtualenv: Traceback (most recent call last):
15:10:01    ERROR -   File "/home/cltbld/talos-slave/test/scripts/mozharness/base/script.py", line 1065, in run_action
15:10:01    ERROR -     method(action, success=success and self.return_code == 0)
15:10:01    ERROR -   File "/home/cltbld/talos-slave/test/scripts/mozharness/base/python.py", line 419, in _start_resource_monitoring
15:10:01    ERROR -     self.activate_virtualenv()
15:10:01    ERROR -   File "/home/cltbld/talos-slave/test/scripts/mozharness/base/python.py", line 391, in activate_virtualenv
15:10:01    ERROR -     execfile(activate, dict(__file__=activate))
15:10:01    ERROR - IOError: [Errno 2] No such file or directory: '/home/cltbld/talos-slave/test/build/venv/bin/activate_this.py'
15:10:01    FATAL - Aborting due to failure in post-action listener.
15:10:01    FATAL - Running post_fatal callback...
15:10:01    FATAL - Exiting -1

Comment 17

[Aki Sasaki (not active)]

Comment on attachment 806194 [details] [diff] [review]
Part 3: Distribute and install virtualenv from repository

Minusing for bustage.

Comment 18

[Gregory Szorc [:gps]]

I'm not actively working on this. Remaining work seems to be a lot of $PATH foo in buildbot land?

Comment 19

[Gregory Szorc [:gps]]

Attachment: Part 2: Support installing Python packages from wheels

Comment 20

[Gregory Szorc [:gps]]

Comment on attachment 8452617 [details] [diff] [review]
Part 2: Support installing Python packages from wheels

wtf bzexport. this is not what I wanted to upload.

Comment 21

[Chris AtLee [:catlee]]

Comment on attachment 8452617 [details] [diff] [review]
Part 2: Support installing Python packages from wheels

Review of attachment 8452617 [details] [diff] [review]:
-----------------------------------------------------------------

Looks ok to me. Which versions of pip is this supported in?

Comment 22

[Selena Deckelmann :selenamarie :selena]

(In reply to Aki Sasaki (not actively reading bugmail) from comment #8)
> Adding :jhopkins for comment 5, and this one:
> 
> Looks like we uploaded a psutil 1.0.1 for win32 in bug 899599.
> We'll need at least the base psutil 1.0.1 package (non-wheel) on puppetagain
> to land these patches without regressing our resource monitoring.

Just uploaded the non-wheel tarball. 

selena@taiping:~/repos/build/braindump/utils 10:26 ♥: ./publish_package_our_pypi.sh psutil-1.0.1.tar.gz 
psutil-1.0.1.tar.gz                                                                                                                                                                                                                               100%  153KB 152.9KB/s   00:00    
`psutil-1.0.1.tar.gz' -> `/mnt/netapp/relengweb/pypi/pub/psutil-1.0.1.tar.gz'
removed `psutil-1.0.1.tar.gz'

Testing http://pypi.pub.build.mozilla.org/pub/psutil-1.0.1.tar.gz ...
HTTP/1.1 200 OK
Date: Thu, 04 Jun 2015 17:26:19 GMT
Server: Apache
X-Backend-Server: web2.releng.webapp.scl3.mozilla.com
Last-Modified: Thu, 04 Jun 2015 17:26:17 GMT
ETag: "26364"
Accept-Ranges: bytes
Content-Length: 156516
Content-Type: application/x-gzip

Comment 23

[Selena Deckelmann :selenamarie :selena]

(In reply to Chris AtLee [:catlee] from comment #21)
> Comment on attachment 8452617 [details] [diff] [review]
> Part 2: Support installing Python packages from wheels
> 
> Review of attachment 8452617 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Looks ok to me. Which versions of pip is this supported in?

pip version 1.4+ and virtualenv 1.10+

Comment 24

[Gregory Szorc [:gps]]

https://pypi.python.org/pypi/psutil is now listing .whl files, at least for Windows. That should make things slightly easier, since we won't have to generate wheels on that platform.

Comment 25

[Chris AtLee [:catlee]]

I'm assuming this has been done elsewhere, or isn't a valid request any more.