Closed Bug 917346 Opened 7 years ago Closed 2 years ago

Use newer psutil, install from wheel

Categories

(Release Engineering :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WORKSFORME

People

(Reporter: gps, Unassigned)

References

Details

(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2544] )

Attachments

(3 files, 1 obsolete file)

Bug 899599 is packaging a modern psutil as a wheel. We should modify mozharnes to support installing from wheels and we should grab a newer psutil version.
Trivial patch. Just changing psutil version number. Can't land until we
have the 1.0.1 package staged everywhere, preferably as a wheel.
Attachment #806035 - Flags: review?(aki)
Assignee: nobody → gps
I think this should "just work." If a wheel is available, the virtualenv
populator will use it. This is probably a good idea for global behavior
since wheels rock. If a wheel isn't available, it should fall back to
source install.
Attachment #806039 - Flags: review?(aki)
Comment on attachment 806039 [details] [diff] [review]
Part 2: Support installing Python packages from wheels

In my testing, the psutil wheel package could not install on a mw32-ix-slaveXX host which ESR17 uses.  We may need to adjust this patch to take that into account (eg. detect a NO_PIP_WHEEL env var and set that in buildbot-configs).
(In reply to John Hopkins (:jhopkins) from comment #3)
> Comment on attachment 806039 [details] [diff] [review]
> Part 2: Support installing Python packages from wheels
> 
> In my testing, the psutil wheel package could not install on a
> mw32-ix-slaveXX host which ESR17 uses.  We may need to adjust this patch to
> take that into account (eg. detect a NO_PIP_WHEEL env var and set that in
> buildbot-configs).

We can lock this down for ESR17 release automation by adjusting https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla/release-firefox-mozilla-esr17.py.template#L72

Not sure if dep builds depend on it or whether or not we'd be able to lock it down for them.
Attachment #806035 - Flags: review?(aki) → review+
(In reply to John Hopkins (:jhopkins) from comment #3)
> Comment on attachment 806039 [details] [diff] [review]
> Part 2: Support installing Python packages from wheels
> 
> In my testing, the psutil wheel package could not install on a
> mw32-ix-slaveXX host which ESR17 uses.  We may need to adjust this patch to
> take that into account (eg. detect a NO_PIP_WHEEL env var and set that in
> buildbot-configs).

I don't think esr17 runs anything on mozharness... am I wrong?
Comment on attachment 806039 [details] [diff] [review]
Part 2: Support installing Python packages from wheels

We should be able to see this go green on Cedar (default mozharness branch) before pushing live to the production branch.
Attachment #806039 - Flags: review?(aki) → review+
Adding :jhopkins for comment 5, and this one:

Looks like we uploaded a psutil 1.0.1 for win32 in bug 899599.
We'll need at least the base psutil 1.0.1 package (non-wheel) on puppetagain to land these patches without regressing our resource monitoring.
So I'm testing my changes for mozharness talos with the default branch of mozharness and it looks like this broke the pip install for mozpoolclient in the venv on the foopies

12:00:15     INFO - Installing mozpoolclient into virtualenv /builds/panda-0318/test/build/venv
12:00:15     INFO - Running command: ['/builds/panda-0318/test/build/venv/bin/pip', 'install', '--use-wheel', '--download-cache', '/builds/panda-0318/test/build/venv/cache', '--find-links', 'http://repos/python/packages', 'mozpoolclient'] in /builds/panda-0318/test/build
12:00:15     INFO - Copy/paste: /builds/panda-0318/test/build/venv/bin/pip install --use-wheel --download-cache /builds/panda-0318/test/build/venv/cache --find-links http://repos/python/packages mozpoolclient
12:00:15     INFO -  Usage: /builds/panda-0318/test/build/venv/bin/pip install [OPTIONS] PACKAGE_NAMES...
12:00:15     INFO -  /builds/panda-0318/test/build/venv/bin/pip install: error: no such option: --use-wheel

Could this be fixed otherwise it will break things in production for mozharness non-talos tests :-)
We'll need pip 1.4+ for wheel support. pip likely comes from virtualenv, so we'll need virtualenv 1.10+ everywhere.

Where does virtualenv come from in mozharness land? I don't think it would be the worst idea in the world to check virtualenv into the mozharness repo. Otherwise, a very low-level component of the stack is at the mercy of the system. virtualenv 1.10 and pip 1.4 are relatively new, so I don't think we can trust the system's virtualenv particularly well.
We install it via /tools/misc-python/virtualenv.py which lists it as virtualenv_version = "1.7.1.2"

2:01:44     INFO - Running main action method: create_virtualenv
12:01:44     INFO - Creating virtualenv /builds/panda-0318/test/build/venv
12:01:44     INFO - Running command: ['/tools/buildbot/bin/python', '/tools/misc-python/virtualenv.py', '--no-site-packages', '--distribute', '/builds/panda-0318/test/build/venv'] in /builds/panda-0318/test/build
12:01:44     INFO - Copy/paste: /tools/buildbot/bin/python /tools/misc-python/virtualenv.py --no-site-packages --distribute /builds/panda-0318/test/build/venv
12:01:44     INFO -  The --no-site-packages flag is deprecated; it is now the default behavior.
12:01:44     INFO -  Using real prefix '/tools/python27'
12:01:44     INFO -  New python executable in /builds/panda-0318/test/build/venv/bin/python
12:01:49     INFO -  Installing distribute.............................................................................................................................................................................................done.
12:01:53     INFO -  Installing pip..............done.
12:01:53     INFO - Return code: 0
12:01:53     INFO - Installing mozpoolclient into virtualenv /builds/panda-0318/test/build/venv
12:01:53     INFO - Running command: ['/builds/panda-0318/test/build/venv/bin/pip', 'install', '--use-wheel', '--download-cache', '/builds/panda-0318/test/build/venv/cache', '--find-links', 'http://repos/python/packages', 'mozpoolclient'] in /builds/panda-0318/test/build
12:01:53     INFO - Copy/paste: /builds/panda-0318/test/build/venv/bin/pip install --use-wheel --download-cache /builds/panda-0318/test/build/venv/cache --find-links http://repos/python/packages mozpoolclient
12:01:53     INFO -  Usage: /builds/panda-0318/test/build/venv/bin/pip install [OPTIONS] PACKAGE_NAMES...
12:01:53     INFO -  /builds/panda-0318/test/build/venv/bin/pip install: error: no such option: --use-wheel
12:01:53    ERROR - Return code: 2
12:01:53    FATAL - Unable to install mozpoolclient!

http://dev-master01.build.scl1.mozilla.com:8036/builders/Android%204.0%20Panda%20cedar%20talos%20remote-trobopan/builds/9/steps/run_script/logs/stdio
The system-provided virtualenv may not be as modern as we require. It is
easier to distribute it as part of mozharness than to rely on
system-provided versions.

--distribute does nothing in virtualenv 1.10.1, so it has been removed.

pip and setuptools are provided by virtualenv, so there is no need to
download them. They have been removed.
Attachment #806194 - Flags: review?(jhopkins)
Backed out the previous two patches so mtabara can continue working on cedar; we'll need to land all 3 when we're ready.
Attachment #806194 - Flags: review?(jhopkins) → review?(aki)
I can test these three patches on ash when my todo list gets a little less hectic.
(In reply to Aki Sasaki [:aki] from comment #14)
> I can test these three patches on ash when my todo list gets a little less
> hectic.

https://tbpl.mozilla.org/?tree=Ash&rev=de4a247ddeea
Windows bustage across the board for talos+unittests:
15:05:39    FATAL - caught OS error 8: %1 is not a valid Win32 application while running ['C:\\slave\\test\\scripts\\external_tools\\virtualenv\\virtualenv.py', '--no-site-packages', 'C:\\slave\\test\\build\\venv']
15:05:39    FATAL - Running post_fatal callback...
15:05:39    FATAL - Exiting -1
15:05:39     INFO - Running post-action listener: _resource_record_post_action
15:05:39     INFO - Running post-action listener: _start_resource_monitoring
15:05:39    ERROR - Exception during post-action for create-virtualenv: Traceback (most recent call last):
15:05:39    ERROR -   File "C:\slave\test\scripts\mozharness\base\script.py", line 1065, in run_action
15:05:39    ERROR -     method(action, success=success and self.return_code == 0)
15:05:39    ERROR -   File "C:\slave\test\scripts\mozharness\base\python.py", line 419, in _start_resource_monitoring
15:05:39    ERROR -     self.activate_virtualenv()
15:05:39    ERROR -   File "C:\slave\test\scripts\mozharness\base\python.py", line 391, in activate_virtualenv
15:05:39    ERROR -     execfile(activate, dict(__file__=activate))
15:05:39    ERROR - IOError: [Errno 2] No such file or directory: 'C:\\slave\\test\\build\\venv\\Scripts\\activate_this.py'
15:05:39    FATAL - Aborting due to failure in post-action listener.
15:05:39    FATAL - Running post_fatal callback...
15:05:39    FATAL - Exiting -1

Linux talos bustage: looks like we do everything in /builds/slave/talos-slave/test/build but our venv is in /home/cltbld/talos-slave/test/build/venv for some reason, and doesn't get clobbered.  (We have to clobber the virtualenv for every mozharness job everywhere to avoid bustage with this patch; I also found this in my hg-git conversion working directories.)
14:31:47    ERROR - Return code: 2
14:31:47    FATAL - Unable to install PyYAML!
14:31:47    FATAL - Running post_fatal callback...
14:31:47    FATAL - Exiting -1 f

Linux debug browser-chrome bustage: looks like we have to prepend the python 27 path before the virtualenv path ?
14:59:39     INFO - Copy/paste: /home/cltbld/talos-slave/test/scripts/external_tools/virtualenv/virtualenv.py --no-site-packages /home/cltbld/talos-slave/test/build/venv
14:59:39     INFO -  ERROR: None
14:59:39     INFO -  ERROR: this script requires Python 2.6 or greater.
14:59:39    ERROR - Return code: 101
14:59:39    FATAL - Halting on failure while running ['/home/cltbld/talos-slave/test/scripts/external_tools/virtualenv/virtualenv.py', '--no-site-packages', '/home/cltbld/talos-slave/test/build/venv']
14:59:39    FATAL - Running post_fatal callback...
14:59:39    FATAL - Exiting 101

Emulator unittest bustage: same
15:10:00     INFO - Copy/paste: /home/cltbld/talos-slave/test/scripts/external_tools/virtualenv/virtualenv.py --no-site-packages /home/cltbld/talos-slave/test/build/venv
15:10:01     INFO -  ERROR: None
15:10:01     INFO -  ERROR: this script requires Python 2.6 or greater.
15:10:01    ERROR - Return code: 101
15:10:01    FATAL - Halting on failure while running ['/home/cltbld/talos-slave/test/scripts/external_tools/virtualenv/virtualenv.py', '--no-site-packages', '/home/cltbld/talos-slave/test/build/venv']
15:10:01    FATAL - Running post_fatal callback...
15:10:01    FATAL - Exiting 101
15:10:01     INFO - Running post-action listener: _resource_record_post_action
15:10:01     INFO - Running post-action listener: _start_resource_monitoring
15:10:01    ERROR - Exception during post-action for create-virtualenv: Traceback (most recent call last):
15:10:01    ERROR -   File "/home/cltbld/talos-slave/test/scripts/mozharness/base/script.py", line 1065, in run_action
15:10:01    ERROR -     method(action, success=success and self.return_code == 0)
15:10:01    ERROR -   File "/home/cltbld/talos-slave/test/scripts/mozharness/base/python.py", line 419, in _start_resource_monitoring
15:10:01    ERROR -     self.activate_virtualenv()
15:10:01    ERROR -   File "/home/cltbld/talos-slave/test/scripts/mozharness/base/python.py", line 391, in activate_virtualenv
15:10:01    ERROR -     execfile(activate, dict(__file__=activate))
15:10:01    ERROR - IOError: [Errno 2] No such file or directory: '/home/cltbld/talos-slave/test/build/venv/bin/activate_this.py'
15:10:01    FATAL - Aborting due to failure in post-action listener.
15:10:01    FATAL - Running post_fatal callback...
15:10:01    FATAL - Exiting -1
Comment on attachment 806194 [details] [diff] [review]
Part 3: Distribute and install virtualenv from repository

Minusing for bustage.
Attachment #806194 - Flags: review?(aki) → review-
I'm not actively working on this. Remaining work seems to be a lot of $PATH foo in buildbot land?
Assignee: gps → nobody
Attachment #806039 - Attachment is obsolete: true
Assignee: nobody → gps
Status: NEW → ASSIGNED
Comment on attachment 8452617 [details] [diff] [review]
Part 2: Support installing Python packages from wheels

wtf bzexport. this is not what I wanted to upload.
Attachment #8452617 - Flags: review?(catlee)
Assignee: gps → nobody
Status: ASSIGNED → NEW
Comment on attachment 8452617 [details] [diff] [review]
Part 2: Support installing Python packages from wheels

Review of attachment 8452617 [details] [diff] [review]:
-----------------------------------------------------------------

Looks ok to me. Which versions of pip is this supported in?
Attachment #8452617 - Flags: review+
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/2544]
(In reply to Aki Sasaki (not actively reading bugmail) from comment #8)
> Adding :jhopkins for comment 5, and this one:
> 
> Looks like we uploaded a psutil 1.0.1 for win32 in bug 899599.
> We'll need at least the base psutil 1.0.1 package (non-wheel) on puppetagain
> to land these patches without regressing our resource monitoring.

Just uploaded the non-wheel tarball. 

selena@taiping:~/repos/build/braindump/utils 10:26 ♥: ./publish_package_our_pypi.sh psutil-1.0.1.tar.gz 
psutil-1.0.1.tar.gz                                                                                                                                                                                                                               100%  153KB 152.9KB/s   00:00    
`psutil-1.0.1.tar.gz' -> `/mnt/netapp/relengweb/pypi/pub/psutil-1.0.1.tar.gz'
removed `psutil-1.0.1.tar.gz'

Testing http://pypi.pub.build.mozilla.org/pub/psutil-1.0.1.tar.gz ...
HTTP/1.1 200 OK
Date: Thu, 04 Jun 2015 17:26:19 GMT
Server: Apache
X-Backend-Server: web2.releng.webapp.scl3.mozilla.com
Last-Modified: Thu, 04 Jun 2015 17:26:17 GMT
ETag: "26364"
Accept-Ranges: bytes
Content-Length: 156516
Content-Type: application/x-gzip
(In reply to Chris AtLee [:catlee] from comment #21)
> Comment on attachment 8452617 [details] [diff] [review]
> Part 2: Support installing Python packages from wheels
> 
> Review of attachment 8452617 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Looks ok to me. Which versions of pip is this supported in?

pip version 1.4+ and virtualenv 1.10+
https://pypi.python.org/pypi/psutil is now listing .whl files, at least for Windows. That should make things slightly easier, since we won't have to generate wheels on that platform.
I'm assuming this has been done elsewhere, or isn't a valid request any more.
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → WORKSFORME
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.