1194074 - make psutil a dependency of Talos

Status: RESOLVED FIXED

(Testing :: Talos, defect)

Description

[Julien Pagès (:parkouss)]

Adding psutil as a Talos dependency will help us in at least two ways:

 - add some counters (bug 1193707 for example)
 - remove custom Talos code to get process list and kill them (ffprocess.py)

There is a working patch in bug 1193707 comment 10 - this needs to be reworked a bit, but it is on the good road I think.

Comment 1

[Julien Pagès (:parkouss)]

:gps, we would need to install psutil when running talos jobs on harness.

We added in internal pypi:

[ ]	psutil-3.1.1-cp27-none-win32.whl	12-Aug-2015 05:48 	86K	 
[ ]	psutil-3.1.1-cp27-none-win_amd64.whl	12-Aug-2015 05:48 	88K	 
[ ]	psutil-3.1.1.tar.gz	12-Aug-2015 05:47 	241K	 

And with the following mozharness patch, I am able to require psutil from talos, even on windows:

https://hg.mozilla.org/try/rev/27c2318c59b5

try is good, and looking at the logs the right psutil is installed:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f4dd950feab2

The patch basically does two things:

 - require pip >= 1.5 before installing dependencies (pip >=1.5 will use wheels when available)
 - install psutil >= 0.7.1 instead of ==0.7.1 in testing/mozharness/mozharness/base/python.py

I'm a bit worried about the second point, as it will not impact talos only. But that should work if:

 - only mozsystemmonitor uses the psutil installed in harness (https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/base/python.py#443)
 - mozsystemmonitor works with psutil >=0.7.1 (https://dxr.mozilla.org/mozilla-central/source/testing/mozbase/mozsystemmonitor/setup.py#28)


So, do you think that patch is applicable ? or should I try to impact talos only ? or something else ?

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

gps, it would be great to hear from you here so we can move forward.

Comment 3

[Gregory Szorc [:gps]]

Sorry for delayed response - on PTO for a long weekend and traveling yesterday.

mozsystemmonitor likely hasn't been explicitly tested with psutil 3.1. You should manually run tests against that version and make sure things still work. Upgrading the in-tree copy of psutil to 3.1.1 would also be a good idea to ensure we stay compatible with 3.1+ going forward.

Comment 4

[Gregory Szorc [:gps]]

I would also be very aggressive about using the newest available psutil version. If you are going to deploy 3.1.1 somewhere, we should see about upgrading everything to using 3.1.1+ if possible.

Comment 5

[Chris Manchester (limited bugmail, email directly)]

I just ran the mozsystemmonitor tests (testing/mozbase/mozsystemmonitor/mozsystemmonitor/test/test_resource_monitor.py) locally with psutil 3.1.1 and didn't encounter any failures.

Comment 6

[Julien Pagès (:parkouss)]

Attachment: 1194074.patch

So I just added some comments, but this is the same patch as in comment 1. So try was good for Talos, and as :chmanchester run the mozsystemmonitor tests with psutil 3.1.1, we should be good to land I think.

Comment 7

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/6e2dea2674da

Comment 8

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/6e2dea2674da

Comment 9

[Chris Manchester (limited bugmail, email directly)]

Attachment: MozReview Request: Bug 1194074 - Install psutil 3.1.1 in the desktop_unittests vitualenv so it will be available to desktop unittests. r=jlund

Bug 1194074 - Install psutil 3.1.1 in the desktop_unittests vitualenv so it will be available to desktop unittests. r=jlund

Comment 10

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8652567 [details]
MozReview Request: Bug 1194074 - Install psutil 3.1.1 in the desktop_unittests vitualenv so it will be available to desktop unittests. r=jlund

Sorry, wrong bug.