Closed Bug 1436857 Opened 6 years ago Closed 6 years ago

Upgrade psutil Python package from version 3.1.1 to 5.4.3

Categories

(Firefox Build System :: General, enhancement, P3)

Product:
Firefox Build System
Component:
General
Version:
60 Branch
Type:
enhancement

Tracking

(firefox60 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla60
Tracking Flags:
Tracking Status
firefox60 --- fixed

People

(Reporter: whimboo, Assigned: whimboo)

References

Details

Attachments

(1 file, 1 obsolete file)

Bug 1436857 - Upgrade psutil from version 3.1.1 to 5.4.3.
59 bytes, text/x-review-board-request
gps
: review+
Details 
To run the mozprocess unit tests on Windows the psutil Python package is required. While we have the sources of this package in-tree [1], we cannot easily compile it due to the following requirements:

> <@ted> it has a C module, and those are a PITA to build on windows
> <@ted> because they have to be built with a version of MSVC that matches what python was built with
> <@ted> and for python 2.x that's like VC9

The alternative could be:

> <@ted> i assume it should be possible to publish a psutil wheel, and maybe there already is one?
> <@ted> https://pypi.python.org/pypi/psutil shows wheels of the latest release, certainly
> <ahal> we'd need still need the source for linux/mac + 2 windows wheels per python version we want to support

Gregory, given that you added the package via bug 783727, do you have an idea what might be best to do here? I never touched the build system at this level, so more suggestions are welcome.

[1] https://hg.mozilla.org/mozilla-central/file/tip/third_party/python/psutil
Flags: needinfo?(gps)
Also we are using a very outdated version of psutil: 3.1.1 vs. 5.4.3.

I tried to install the wheel locally for py27, and was only able to do so for the win32 version. The amd64 wheel failed with unsupported platform even I have Win7 64bit.
I tried to upgrade to version 5.4.3 and it looks all fine except that test machines miss the python(3)-dev package:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=8edcedcd659574e1eaee6e81e1e8b3f1b71216a2
Depends on: 1436941
Actually we don't want that package to be installed. The problems were related to two cases with a reference to version 3.1.1 of this package in the tree. Fixing those made it work:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=eb8e6990bf20f62793d28178ca465bf9c8a82bad

Interestingly I noticed that with the new version we don't need the pre-compiled code in the tree.

So the only question which remains now is how we could handle Windows, and if we could easily just use the wheel.
Turns out that just unpacking the wheel and putting the compiled C extension for Windows into the psutil folder works just fine. With that I can run the mozprocess tests.

I will upload a patch which includes a temporary commit to enable mozprocess unit tests on Windows.
Depends on: 1437163
Attachment #8949710 - Attachment is obsolete: true
With the mozprocess patch it failed to detect the compiled cext as valid DLL. Not sure why this happens because it works fine locally. Also other test jobs want to retrieve psutil via pip and fail, because the version 5.4.3 has not been made available yet on our internal PyPI mirror. I requested an upload. Once this is done I will request another try build.

Also after spoken to Andrew we might want to get rid of psutil in mozprocess, which is only used to check if a pid exists. On Windows we could perfectly do that via ctypes.
Assignee: nobody → hskupin
Status: NEW → ASSIGNED
Priority: -- → P3
Flags: needinfo?(gps)
Summary: psutil Python package under third party misses compiled C extension for Windows → Upgrade psutil Python package from version 3.1.1 to 5.4.3
No longer blocks: 892902
Attachment #8949655 - Flags: review?(gps)
Comment on attachment 8949655 [details]
Bug 1436857 - Upgrade psutil from version 3.1.1 to 5.4.3.

https://reviewboard.mozilla.org/r/219006/#review225356

Yay - thank you for upgrading this dated package!

One nit: the commit message says the Windows binary extension is committed. That isn't true. But I'm OK with that: if we commit the binary extension, it should be in its own commit, because that will likely require changes.

Also, historically it has been a PITA to get wheels working in CI. Blame the physical Windows buildbot machines, which I believe still to this day run a seriously ancient pip version that doesn't support wheels. If we get everything using the build-derived mozharness.zip file (or source checkouts), we could use the virtualenv/pip that is vendored in mozilla-central. See bug 1436612 for recent developments here.
Attachment #8949655 - Flags: review?(gps) → review+
Comment on attachment 8949655 [details]
Bug 1436857 - Upgrade psutil from version 3.1.1 to 5.4.3.

https://reviewboard.mozilla.org/r/219006/#review225356

Oh, yes this extra message is wrong. The compiled C extension has been removed again because it caused problems. I will remove that from the commit description.
Pushed by hskupin@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f3bfda748e33
Upgrade psutil from version 3.1.1 to 5.4.3. r=gps
https://hg.mozilla.org/mozilla-central/rev/f3bfda748e33
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
status-firefox60: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla60
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: