Closed Bug 1116194 Opened 6 years ago Closed 6 years ago
System-installed psutil conflicts with tree psutil (Attribute
Error: 'module' object has no attribute 'linux _sysinfo')
+++ This bug was initially created as a clone of Bug #930808 +++ Upgrading psutil in bug 930808 didn't resolve the problem as I suspected it might. It looks like we have a problem of the system's psutil mixing with psutil .py files from python/psutil. Apparent STR are: 1) Install Ubuntu 14.04 (or similar) 2) apt-get install python-psutil 3) mach build I'll look into this today.
I can confirm STR. 1) New Ubuntu install 2) apt-get install python-psutil 3) Run bootstraph.py 4) Clone mozilla-central 5) ./mach build $ ./mach build Error running mach: ['build'] The error occurred in code that was called by the mach command. This is either a bug in the called code itself or in the way that mach is calling it. You should consider filing a bug for this issue. If filing a bug, please include the full output of mach, including this error message. The details of the failure are as follows: AttributeError: 'module' object has no attribute 'linux_sysinfo' File "/home/gps/mozilla-central/python/mozbuild/mozbuild/mach_commands.py", line 290, in build monitor.init(warnings_path) File "/home/gps/mozilla-central/python/mozbuild/mozbuild/controller/building.py", line 164, in init self.resources = SystemResourceMonitor(poll_interval=1.0) File "/home/gps/mozilla-central/testing/mozbase/mozsystemmonitor/mozsystemmonitor/resourcemonitor.py", line 188, in __init__ virt = psutil.virtual_memory() File "/home/gps/mozilla-central/python/psutil/psutil/__init__.py", line 1662, in virtual_memory ret = _psplatform.virtual_memory() File "/home/gps/mozilla-central/python/psutil/psutil/_pslinux.py", line 144, in virtual_memory total, free, buffers, shared, _, _ = cext.linux_sysinfo() This isn't going to be fun to fix :/
Summary: System-installed psutil conflicts with tree psutil → System-installed psutil conflicts with tree psutil (AttributeError: 'module' object has no attribute 'linux_sysinfo')
Confirming my suspicion that our tree's psutil .py file is loading a C extension from the system: /home/gps/mozilla-central/python/psutil/psutil/__init__.py(1662)virtual_memory() -> ret = _psplatform.virtual_memory() (Pdb) p _psplatform <module 'psutil._pslinux' from '/home/gps/mozilla-central/python/psutil/psutil/_pslinux.pyc'> (Pdb) s --Call-- > /home/gps/mozilla-central/python/psutil/psutil/_pslinux.py(143)virtual_memory() -> def virtual_memory(): (Pdb) n > /home/gps/mozilla-central/python/psutil/psutil/_pslinux.py(144)virtual_memory() -> total, free, buffers, shared, _, _ = cext.linux_sysinfo() (Pdb) p cext <module '_psutil_linux' from '/usr/lib/python2.7/dist-packages/_psutil_linux.x86_64-linux-gnu.so'>
The build system / mach currently has a very hacky virtualenv setup. Essentially, it resorts to sys.path munging instead of a proper, isolated environment. During initialization, mach installs python/psutil in sys.path. Later on, some code does an |import psutil|. This fails iff the psutil C extension can't be found. If there is a psutil C extension installed outside of mach and python/psutil, |import psutil| may load it. The version mismatch isn't detected until an extension-using psutil API is called. This has manifested inside |mach build| via the resource monitor as an |AttributeError: 'module' object has no attribute 'linux_sysinfo'| exception during psutil.virtual_memory(). The proper fix for this is for the Python environment to ensure the psutil C extension is built before attempting to import and use psutil. Arguably, psutil itself should perform some kind of version check when it imports the C extension to ensure things are in sync and fail at import time. Fixing mach and the build system Python environment to build psutil earlier/properly is a long outstanding bug. It needs to be addressed. But it is considerable effort. This patch continues the long history of wallpapering over psutil import/run failures because using a proper virutalenv from mach/build system is a lot of work. Sad panda.
Attachment #8542250 - Flags: review?(ted)
Attachment #8542250 - Flags: review?(ted) → review+
I filed https://github.com/giampaolo/psutil/issues/564 so psutil hopefully becomes more strict about version mismatches.
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla37
https://github.com/giampaolo/psutil/issues/564 is closed: the next release of psutil will raise an error if the C extension version doesn't match the source version!
You need to log in before you can comment on or make changes to this bug.