Closed Bug 1116194 Opened 6 years ago Closed 6 years ago

System-installed psutil conflicts with tree psutil (AttributeError: 'module' object has no attribute 'linux_sysinfo')


(Firefox Build System :: General, defect)

Not set


(Not tracked)



(Reporter: gps, Assigned: gps)




(1 file)

+++ This bug was initially created as a clone of Bug #930808 +++

Upgrading psutil in bug 930808 didn't resolve the problem as I suspected it might. It looks like we have a problem of the system's psutil mixing with psutil .py files from python/psutil.

Apparent STR are:

1) Install Ubuntu 14.04 (or similar)
2) apt-get install python-psutil
3) mach build

I'll look into this today.
I can confirm STR.

1) New Ubuntu install
2) apt-get install python-psutil
3) Run
4) Clone mozilla-central
5) ./mach build

$ ./mach build
Error running mach:


The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error

The details of the failure are as follows:

AttributeError: 'module' object has no attribute 'linux_sysinfo'

  File "/home/gps/mozilla-central/python/mozbuild/mozbuild/", line 290, in build
  File "/home/gps/mozilla-central/python/mozbuild/mozbuild/controller/", line 164, in init
    self.resources = SystemResourceMonitor(poll_interval=1.0)
  File "/home/gps/mozilla-central/testing/mozbase/mozsystemmonitor/mozsystemmonitor/", line 188, in __init__
    virt = psutil.virtual_memory()
  File "/home/gps/mozilla-central/python/psutil/psutil/", line 1662, in virtual_memory
    ret = _psplatform.virtual_memory()
  File "/home/gps/mozilla-central/python/psutil/psutil/", line 144, in virtual_memory
    total, free, buffers, shared, _, _ = cext.linux_sysinfo()

This isn't going to be fun to fix :/
Duplicate of this bug: 1115538
Summary: System-installed psutil conflicts with tree psutil → System-installed psutil conflicts with tree psutil (AttributeError: 'module' object has no attribute 'linux_sysinfo')
Confirming my suspicion that our tree's psutil .py file is loading a C extension from the system:

-> ret = _psplatform.virtual_memory()
(Pdb) p _psplatform
<module 'psutil._pslinux' from '/home/gps/mozilla-central/python/psutil/psutil/_pslinux.pyc'>
(Pdb) s
> /home/gps/mozilla-central/python/psutil/psutil/
-> def virtual_memory():
(Pdb) n
> /home/gps/mozilla-central/python/psutil/psutil/
-> total, free, buffers, shared, _, _ = cext.linux_sysinfo()
(Pdb) p cext
<module '_psutil_linux' from '/usr/lib/python2.7/dist-packages/'>
The build system / mach currently has a very hacky virtualenv setup.
Essentially, it resorts to sys.path munging instead of a proper,
isolated environment.

During initialization, mach installs python/psutil in sys.path. Later
on, some code does an |import psutil|. This fails iff the psutil C
extension can't be found.

If there is a psutil C extension installed outside of mach and
python/psutil, |import psutil| may load it. The version mismatch isn't
detected until an extension-using psutil API is called. This has
manifested inside |mach build| via the resource monitor as an
|AttributeError: 'module' object has no attribute 'linux_sysinfo'|
exception during psutil.virtual_memory().

The proper fix for this is for the Python environment to ensure the
psutil C extension is built before attempting to import and use psutil.
Arguably, psutil itself should perform some kind of version check when
it imports the C extension to ensure things are in sync and fail at
import time.

Fixing mach and the build system Python environment to build psutil
earlier/properly is a long outstanding bug. It needs to be addressed.
But it is considerable effort. This patch continues the long history of
wallpapering over psutil import/run failures because using a proper
virutalenv from mach/build system is a lot of work. Sad panda.
Attachment #8542250 - Flags: review?(ted)
Assignee: nobody → gps
Attachment #8542250 - Flags: review?(ted) → review+
I filed so psutil hopefully becomes more strict about version mismatches.
Closed: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla37 is closed: the next release of psutil will raise an error if the C extension version doesn't match the source version!
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.