1328940 - mach install fails with: cannot import name _psutil_linux

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, enhancement)

Description

[Ralph Giles (:rillian)]

./mach install fails on my machine:

$ ./mach install
Error running mach:

    ['install']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

ImportError: cannot import name _psutil_linux

  File "/home/giles/firefox/python/mozbuild/mozbuild/mach_commands.py", line 1088, in install
    from mozrunner.devices.android_device import verify_android_device
  File "/home/giles/firefox/build/mach_bootstrap.py", line 373, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/giles/firefox/testing/mozbase/mozrunner/mozrunner/devices/android_device.py", line 9, in <module>
    import psutil
  File "/home/giles/firefox/build/mach_bootstrap.py", line 373, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/giles/firefox/python/psutil/psutil/__init__.py", line 63, in <module>
    from . import _pslinux as _psplatform
  File "/home/giles/firefox/build/mach_bootstrap.py", line 373, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "/home/giles/firefox/python/psutil/psutil/_pslinux.py", line 24, in <module>
    from . import _psutil_linux as cext

I've run ./mach bootstrap and am using the android toolchain it installed.

Comment 1

[Ralph Giles (:rillian)]

This is Fedora 25 on x86_64 linux. The python2-psutil package is installed.

Typing `make` in `python/psutil` results in the following error:

> gcc: error: /usr/lib/rpm/redhat/redhat-hardened-cc1: No such file or directory
> error: command 'gcc' failed with exit status 1

Comment 2

[Ralph Giles (:rillian)]

The issue is this switch passed to gcc: `-specs=/usr/lib/rpm/redhat/redhat-hardened-cc1`

Installing `redhat-rpm-config` provides the missing file. I don't see this in the makefile itself, so it might be a fedora python packaging bug. We should implement the workaround in mach bootstrap in the meantime.

Comment 3

[Ralph Giles (:rillian)]

Attachment: Bug 1328940 - mozboot: Install hardened cc for python.

Work around missing redhat-hardened-cc1 when building psutil_linux
on Fedora by installing the redhat-rpm-config package. This unbreaks
`./mach install` for fennec builds.

Review commit: https://reviewboard.mozilla.org/r/102706/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/102706/

Comment 4

[Nathan Froyd [:froydnj]]

Comment on attachment 8824169 [details]
Bug 1328940 - mozboot: Install hardened cc for python.

https://reviewboard.mozilla.org/r/102706/#review103042

Do you have insight into the question below?

::: python/mozboot/mozboot/centosfedora.py:78
(Diff revision 1)
>                  'python-dbus',
>              ]
>  
>              self.mobile_android_packages += [
>                  'java-1.8.0-openjdk-devel',
> +                'redhat-rpm-config',

It is curious that this isn't required for desktop browser development, since I'd expect the same bits of python to be executed for both?

Comment 5

[Ralph Giles (:rillian)]

Comment on attachment 8824169 [details]
Bug 1328940 - mozboot: Install hardened cc for python.

https://reviewboard.mozilla.org/r/102706/#review103086

::: python/mozboot/mozboot/centosfedora.py:78
(Diff revision 1)
>                  'python-dbus',
>              ]
>  
>              self.mobile_android_packages += [
>                  'java-1.8.0-openjdk-devel',
> +                'redhat-rpm-config',

I didn't investigate at the time, since I'd only hit the issue with deploying fennec builds, but it looks like it's used generally for resource measurement. However, that code is optional, in case we haven't bootstrapped yet when mach is invoked.

`mach doctor` also tries to build it, but seems to work when that fails.

`mach mochitest` *does* require it, so we should install it for all uses, not just fennec.

Comment 6

[Ralph Giles (:rillian)]

Comment on attachment 8824169 [details]
Bug 1328940 - mozboot: Install hardened cc for python.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/102706/diff/1-2/

Comment 7

[Ralph Giles (:rillian)]

Comment on attachment 8824169 [details]
Bug 1328940 - mozboot: Install hardened cc for python.

https://reviewboard.mozilla.org/r/102706/#review103088

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/34282e47da35
mozboot: Install hardened cc for python. r=froydnj

Comment 9

[Iris Hsiao [:ihsiao]]

https://hg.mozilla.org/mozilla-central/rev/34282e47da35

Comment 10

[Gregory Szorc [:gps]]

We've historically surrounded all "import psutil" statements inside a `try...except Exception:` because psutil has a history of failing to compile or work, especially on Windows. I'd love to make it a required dependency because it's such a nice library for inspecting process and system state.

It looks like testing/mozbase/mozrunner/mozrunner/devices/android_device.py isn't using psutil as an optional dependency (it uses psutil for inspecting running process names). So I guess psutil is no longer an optional dependency for some Fennec tasks. TIL.