The default bug view has changed. See this FAQ.

psutil breaks in Debian wheezy

RESOLVED FIXED in mozilla26

Status

()

Core
Build Config
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: micahflee, Assigned: gps)

Tracking

Trunk
mozilla26
x86_64
Linux
Points:
---
Bug Flags:
in-testsuite +

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [Run `hg status -in python/psutil | xargs rm` to reset psutil])

Attachments

(1 attachment)

(Reporter)

Description

4 years ago
User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130806 Firefox/17.0 Iceweasel/17.0.8 (Nightly/Aurora)
Build ID: 20130806234539

Steps to reproduce:

I'm running an up-to-date Debian wheezy, but I can't build mozilla-central due to psutil fatal errors. You should be able to reproduce by cloning the repo in wheezy, running:

./mach bootstrap

Then:

./mach build


Actual results:

micah@spock:~/projects/mozilla-central$ ./mach build
/home/micah/eff_projects/httpse/mozilla-central-httpse-tests/python/psutil/psutil/_pslinux.py:102: RuntimeWarning: couldn't determine platform's TOTAL_PHYMEM
  warnings.warn("couldn't determine platform's TOTAL_PHYMEM", RuntimeWarning)
Error running mach:

    ['build']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

AttributeError: 'module' object has no attribute 'get_sysinfo'

  File "/home/micah/eff_projects/httpse/mozilla-central-httpse-tests/python/mozbuild/mozbuild/mach_commands.py", line 299, in build
    monitor.init(warnings_path)
  File "/home/micah/eff_projects/httpse/mozilla-central-httpse-tests/python/mozbuild/mozbuild/controller/building.py", line 277, in init
    self.resources = SystemResourceMonitor(poll_interval=1.0)
  File "/home/micah/eff_projects/httpse/mozilla-central-httpse-tests/testing/mozbase/mozsystemmonitor/mozsystemmonitor/resourcemonitor.py", line 180, in __init__
    virt = psutil.virtual_memory()
  File "/home/micah/eff_projects/httpse/mozilla-central-httpse-tests/python/psutil/psutil/__init__.py", line 1154, in virtual_memory
    return _psplatform.virtual_memory()
  File "/home/micah/eff_projects/httpse/mozilla-central-httpse-tests/python/psutil/psutil/_pslinux.py", line 137, in virtual_memory
    total, free, buffers, shared, _, _ = _psutil_linux.get_sysinfo()


I think this is a bug with the version of psutil that's included with mozilla-central. The version I have installed in Debian seems to work to include:

micah@spock:~/projects/mozilla-central$ python
Python 2.7.3 (default, Jan  2 2013, 13:56:14) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> 


But when I load the mozilla-central version I get an error:

micah@spock:~/projects/mozilla-central/python/psutil$ ls
CREDITS  examples  HISTORY  LICENSE  MANIFEST.in  PKG-INFO  psutil  README  setup.py  test
micah@spock:~/projects/mozilla-central/python/psutil$ python
Python 2.7.3 (default, Jan  2 2013, 13:56:14) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
psutil/_pslinux.py:102: RuntimeWarning: couldn't determine platform's TOTAL_PHYMEM
  warnings.warn("couldn't determine platform's TOTAL_PHYMEM", RuntimeWarning)
>>> 



Expected results:

"./mach build" should have built mozilla-central.
(Assignee)

Comment 1

4 years ago
Perhaps we should upgrade the psutil in m-c. Will submit a patch to try momentarily.
(Assignee)

Comment 2

4 years ago
Created attachment 794322 [details] [diff] [review]
Upgrade psutil to version 1.0.1

Archive obtained from
https://psutil.googlecode.com/files/psutil-1.0.1.tar.gz and extracted
over existing source code without modifications.

https://tbpl.mozilla.org/?tree=Try&rev=b242a872ce53
Attachment #794322 - Flags: review?(mh+mozilla)
(Assignee)

Updated

4 years ago
Assignee: nobody → gps
Comment on attachment 794322 [details] [diff] [review]
Upgrade psutil to version 1.0.1

Review of attachment 794322 [details] [diff] [review]:
-----------------------------------------------------------------

rs=me
Attachment #794322 - Flags: review?(mh+mozilla) → review+
(Assignee)

Comment 4

4 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/ed0b6a11532d
Status: UNCONFIRMED → ASSIGNED
Component: Untriaged → Build Config
Ever confirmed: true
Product: Firefox → Core
Version: 17 Branch → Trunk
(Assignee)

Comment 5

4 years ago
Despite the conversions on IRC, I forgot to touch a virtualenv file when I landed this, so it caused some minor bustage on inbound. Pushed a trivial followup touching populate_virtualenv.py in a non-significant way.

https://hg.mozilla.org/integration/mozilla-inbound/rev/1ad8a6674785
https://hg.mozilla.org/mozilla-central/rev/ed0b6a11532d
https://hg.mozilla.org/mozilla-central/rev/1ad8a6674785
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Flags: in-testsuite+
Resolution: --- → FIXED
Target Milestone: --- → mozilla26
Backed out at the request of #developers for causing OSX mach bustage.
https://hg.mozilla.org/integration/mozilla-inbound/rev/31aa3eb1a40d
Target Milestone: mozilla26 → ---
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Comment 8

4 years ago
The update to psutil 1.0.1 broke mach under OS X (confirmed independently by two developers under both OSX 10.8 and OSX 10.9). The specific issue (which seems to be the same problem fixed by 1ad8a6674785) is the absence of a TCPS_ESTABLISHED attribute on the _psutil_osx module:

> $ ./mach build
> Error running mach:
> 
>     ['build']
> 
> The error occurred in code that was called by the mach command. This is either
> a bug in the called code itself or in the way that mach is calling it.
> 
> You should consider filing a bug for this issue.
> 
> If filing a bug, please include the full output of mach, including this error
> message.
> 
> The details of the failure are as follows:
> 
> AttributeError: 'module' object has no attribute 'TCPS_ESTABLISHED'
> 
>   File "/Users/Adam/devel/mozilla/mozilla-inbound/python/mozbuild/mozbuild/mach_commands.py", line 293, in build
>     from mozbuild.controller.building import BuildMonitor
>   File "/Users/Adam/devel/mozilla/mozilla-inbound/python/mozbuild/mozbuild/controller/building.py", line 22, in <module>
>     import psutil
>   File "/Users/Adam/devel/mozilla/mozilla-inbound/python/psutil/psutil/__init__.py", line 95, in <module>
>     import psutil._psosx as _psplatform
>   File "/Users/Adam/devel/mozilla/mozilla-inbound/python/psutil/psutil/_psosx.py", line 48, in <module>
>     _TCP_STATES_TABLE = {_psutil_osx.TCPS_ESTABLISHED : CONN_ESTABLISHED,
Removing

python/psutil/build
python/psutil/_psutil_osx.so
python/psutil/_psutil_posix.so

fixed my OS X build.

Comment 10

4 years ago
Wow. I'm surprised we have binary build output showing up in the tree in locations other than obj-*. Would it be possible to move these outputs into the same directory as the rest of the binary build objects so that clobbers clean them up?
(Assignee)

Comment 11

4 years ago
This is a dependency problem. The virtualenv needs to be repopulated before psutil can be imported or the aforementioned TCPS_ESTABLISHED error will occur. By backing this out, you've likely made the problem worse, as now developers who've upgraded will get errors due to newer psutil being used on import. The correct solution is to tell people to run `hg status -in python/psutil | xargs rm`.

(In reply to Adam Roach [:abr] from comment #10)
> Wow. I'm surprised we have binary build output showing up in the tree in
> locations other than obj-*. Would it be possible to move these outputs into
> the same directory as the rest of the binary build objects so that clobbers
> clean them up?

Known issue. It really sucks. It will take some knowledge of Python packaging to fix. It needs to be fixed. Fortunately, we've only upgraded psutil twice in over a year, so as annoying as it is, it's not frequent enough to highly prioritize, IMO.

Comment 12

4 years ago
(In reply to Gregory Szorc [:gps] from comment #11)
> The correct solution is to tell people to run `hg status -in
> python/psutil | xargs rm`.
 
This kind of "hallway lore" is very brittle -- most people who need it won't have access to it when they need it. Can we be careful to at least include this information in the bug the next time we update the package? Or, even better, put a try/catch around the package inclusion that takes care of running this command for developers?
(Assignee)

Comment 13

4 years ago
Relanded because the backout will cause bustage as well.

https://hg.mozilla.org/integration/mozilla-inbound/rev/037e6a08032f
Whiteboard: [Run `hg status -in python/psutil | xargs rm` to reset psutil]
FWIW, I don't think it's acceptable to break most developers trees on OS X and let them all figure out by themselves how to unbreak them.
(In reply to Gregory Szorc [:gps] from comment #11)
> The correct solution is to tell people to run `hg status -in python/psutil | xargs rm`.

For those using a clone of the Git mirror, an equivalent is:

git clean -xf python/psutil/

# -x to include ignored files; -f to force the clean (replace with -n to see what would be cleaned first)

Comment 16

4 years ago
(In reply to Peter Van der Beken [:peterv] from comment #14)
> FWIW, I don't think it's acceptable to break most developers trees on OS X
> and let them all figure out by themselves how to unbreak them.
I definitely agree!
We lived without it before, and now that will be painful while bisecting on OS X after that patch. What about landing a patch that would fix all bustages instead?
(Assignee)

Comment 17

4 years ago
Applied some wallpaper to catch all Exceptions when importing psutil. This should limit the impact of this patch. People building a changeset with new psutil but not this changeset may still run into issues, however. Perhaps we should cherry-pick it to central before the inbound merge?

https://hg.mozilla.org/integration/mozilla-inbound/rev/7be399d00f9a
https://hg.mozilla.org/mozilla-central/rev/037e6a08032f
https://hg.mozilla.org/mozilla-central/rev/7be399d00f9a
Status: REOPENED → RESOLVED
Last Resolved: 4 years ago4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla26
Duplicate of this bug: 911217
You need to log in before you can comment on or make changes to this bug.