Last Comment Bug 908296 - psutil breaks in Debian wheezy
: psutil breaks in Debian wheezy
Status: RESOLVED FIXED
[Run `hg status -in python/psutil | x...
:
Product: Core
Classification: Components
Component: Build Config (show other bugs)
: Trunk
: x86_64 Linux
-- normal (vote)
: mozilla26
Assigned To: Gregory Szorc [:gps]
:
: Gregory Szorc [:gps]
Mentors:
: 911217 (view as bug list)
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2013-08-22 10:43 PDT by micahflee
Modified: 2013-08-30 08:53 PDT (History)
7 users (show)
ryanvm: in‑testsuite+
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
Upgrade psutil to version 1.0.1 (286.86 KB, patch)
2013-08-22 16:20 PDT, Gregory Szorc [:gps]
mh+mozilla: review+
Details | Diff | Splinter Review

Description User image micahflee 2013-08-22 10:43:17 PDT
User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130806 Firefox/17.0 Iceweasel/17.0.8 (Nightly/Aurora)
Build ID: 20130806234539

Steps to reproduce:

I'm running an up-to-date Debian wheezy, but I can't build mozilla-central due to psutil fatal errors. You should be able to reproduce by cloning the repo in wheezy, running:

./mach bootstrap

Then:

./mach build


Actual results:

micah@spock:~/projects/mozilla-central$ ./mach build
/home/micah/eff_projects/httpse/mozilla-central-httpse-tests/python/psutil/psutil/_pslinux.py:102: RuntimeWarning: couldn't determine platform's TOTAL_PHYMEM
  warnings.warn("couldn't determine platform's TOTAL_PHYMEM", RuntimeWarning)
Error running mach:

    ['build']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

AttributeError: 'module' object has no attribute 'get_sysinfo'

  File "/home/micah/eff_projects/httpse/mozilla-central-httpse-tests/python/mozbuild/mozbuild/mach_commands.py", line 299, in build
    monitor.init(warnings_path)
  File "/home/micah/eff_projects/httpse/mozilla-central-httpse-tests/python/mozbuild/mozbuild/controller/building.py", line 277, in init
    self.resources = SystemResourceMonitor(poll_interval=1.0)
  File "/home/micah/eff_projects/httpse/mozilla-central-httpse-tests/testing/mozbase/mozsystemmonitor/mozsystemmonitor/resourcemonitor.py", line 180, in __init__
    virt = psutil.virtual_memory()
  File "/home/micah/eff_projects/httpse/mozilla-central-httpse-tests/python/psutil/psutil/__init__.py", line 1154, in virtual_memory
    return _psplatform.virtual_memory()
  File "/home/micah/eff_projects/httpse/mozilla-central-httpse-tests/python/psutil/psutil/_pslinux.py", line 137, in virtual_memory
    total, free, buffers, shared, _, _ = _psutil_linux.get_sysinfo()


I think this is a bug with the version of psutil that's included with mozilla-central. The version I have installed in Debian seems to work to include:

micah@spock:~/projects/mozilla-central$ python
Python 2.7.3 (default, Jan  2 2013, 13:56:14) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
>>> 


But when I load the mozilla-central version I get an error:

micah@spock:~/projects/mozilla-central/python/psutil$ ls
CREDITS  examples  HISTORY  LICENSE  MANIFEST.in  PKG-INFO  psutil  README  setup.py  test
micah@spock:~/projects/mozilla-central/python/psutil$ python
Python 2.7.3 (default, Jan  2 2013, 13:56:14) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import psutil
psutil/_pslinux.py:102: RuntimeWarning: couldn't determine platform's TOTAL_PHYMEM
  warnings.warn("couldn't determine platform's TOTAL_PHYMEM", RuntimeWarning)
>>> 



Expected results:

"./mach build" should have built mozilla-central.
Comment 1 User image Gregory Szorc [:gps] 2013-08-22 16:13:44 PDT
Perhaps we should upgrade the psutil in m-c. Will submit a patch to try momentarily.
Comment 2 User image Gregory Szorc [:gps] 2013-08-22 16:20:08 PDT
Created attachment 794322 [details] [diff] [review]
Upgrade psutil to version 1.0.1

Archive obtained from
https://psutil.googlecode.com/files/psutil-1.0.1.tar.gz and extracted
over existing source code without modifications.

https://tbpl.mozilla.org/?tree=Try&rev=b242a872ce53
Comment 3 User image Mike Hommey [:glandium] 2013-08-22 16:38:54 PDT
Comment on attachment 794322 [details] [diff] [review]
Upgrade psutil to version 1.0.1

Review of attachment 794322 [details] [diff] [review]:
-----------------------------------------------------------------

rs=me
Comment 5 User image Gregory Szorc [:gps] 2013-08-23 00:43:40 PDT
Despite the conversions on IRC, I forgot to touch a virtualenv file when I landed this, so it caused some minor bustage on inbound. Pushed a trivial followup touching populate_virtualenv.py in a non-significant way.

https://hg.mozilla.org/integration/mozilla-inbound/rev/1ad8a6674785
Comment 7 User image Ryan VanderMeulen [:RyanVM] 2013-08-23 09:18:40 PDT
Backed out at the request of #developers for causing OSX mach bustage.
https://hg.mozilla.org/integration/mozilla-inbound/rev/31aa3eb1a40d
Comment 8 User image Adam Roach [:abr] 2013-08-23 09:22:36 PDT
The update to psutil 1.0.1 broke mach under OS X (confirmed independently by two developers under both OSX 10.8 and OSX 10.9). The specific issue (which seems to be the same problem fixed by 1ad8a6674785) is the absence of a TCPS_ESTABLISHED attribute on the _psutil_osx module:

> $ ./mach build
> Error running mach:
> 
>     ['build']
> 
> The error occurred in code that was called by the mach command. This is either
> a bug in the called code itself or in the way that mach is calling it.
> 
> You should consider filing a bug for this issue.
> 
> If filing a bug, please include the full output of mach, including this error
> message.
> 
> The details of the failure are as follows:
> 
> AttributeError: 'module' object has no attribute 'TCPS_ESTABLISHED'
> 
>   File "/Users/Adam/devel/mozilla/mozilla-inbound/python/mozbuild/mozbuild/mach_commands.py", line 293, in build
>     from mozbuild.controller.building import BuildMonitor
>   File "/Users/Adam/devel/mozilla/mozilla-inbound/python/mozbuild/mozbuild/controller/building.py", line 22, in <module>
>     import psutil
>   File "/Users/Adam/devel/mozilla/mozilla-inbound/python/psutil/psutil/__init__.py", line 95, in <module>
>     import psutil._psosx as _psplatform
>   File "/Users/Adam/devel/mozilla/mozilla-inbound/python/psutil/psutil/_psosx.py", line 48, in <module>
>     _TCP_STATES_TABLE = {_psutil_osx.TCPS_ESTABLISHED : CONN_ESTABLISHED,
Comment 9 User image Peter Van der Beken [:peterv] 2013-08-23 09:39:43 PDT
Removing

python/psutil/build
python/psutil/_psutil_osx.so
python/psutil/_psutil_posix.so

fixed my OS X build.
Comment 10 User image Adam Roach [:abr] 2013-08-23 09:45:29 PDT
Wow. I'm surprised we have binary build output showing up in the tree in locations other than obj-*. Would it be possible to move these outputs into the same directory as the rest of the binary build objects so that clobbers clean them up?
Comment 11 User image Gregory Szorc [:gps] 2013-08-23 09:52:01 PDT
This is a dependency problem. The virtualenv needs to be repopulated before psutil can be imported or the aforementioned TCPS_ESTABLISHED error will occur. By backing this out, you've likely made the problem worse, as now developers who've upgraded will get errors due to newer psutil being used on import. The correct solution is to tell people to run `hg status -in python/psutil | xargs rm`.

(In reply to Adam Roach [:abr] from comment #10)
> Wow. I'm surprised we have binary build output showing up in the tree in
> locations other than obj-*. Would it be possible to move these outputs into
> the same directory as the rest of the binary build objects so that clobbers
> clean them up?

Known issue. It really sucks. It will take some knowledge of Python packaging to fix. It needs to be fixed. Fortunately, we've only upgraded psutil twice in over a year, so as annoying as it is, it's not frequent enough to highly prioritize, IMO.
Comment 12 User image Adam Roach [:abr] 2013-08-23 09:59:19 PDT
(In reply to Gregory Szorc [:gps] from comment #11)
> The correct solution is to tell people to run `hg status -in
> python/psutil | xargs rm`.
 
This kind of "hallway lore" is very brittle -- most people who need it won't have access to it when they need it. Can we be careful to at least include this information in the bug the next time we update the package? Or, even better, put a try/catch around the package inclusion that takes care of running this command for developers?
Comment 13 User image Gregory Szorc [:gps] 2013-08-23 10:20:39 PDT
Relanded because the backout will cause bustage as well.

https://hg.mozilla.org/integration/mozilla-inbound/rev/037e6a08032f
Comment 14 User image Peter Van der Beken [:peterv] 2013-08-23 10:23:13 PDT
FWIW, I don't think it's acceptable to break most developers trees on OS X and let them all figure out by themselves how to unbreak them.
Comment 15 User image Myk Melez [:myk] [@mykmelez] 2013-08-23 10:38:55 PDT
(In reply to Gregory Szorc [:gps] from comment #11)
> The correct solution is to tell people to run `hg status -in python/psutil | xargs rm`.

For those using a clone of the Git mirror, an equivalent is:

git clean -xf python/psutil/

# -x to include ignored files; -f to force the clean (replace with -n to see what would be cleaned first)
Comment 16 User image Guillaume Abadie 2013-08-23 10:48:36 PDT
(In reply to Peter Van der Beken [:peterv] from comment #14)
> FWIW, I don't think it's acceptable to break most developers trees on OS X
> and let them all figure out by themselves how to unbreak them.
I definitely agree!
We lived without it before, and now that will be painful while bisecting on OS X after that patch. What about landing a patch that would fix all bustages instead?
Comment 17 User image Gregory Szorc [:gps] 2013-08-23 10:53:38 PDT
Applied some wallpaper to catch all Exceptions when importing psutil. This should limit the impact of this patch. People building a changeset with new psutil but not this changeset may still run into issues, however. Perhaps we should cherry-pick it to central before the inbound merge?

https://hg.mozilla.org/integration/mozilla-inbound/rev/7be399d00f9a
Comment 19 User image Honza Bambas (:mayhemer) 2013-08-30 08:53:45 PDT
*** Bug 911217 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.