851499 - Make psutil an optional part of the build system if errors such as "Fatal Python error: PyThreadState_Get: no current thread" can be detected

Status: REOPENED

(Firefox Build System :: General, defect)

Description

[(no longer active)]

Starting this morning, I can't use mach to build any more!

$ ./mach build
Fatal Python error: PyThreadState_Get: no current thread
Abort trap: 6
$ git log
commit f4a0805016857de4dbd7416f64d14ed35a1abab0 (HEAD, audio)
Author: Ehsan Akhgari <[e-mail]>
Date:   Thu Mar 14 11:30:47 2013 -0400

    Bug 851073 - Move MouseEvent to Web IDL; r=smaug

commit 52a683fa6e41524dba9cac7aff070e9c25075b6a (mc/inbound)
Author: Ehsan Akhgari <[e-mail]>
Date:   Thu Mar 14 21:01:02 2013 -0400

    Bug 851338 - Implement AudioContext.currentTime; r=roc

commit 7d903f9da8806b2f65ede7fbb4b91666713a3af4
Author: Trevor Saunders <[e-mail]>
Date:   Sat Jan 19 10:38:01 2013 -0500

    bug 833164 - code gen xpcom accessible events r=surkov ted

commit ed02c9777867ed3f40f6b04c36286db64afe4373
Author: Ted Mielczarek <[e-mail]>
Date:   Fri Mar 15 08:31:05 2013 -0400

    Remove some orphaned test files. no bug, DONTBUILD
    
    --HG--
    extra : rebase_source : da04a6ee5cd254c69886b10926bd702dbea0f14d

Comment 1

[(not currently active) Ted Mielczarek]

Did your system Python or anything like that change?

Comment 2

[(no longer active)]

No.

Comment 3

[(no longer active)]

$ python --version
Python 2.7.2

Comment 4

[Gregory Szorc [:gps]]

My spider senses tell me this is related to bug 844204.

Do you get any kind of stack trace at all? It literally spews those two lines when you type |mach build|?

Try the following:

hg purge --all -I 'python/psutil'
make -f client.mk configure
mach build

Comment 5

[(no longer active)]

(In reply to comment #4)
> My spider senses tell me this is related to bug 844204.
> 
> Do you get any kind of stack trace at all? It literally spews those two lines
> when you type |mach build|?

Yeah :(

> Try the following:
> 
> hg purge --all -I 'python/psutil'

hg? what's that? ;-)

$ git status --ignored python/psutil/
# On branch audio
# Ignored files:
#   (use "git add -f <file>..." to include in what will be committed)
#
#	python/psutil/_psutil_osx.so
#	python/psutil/_psutil_posix.so
#	python/psutil/build/
#	python/psutil/psutil/__init__.pyc
#	python/psutil/psutil/_common.pyc
#	python/psutil/psutil/_compat.pyc
#	python/psutil/psutil/_psosx.pyc
#	python/psutil/psutil/error.pyc
nothing to commit (working directory clean)
$ git clean -xf python/psutil/
Removing python/psutil/_psutil_osx.so
Removing python/psutil/_psutil_posix.so
Not removing python/psutil/build/
Removing python/psutil/psutil/__init__.pyc
Removing python/psutil/psutil/_common.pyc
Removing python/psutil/psutil/_compat.pyc
Removing python/psutil/psutil/_psosx.pyc
Removing python/psutil/psutil/error.pyc
$ rm -rf python/psutil/build/

> make -f client.mk configure
> mach build

This indeed fixed the problem for me!  Now, what does that tell us?

Comment 6

[Gregory Szorc [:gps]]

It tells us that the psutil binary Python extension was likely compiled against "not your current Python."

If psutil didn't write files in the source dir, this likely wouldn't be an issue since clobbers would fix it.

Bug 841678 tracks.

Comment 7

[Gregory Szorc [:gps]]

Oh fail. I just tried the STR locally and it looks like psutil wasn't rebuilt after removing the temp files. Please do the following instead:

rm -rf objdir/_virtualenv
make -f client.mk configure
mach build

*crosses fingers you don't reopen*

Comment 8

[(no longer active)]

(In reply to comment #7)
> Oh fail. I just tried the STR locally and it looks like psutil wasn't rebuilt
> after removing the temp files. Please do the following instead:
> 
> rm -rf objdir/_virtualenv
> make -f client.mk configure
> mach build
> 
> *crosses fingers you don't reopen*

What you suggested earlier _did_ fix the problem for me.  Do you still want me to do this?  (Disclaimer: I have no idea what these commands do!)

Also, how is it possible for psutil to be compiled against not my python?!  I don't have multiple python installations, to the best of my knowledge.

Comment 9

[Gregory Szorc [:gps]]

psutil is optional. When you removed the temp files earlier, it was silently failing because import failed. Please remove the virtualenv completely to force a virtualenv rebuild during configure. Then, psutil should be present again and hopefully the error won't persist.

I'm guessing the psutil binary extension was built with a slightly different Python binary or even a different toolchain. These things are fickle.

Comment 10

[(no longer active)]

OK, I did get rid of the virtualenv, and now I am getting the exact same error.

Comment 11

[Gregory Szorc [:gps]]

Damn.

I'm guessing your Python isn't compiled optimally. Is this the system Python? Homebrew?

Perhaps you should install Python 2.7.3 from Homebrew, MacPorts, etc and try a clobber?

I Googled this error and it's pretty common. Here are some of the more promising links:

https://github.com/titanous/homebrew-gnuradio/issues/7
https://github.com/mxcl/homebrew/issues/13654
http://comments.gmane.org/gmane.comp.programming.swig/18600
https://www.nsnam.org/bugzilla/show_bug.cgi?id=1317

Comment 12

[(no longer active)]

I think this is the system python.  I installed python27 from fink and with that, it seems like this problem doesn't occur anymore.  So, what gives?

Comment 13

[(no longer active)]

OK, everyone, whatever you do, DO NOT INSTALL A NON-SYSTEM PYTHON.  That will completely break things like vim, oh, and also, Mozilla builds.

:(

Comment 14

[(no longer active)]

(In reply to comment #13)
> OK, everyone, whatever you do, DO NOT INSTALL A NON-SYSTEM PYTHON.  That will
> completely break things like vim, oh, and also, Mozilla builds.

So, out of despair, I grabbed a python.org package and installed it.  And then things seemed to work again, at least things such as vim and the Mozilla build (after a clobber, of course.)  But hg stopped working.  I grabbed the official mercurial package, the trick.  I think my system is now screwed beyond repair, so I may not be able to provide much useful information in this bug any more :(  Sorry.

But we need to find a solution to this bug if (when?) other people hit it.  This was a regression in Mach since I have been using mach all along and nothing else has changed in my config as far as I can tell, but after my panic tonight I won't be able to suggest installing a non-system python to anyone with a clear conscience.  :/

Comment 15

[Gregory Szorc [:gps]]

I use Homebrew's Python and all works fine. I've had issues getting the system Python to play nice with some aspects of Python. Homebrew's "just works." Maybe Fink isn't as graceful as Homebrew. Who knows.

I'll leave this bug open for other people to stumble across. But, you are so far the only one. So, I'm inclined to believe your machine was special in some way.

Comment 16

[(no longer active)]

Actually, forget what I said last night.  My problem has not been fixed at all.  I guess no mach for me, any more. :(

Comment 17

[(no longer active)]

Backing out bug 844204 locally fixes this for me, of course.  Any chance we can add a way to disable what that code does through some env var or mozconfig setting?

Comment 18

[(no longer active)]

OK, looks like we build psutil as part of the build process ourselves.  In this case, we need to detect this error at configure time and opt out of building psutil and depending on it.

Comment 19

[Gregory Szorc [:gps]]

This is going to be difficult since building and installing psutil is part of virtualenv population. There will be unholy hackitude going on somewhere. Maybe we should execute the psutil tests as part of virtualenv population?

Comment 20

[Jeff Hammel]

I don't really have time to look at this issue in depth (and no squat about Mac), but I would guess it not *too* hard to add a way to use psutil that is already built (whether it works or not...hard to say).  That said, I do believe this should be fixed "correctly" which I'm not even sure what a good fix is for it right now.