129216 - Integrate Python xpcom bindings with build system

Status: RESOLVED FIXED

(SeaMonkey :: Build Config, enhancement)

Description

[Mark Hammond [:markh] [:mhammond]]

I want to allow building of the python/xpcom directory a standard Moz 
extension, so it can be built without any source-file changes - just configure 
and go!

Intend having Linux and win32 gmake builds using the standard ./configure 
procedure, while the win32 nmake build is likely to look for a special 
environment variable to trigger the build.

Comment 1

[Mark Hammond [:markh] [:mhammond]]

Attachment: Initial patch that adds support for ./configure to allow us to build.

Comment 2

[hacker formerly known as [e-mail]]

Comment on attachment 72748 [details] [diff] [review]
Initial patch that adds support for ./configure to allow us to build.

Since pyxpcom is an extensions, I'd prefer to use the standard extensions
mechanism for it so that --enable-extensions=default,pyxpcom works.  If we need
to add the PYTHON_SRC check ontop  of as well, we can do that by following the
gtk/xmlterm example.

Comment 3

[Mark Hammond [:markh] [:mhammond]]

Attachment: New patch that uses autoconf the way it was designed

Here is a new patch.  It simply wants "--enable-extensions=extensions/pyxpcom"
and it will attempt to locate Python and configure everything automatically -
no need for PYTHON_SRC etc, it automatically locates everything!

I have never written an autoconf script before, so comments appreciated.

Comment 4

[Ken A. Redergård]

The build system seems to have some problems interpolating the MOZ_PYTHON_*
variables. I'm running MDK8.2 (autoconf-2.13).

If you look in extensions/python/xpcom/source:

Makefile:
DEPTH=../../../..
topsrcdir       = ../../../..
srcdir          = .
PYTHON_INCLUDES = @MOZ_PYTHON_INCLUDES@
PYTHON_LIBS = @MOZ_PYTHON_LIBS@
PYTHON_DEBUG_SUFFIX = @MOZ_PYTHON_DEBUG_SUFFIX@
PYTHON_DLL_SUFFIX = @MOZ_PYTHON_DLL_SUFFIX@

Makefile.in:
DEPTH=../../../..
topsrcdir       = @top_srcdir@
srcdir          = @srcdir@
VPATH           = @srcdir@
PYTHON_INCLUDES = @MOZ_PYTHON_INCLUDES@
PYTHON_LIBS = @MOZ_PYTHON_LIBS@
PYTHON_DEBUG_SUFFIX = @MOZ_PYTHON_DEBUG_SUFFIX@
PYTHON_DLL_SUFFIX = @MOZ_PYTHON_DLL_SUFFIX@

autoconf obviously is beeing ran on the Makefile.in, but not for the
@MOZ_PYTHON_* variables.

Comment 5

[Mark Hammond [:markh] [:mhammond]]

Re problems with MOZ_PYTHON_ variables.

Is it possible that you simply need to run "autoconf" after patching
configure.in?  Maybe the new code isn't in "./configure"?

[My understanding from #mozilla is that I only need to provide a patch for
./configure.in and magically a new ./configure will end up in CVS.  Please
correct me if I am wrong.]

Comment 6

[Ken A. Redergård]

I've ran autoconf, but it did not work. What I had to do was to delete the
makefiles found in e*/python/xpcom/src and e*/python/xpcom/src/loader/, then the
build system would insert the correct values for MOZ_PYTHON_*. I'm not sure why
it does this, but now it works. Looking at the timestamp on the files I see that
the Makefile.in files are older than the Makefile files. Perhaps a touch on them
would do the same trick.

You are correct regarding the autoconf magic.

Comment 7

[Mark Hammond [:markh] [:mhammond]]

Attachment: Fixes a build problem by never using backslashes

I had a strange build problem that has gone away by using paths of the format
"C:/foo" rather than "C:\foo" - someone, somewhere was obviously escaping.

Comment 8

[Shane Caraveo]

Attachment: diff to build against 1.2 release

On windows, the autoconf stuff would destroy the makefiles on second pass if
the python variables were defined in the makefiles.  This moves those variables
into autoconf.mk.in and allows for a successful build on win32.  The
test_misc.py passes everything, but have not done futher testing beyond that.

Comment 9

[Shane Caraveo]

By "Destroy" I mean that AC_OUTPUT in configure used /cygwin/c/ paths for srcdir
and top_srcdir.  Since neither VS or ActivePerl understands those paths, build
failed.

Comment 10

[hacker formerly known as [e-mail]]

Comment on attachment 107653 [details] [diff] [review]
diff to build against 1.2 release

In the configure tests, isn't MOZ_PYTHON_SRC always going to be empty?	So why
bother with the test -z ?

Comment 11

[Shane Caraveo]

Ahh, yes.  I based my patch off the earlier patches, and did not catch that.  I
can create a new patch soon.

Comment 12

[Remy C. Cool]

When patching the 1.2.1 source distro with the latest patch from this bug report, the detection of the 'python directories' doesn't work on my debian sarge system.  It uses the MOZ_PYTHON_SRC (/usr/bin) as the base for the include and lib dirs, ie: $MOZ_PYTHON_SRC/lib ... however the directory is located at /usr/lib and not /usr/bin/lib. When I hardcode the location of the lib and include directories in the configure.in file, gmake -f client.mk build runs until ErrorUtils.cpp where it exits with the following error:  gmake[5]: Entering directory `/home/remy/development/mozilla/extensions/python/xpcom/src' ErrorUtils.cpp c++ -o ErrorUtils.o -c -DOSTYPE=\"Linux2.4\" -DOSARCH=\"Linux\" -DOJI -I/usr/include/python2.1 -I../../../../dist/include/xpcom -I../../../../dist/include/string -I../../../../dist/include/pyxpcom -I../../../../dist/include -I/home/remy/development/mozilla/dist/include/nspr      -I/usr/X11R6/include   -fPIC  -I/usr/X11R6/include -fno-rtti -fno-exceptions -Wall -Wconversion -Wpointer-arith -Wcast-align -Woverloaded-virtual -Wsynth -Wno-ctor-dtor-privacy -pedantic -Wno-long-long -pthread -pipe  -DNDEBUG -DTRIMMED  -I/usr/X11R6/include -DMOZILLA_CLIENT -include ../../../../mozilla-config.h -Wp,-MD,.deps/ErrorUtils.pp ErrorUtils.cpp ErrorUtils.cpp: In function `void LogMessage(const char *, const char *)': ErrorUtils.cpp:40: `nsOutputConsoleStream' undeclared (first use this function) ErrorUtils.cpp:40: (Each undeclared identifier is reported only once ErrorUtils.cpp:40: for each function it appears in.) ErrorUtils.cpp:40: parse error before `;' ErrorUtils.cpp:41: `console' undeclared (first use this function) ErrorUtils.cpp: In function `void PyXPCOM_LogError(const char *, ...)': ErrorUtils.cpp:65: `nsOutputStringStream' undeclared (first use this function) ErrorUtils.cpp:65: parse error before `(' ErrorUtils.cpp:70: `streamout' undeclared (first use this function)  Could it be that this error has something to do with a missing or incorrect path?        

Comment 13

[Remy C. Cool]

Building XPCOM/pyXPCOM from CVS works if the /usr/lib /usr/include is 
hardcoded. 

Comment 14

[Remy C. Cool]

Attachment: Modded patch (diff for 1.2) to get the include and lib dirs right for linux systems (also works for 1.3a)

Comment 15

[Mark Hammond [:markh] [:mhammond]]

Attachment: New for 1.4, and updated

This version should work much better, and integrates other patches as well as I
can.  For both Windows and Linux it should build from either a "pre-installed"
Python, or a built source-tree.  For both platforms, setting PYTHON_SRC before
configuring should configure for the Python in that directory, otherwise your
path will be searched and that Python used. (You may need to delete
config.cache after changing this var)

Tested fully on Windows.  On my RH8 system, it configures and builds
out-of-the-box using the RH supplied Python.  pyloader link fails however - I
have to give up now, but help for Linux would be lovely!

Comment 16

[Mark Hammond [:markh] [:mhammond]]

Attachment: New patch

This patch is a lot simpler.  This patch no longer uses PYTHON_SRC, but instead
borrows conventions from other Python based autoconf tools.

You may specify $PYTHON, which must be the Python executable (or your PATH is
searched for any python).  This Python's sys.prefix is used as the base
directory.  Thus, a source directory is no longer supported on Linux - you must
do a "make install" on your Python tree (or use a pre-built/pre-installed one).
 The good news is that this actually builds and works on Linux :)  If you use
Python 2.3, configure with "--enable-shared", do a "make install", set
LD_LIBRARY_PATH and PYTHON_PATH appropriately, you can convince it to work!

Comment 17

[Thomas Schilz]

Attachment: New patch for building with Mozilla 1.5

This is a patch I used to build PyXPCOM with Mozilla 1.5. The previous patch
didn't work for me. The only differences between this patch and the previous
one are in the line numbers.

Don't forget to run autoconf before configuring.

Under Linux I had to prevent binding with the static Python library by
temporarily renaming that library. If bound with the static library I got a
"Fatal Python error: Interpreter not initialized (version mismatch?)".

Under Windows I had to manually add PYTHON_INCLUDES and PYTHON_LIBS to
src/Makefile and src/loader/Makefile.

Comment 18

[Mark Hammond [:markh] [:mhammond]]

Attachment: Fresh patch off the trunk

Comment 19

[Matthew Sherborne]

I have been working on getting pyxpcom going on linux with firefox >= 1.0.3.

I know others want to get it working too. We should get together somehow.

I've made freenode channel #pyxpcom for collaboration purposes.

The Mozilla 1.7 branch compiles pyxpcom ok but when running against firefox has
a few segfaults (maybe just the way I'm using it).

Any discussion/feedback appreciated.

Comment 20

[Matthew Sherborne]

Attachment: Patch off 1.0.4 branch

Don't know if this obsoletes the trunk patch...

Comment 21

[Shane Caraveo]

Attachment: patch to build pyxpcom in static moz builds

this patch allows pyxpcom to be built when building a static firefox build. 
Tested with 1.0.4 on osx

Comment 22

[Shane Caraveo]

Attachment: pyxpcom build patch for osx

this is the patch we use at ActiveState to build pyxpcom on win/linux and os x.
 If  you are doing static builds, get the previous patch I added also.

Comment 23

[Christian Persch (GNOME) (away; not receiving bug mail)]

Attachment: updated configure + build patch

This patch:
- adds the configure checks from attachment 153029 [details] [diff] [review]
- adds necessary MOZ_INTERNAL_API=1 to Makefile.in:s so you can actually build
on trunk (this part is just necessary, not sufficient to build on trunk)
- fixes the Makefile.in:s install targets to work with objdir != srcdir builds,

- adds install targets to actually install the python package

Comment 24

[Christian Persch (GNOME) (away; not receiving bug mail)]

Comment on attachment 186149 [details] [diff] [review]
updated configure + build patch

Asking for review.

Comment 25

[Benjamin Smedberg]

Comment on attachment 186149 [details] [diff] [review]
updated configure + build patch

r- for only one thing: please use "cygpath -t mixed" instead of "cygpath -m"
because some older versions of cygpath don't understand -m.

But, I would really like to avoid the *.py rule in
extensions/python/xpcom/Makefile.in; could the list of .py files be coded in
variables such as

PYSRCS_XPCOM
PYSRCS_XPCOMCLIENT
PYSRCS_XPCOMSERVER
PYSRCS_XPCOMTOOLS

And then use these variables instead of *.py?

Comment 26

[Christian Persch (GNOME) (away; not receiving bug mail)]

Attachment: updated patch with reviewer's comments addressed

I changed the cygpath argument as requested, and changed the *.py rules to use
variables instead. The patch also includes one file I forgot to diff in the old
patch, config/system-headers, where I added Python.h to the list of wrapped
system headers.

Comment 27

[Christian Persch (GNOME) (away; not receiving bug mail)]

Attachment: interdiff of attachment 186149 and attachment 186523, to ease the review

Comment 28

[Christian :Biesinger (don't email me, ping me on IRC)]

attachment 186523 [details] [diff] [review] checked in.

Checking in allmakefiles.sh;
/cvsroot/mozilla/allmakefiles.sh,v  <--  allmakefiles.sh
new revision: 1.566; previous revision: 1.565
done
Checking in configure.in;
/cvsroot/mozilla/configure.in,v  <--  configure.in
new revision: 1.1469; previous revision: 1.1468
done
Checking in config/autoconf.mk.in;
/cvsroot/mozilla/config/autoconf.mk.in,v  <--  autoconf.mk.in
new revision: 3.349; previous revision: 3.348
done
Checking in config/system-headers;
/cvsroot/mozilla/config/system-headers,v  <--  system-headers
new revision: 3.4; previous revision: 3.3
done
Checking in extensions/python/xpcom/Makefile.in;
/cvsroot/mozilla/extensions/python/xpcom/Makefile.in,v  <--  Makefile.in
new revision: 1.2; previous revision: 1.1
done
Checking in extensions/python/xpcom/src/Makefile.in;
/cvsroot/mozilla/extensions/python/xpcom/src/Makefile.in,v  <--  Makefile.in
new revision: 1.5; previous revision: 1.4
done
Checking in extensions/python/xpcom/src/loader/Makefile.in;
/cvsroot/mozilla/extensions/python/xpcom/src/loader/Makefile.in,v  <--  Makefile.in
new revision: 1.7; previous revision: 1.6
done
Checking in extensions/python/xpcom/test/test_component/Makefile.in;
/cvsroot/mozilla/extensions/python/xpcom/test/test_component/Makefile.in,v  <--
 Makefile.in
new revision: 1.2; previous revision: 1.1
done

Comment 29

[Luke Matkins]

Attachment: fix to remove references to ns_getglobalcomponentmanager

I still had problems compiling the current version of pyxpcom in CVS. It
couldn't find NS_GetGlobalComponentManager (defined in
nsComponentManagerObsolete.h, which was apparently removed from CVS?).

I ended up taking some of the changes from the second patch listed on
http://live.gnome.org/Epiphany_2fEphyPython_2fPyXPCOM (just the
componentmanager changes, not the unicode changes), and can now successfully
compile pyxpcom. I'm attaching a patch of the changes I applied.

Comment 30

[Christopher Aillon (sabbatical, not receiving bugmail)]

(In reply to comment #29)
> Created an attachment (id=186767) [edit]
> fix to remove references to ns_getglobalcomponentmanager

Note, that issue is addressed by https://bugzilla.mozilla.org/show_bug.cgi?id=297598

I think this bug can be resolved.

Comment 31

[Shane Caraveo]

I dont think this is ready to be resolved until I/someone can integrate the
various build patches.  The latest build patch allows linking against a static
lib on linux, which is a failure case, and the osx build patch I posted needs to
be integrated with it.

Comment 32

[Mark Hammond [:markh] [:mhammond]]

Comment on attachment 186767 [details] [diff] [review]
fix to remove references to ns_getglobalcomponentmanager

Should this patch should be rolled up with bug 297598?	Maybe to keep things
cleaner, this bug should deal only with the Mozilla build system, not pyxpcom
itself.

Comment 33

[Mark Hammond [:markh] [:mhammond]]

Attachment: updated pyxpcom build patch for non-windows

The original attachment 186139 [details] [diff] [review] duplicated many of the patches already checked
into CVS.  Best I could tell, there are only 2 patches that are not duplicated:

* Detection of lib and header files on non-windows.
* The EXTRA_DSO_OPTS patch in config/rules.mk

The EXTRA_DSO_OPTS patch appears to be unnecessary.  As the pyxpcom makefiles
set IS_COMPONENT, that platform should already get a "-bundle" option.

So attached is a new patch that only changes the non-windows branch in
configure.in.  Note I have *NOT* tested this as my Linux dev environment hasn't
caught up with pyxpcom yet.  Requesting review from Shane after testing on the
various platforms he builds with.

Comment 34

[Shane Caraveo]

at oscon now, can try out the changes in a couple weeks.  The rules.mk part of
my patch prevented another part of the IS_COMPONENT build options from being
used.  I dont remember the specifics of why I needed that, I'll look into that also.

Comment 35

[Shane Caraveo]

Attachment: marks last patch fixed for osx and linux

The rules.mk change is necessary to get the pyxpcom shared libs to load
correctly.  This might be able to be handled in the pyxpcom makefiles, but I
already had the rules.mk patch handy.  This is against the 1.8 branch.

Comment 36

[Benjamin Smedberg]

Comment on attachment 193733 [details] [diff] [review]
marks last patch fixed for osx and linux

This is the sort of this we really don't want to be hardcoding into rules.mk.
Please add -bundle to the pyxpcom makefiles as necessary, if these are really
bundles but are not components (which sounds odd to me).

Comment 37

[Shane Caraveo]

Attachment: change to handle osx ld opts from pyxpcom makefile

Ok, this patch gets rid of using rules.mk, and patches the makefile in pyxpcom
to build the python module correctly.

Comment 38

[Shane Caraveo]

(In reply to comment #36)
> (From update of attachment 193733 [details] [diff] [review] [edit])
> This is the sort of this we really don't want to be hardcoding into rules.mk.
> Please add -bundle to the pyxpcom makefiles as necessary, if these are really
> bundles but are not components (which sounds odd to me).
> 

_xpcom.dylib is a python extension, not a mozilla component.  libpyloader.dylib
is the mozilla component that loads python.  IS_COMPONENT adds something else
into the mix which makes python unable to import _xpcom.dylib.  Only the
makefile for _xpcom.dylib needs the work around.

Comment 39

[Shane Caraveo]

Attachment: patch against trunk to build on osx again

This patch fixes building trunk cvs on os x.

Comment 40

[Mark Hammond [:markh] [:mhammond]]

That osx patch is now applied.  As far as I know, no outstanding patches remain.  I will leave this bug open though to track even better build integration after discussing it with the moz guys etc (eg, should configure automatically check for a usable Python and automatically enable things?)

Comment 41

[Marco Pesenti Gritti]

Comment on attachment 191157 [details] [diff] [review]
updated pyxpcom build patch for non-windows

I had a quick try to this on linux, 1_8 branch. It doesnt seem to work. Where is MOZ_PYTHON_SRC defined? Or should that actually be MOZ_PYTHON_PREFIX?

Comment 42

[Marco Pesenti Gritti]

Comment on attachment 191157 [details] [diff] [review]
updated pyxpcom build patch for non-windows

Oh nm, this was fixed and checked in as part of attachment 193960 [details] [diff] [review]

Comment 43

[Jose Espinosa]

(In reply to comment #40)
> That osx patch is now applied.  As far as I know, no outstanding patches
> remain.  I will leave this bug open though to track even better build
> integration after discussing it with the moz guys etc (eg, should configure
> automatically check for a usable Python and automatically enable things?)
> 

The file mozilla/extensions/python/xpcom/src/Makefile.in does not have the patch any more.  I think the problem is that the file mozilla/extensions/python/xpcom/src/module/Makefile.in was removed, and with it the patch.  I am using the branch 1_8

Comment 44

[Mark Hammond [:markh] [:mhammond]]

Attachment: patch allowing --enable-extensions=python

Enhance python/Mozilla building.  From the change to configure.in:

dnl Two ways to enable Python support:
dnl   --enable-extensions=python # select all available.
dnl    (MOZ_PYTHON_EXTENSIONS contains the list of extensions)
dnl or:
dnl   --enable-extensions=python/xpcom,... # select individual ones

This means that the builder can  --enable-extensions=python, and automatically all Python extensions will be built.  Although the trunk currently only has 1 extension (xpcom), the DOM_AGNOSTIC work is introducing a new one.  Once that branch lands, it will be a one-line change to have it included in the default Python extensions.

It introduces a new variable MOZ_PYTHON_EXTENSIONS, and a new extensions/python/Makefile.in.

This patch also renames PYTHON_* to MOZ_PYTHON_* in all makefiles, consistent with all other MOZ build variables.

Comment 45

[Brendan Eich [:brendan]]

Comment on attachment 211961 [details] [diff] [review]
patch allowing --enable-extensions=python

Rubberstamping -- main thing is that it works for you and it doesn't turn any tinderboxes red.

/be

Comment 46

[Mark Hammond [:markh] [:mhammond]]

All checked in - finally retiring this bug!  Thanks all