Open Bug 648701 Opened 14 years ago Updated 3 years ago

Make the build steps familiar to new contributors

Categories

(Firefox Build System :: General, defect)

Product:
Firefox Build System
Component:
General
Platform:
x86
All
Type:
defect

Tracking

(Not tracked)

People

(Reporter: paul.biggar, Unassigned)

References

(Depends on 1 open bug, Blocks 1 open bug)

Details

Attachments

(1 file, 11 obsolete files)

Setup default OBJDIR , potentially default .mozconfig from arguments & make -f client.mk
2.10 KB, text/plain
ted
: review-
Details
From the docs, I believe the build steps are: make -f client.mk build We've removed the need to have a mozconfig file (yay!), and so newbies are not-too-far from just having to run |make|. I tried omitting "build" and it seems to still work. Is is supposed to work? Can we make it so that we can omit |-f client.mk|? The simple first attempt of copying client.mk to Makefile didn't work, as Makefile is generated from Makefile.in. On the other hand, copying to GNUmakefile worked, though renaming didn't. Do we already require GNU make? If so, there may be a tiny GNUmakefile we can write which redirects to |make -f client.mk|. Or maybe there's a better solution.
(In reply to comment #0) > From the docs, I believe the build steps are: > > make -f client.mk build > > We've removed the need to have a mozconfig file (yay!), and so newbies are > not-too-far from just having to run |make|. > > I tried omitting "build" and it seems to still work. Is is supposed to work? Yes. > Can we make it so that we can omit |-f client.mk|? The simple first attempt of > copying client.mk to Makefile didn't work, as Makefile is generated from > Makefile.in. Not easily. client.mk runs autoconf and configure. > On the other hand, copying to GNUmakefile worked, though renaming didn't. Do we > already require GNU make? If so, there may be a tiny GNUmakefile we can write > which redirects to |make -f client.mk|. Or maybe there's a better solution. So the problem is that we already have a Makefile.in in the root directory of the source tree, and adding a Makefile in the root too might melt builds where the objdir and the srcdir are the same.
In theory we can hg mov |Makefile.in| to |main.mk| or some such, and create a |Makefile| shim that translates to client.mk if no target, or |main.mk| if a target and adjust other code as needed. (I think my above will be the shortest path to victory for most of our tooling, dependant projects/repos and users; but it has not been tested and I'm not sure how well it will work)
(In reply to comment #1) > (In reply to comment #0) > > On the other hand, copying to GNUmakefile worked, though renaming didn't. Do we > > already require GNU make? If so, there may be a tiny GNUmakefile we can write > > which redirects to |make -f client.mk|. Or maybe there's a better solution. > > So the problem is that we already have a Makefile.in in the root directory of > the source tree, and adding a Makefile in the root too might melt builds where > the objdir and the srcdir are the same. Yes, that appears to be the problem. FWIW, renaming client.mk to GNUmakefile, and doing s/\bclient\.mk\b/GNUmakefile/ gives me a successful build by just typing |make| (which includes running autoconf and automake).
You know, I'm slightly rethinking this. The goal is to make a build familiar to new contributor. The most familiar thing is: ./configure && make or mkdir objdir && cd objdir && ../configure && make By which I mean, this is more familiar than a system that just uses |make|. So perhaps we should just check in configure?
Summary: Change the build steps to just "make" → Make the build steps familiar to new contributors
hm, my homemade build script (see below) has three main steps (checkout, build, package): IIUC this bug touches just the middle one? Here it is; line 2 has to be customized for your objdir: #!/bin/bash export AJM_OBJDIR=obj-x86_64-unknown-linux-gnu date && \ echo 'python client.py checkout' && \ python client.py checkout && \ date && \ echo 'make -f client.mk build' && \ make -f client.mk build && \ test -n "$AJM_OBJDIR" -a -d $AJM_OBJDIR && \ date && \ echo "make -C $AJM_OBJDIR package" && \ make -C $AJM_OBJDIR package echo 'Exit status' $?
(In reply to comment #4) > You know, I'm slightly rethinking this. The goal is to make a build familiar to > new contributor. The most familiar thing is: > > ./configure && make > or > mkdir objdir && cd objdir && ../configure && make > > By which I mean, this is more familiar than a system that just uses |make|. So > perhaps we should just check in configure? I don't think expecting a user to run autoconf is much more of a burden beyond that. The only tricky bit is that we're still on autoconf 2.13. Today autoconf213 && ./configure && make should work.
Also, back in the ancient days we used to check in configure. Somebody who was around then would know why we don't do that anymore.
So, back then we had a script that watched for checkins to configure.in and checked in the matching configure. We didn't have that when we moved to hg, so we didn't import configure. I think the only issue is people forgetting to check in the matching configure.
It occurs to me that we could easily write a Hg hook to ensure that both configure and configure.in are changed if either one is changed. If we wanted to be super-paranoid, we could have it run autoconf-2.13 on the configure.in and make sure it matches the configure in the patch.
Also, our source tarballs for releases include configure, and there `./configure && make` should work just fine.
Having a hard time composing all my thoughts this morning, but I'd rather not encourage "configure && make" in hg checkouts if we can avoid it, since that creates a srcdir build and pollutes your src tree with a bunch of crap, which is probably not going to be helpful for new contributors.
Yeah, that's the problem here. client.mk is better than running autoconf/configure/make by hand, just less discoverable.
Checking in configure is bug 464053, with the amusing result that if you develop on Windows or Mac, your generated configure will be just fine, but if you develop on the most common Linux distro, you'd be checking in a freaky crap configure. Dunno whether or not Ubuntu still has a hacky autoconf2.13, but somebody could check that, and write the hooks to check for a configure in any configure.in push and to check for #include <sys/types.h> in configure pushes, and land it. Couldn't we solve the srcdir build problem by just copying the bits in client.mk that decide if you don't specify an objdir, then you'll get MOZ_OBJDIR = obj-$(CONFIG_GUESS), into /Makefile.in? We've said for years that we want to disallow srcdir builds: here's our chance!
Once you've run configure it's too late, you have a srcdir build. Although, I wonder if we could add hacks in configure to check if it's being run from $topsrcdir, and do a mkdir, cd, and recursively invoke itself in that directory. I have no idea if that would work, but it might be interesting to try.
Whiteboard: [contrib-engagement]
Have we got consensus on how we'd like it to behave (ignoring the technical details of how we'd make that work). Do we want: 1. $ make or 2. $ ./configure && make (let's assume this makes an objdir) or 3. $ mkdir objdir && cd objdir && ../configure && make Or do we want to support all 3 of the above?
I think #2 is familiar enough, since it's how 99% of open source projects are built. If we can make it create an objdir, then it ought to be basically perfect. (We should not break #3, since that's how client.mk builds work under the hood right now anyway.)
Many people currently do #3, so we should keep that working. Making #2 use an objdir would be good. It could just write a "Makefile" to the srcdir which shells out to the default objdir.
I suggest something similar to the following: 1. Ship a "configure" script that is just something like: #!/bin/bash HOME=${HOME:-`grep `whoami` /etc/passwd | sed 's,^[^:]*:[^:]*:[^:]*:[^:]*:[^:]*:,,;s,:.*$,,'`} MOZ_OBJDIR=${MOZ_OBJDIR:-${OBJDIR:`uname -m`}} if [ ! -rs ${HOME}/.mozconfig ]; then \ echo -e 'mk_add_options AUTOCONF='${AUTOCONF:-autoconf2.13}'"\n" \ 'mk_add_options MOZ_OBJDIR='${MOZ_OBJDIR}"\n" \ $(echo $* | tr ' ' '\n' | grep '^--' | sed 's,^,ac_add_options ') \ > ${HOME}/.mozconfig fi MAKE=${MAKE:`if [ -x /usr/bin/gmake ]; then echo /usr/bin/gmake ; else echo /usr/bin/make;`} $MAKE -f client.mk && cd $MOZ_OBJDIR && ./configure $* && $MAKE 2. As discussed with Paul on email thread : " Re: [Contribute] Inquiry about Mozilla Coding " (sorry, not a very descriptive subject line, but that's what it was - email headers: In-Reply-To: <201104161654.p3GGsl02009993@pp-web01.phx.mozilla.com> References: <201104161654.p3GGsl02009993@pp-web01.phx.mozilla.com> From: Paul Biggar Date: Sat, 16 Apr 2011 19:04:36 +0100 Message-ID: <BANLkTin1p5m5Z-HtTvUiN8dhEfuX=CKebA@mail.gmail.com> Subject: Re: [Contribute] Inquiry about Mozilla Coding To: jason.vas.dias@gmail.com Cc: contribute@mozilla.org, Ted Mielczarek ) I think if the platform is Linux, then the "make install" phase should NOT run the installer ; instead, it should generate a make script that honors all autoconf / configure variables: $DESTDIR , $PREFIX , $LIBDIR , $BINDIR, $SBINDIR, $DATAROOTDIR , $SYSCONFDIR and will optionally install into $PREFIX/$LIBDIR/$MOZ_APP_INSTALL_DIR (as it does currently, or, if some new $MOZ_USE_SYSTEM_TREE environment variable is set, will actually install the app correctly under $PREFIX/{${BINDIR},${LIBDIR},${DATAROOTDIR}} etc. like any "normal" package.
JVD, I think your part 2 might be useful as a different bug. I don't think the install step is what we're talking about here. I believe your step 1 creates a mozconfig file from the configure options, is that right? It's an interesting approach, but probably a little more complicated than necessary. I think we're looking for a special case in the configure file which: - mkdir $(objdir) - cd $(objdir) - ... - creates a makefile in the srcdir which does nothing but the equivalent of `make -C $(object) $*`
Paul - yes, maybe I was going off on my own hobby-horse with part 2 there - I'll raise a different bug about it. But as I understand this bug : 1. You want the mozilla build to be "familiar" and "compatible" with how most users and package maintainers expect open-source software to be built and installed these days, which is autoconf - users expect to be able to unpack the source tar ball , then EITHER: $ cd src; ./configure --prefix=/usr --libdir=/usr/lib ... && make && make install OR $ mkdir /tmp/my_obj_dir; cd /tmp/my_obj_dir; SRC=${dir_where_I_unpacked_src} $ $SRC/configure --prefix=/usr ... && make && make install But mozilla needs a .mozconfig and run of "client.mk" to run autoconf ; so you're shipping something that is very similar to an "autogen.sh" , as is used by for example all the xorg / freedesktop.com git repositories - they do not check in any "configure" script, only "autogen.sh" which basically runs "automake ... && autoconf ..." to generate configure. So you could ship an autogen.sh - which users expect to see if configure does not exist - which is similar to the example fragment in Comment #18 - it will create a .mozconfig from its arguments if $MOZCONFIG is unset and no .mozconfig is found, create an "automatic OBJDIR" if one is not specified, and run client.mk - the last message it emits should be something like "now change directory to $OBJDIR and enter 'make'" . At the moment, your make files correctly translate .mozconfig settings into default configure arguments, but there seems nothing to do the reverse. I suggest : 1. Change the .mozconfig finding code to look for: $(topsrcdir)/.mozconfig
oops, comment #20 continued - as I was saying: 1. Change the .mozconfig finding code to look for: $(topsrcdir)/.mozconfig , $(HOME)/.mozconfig and set $HOME to $topsrcdir if unset, in that order, so the scripts do not have to bother looking up user's home directory and $(topsrcdir)/.mozconfig will override $(HOME)/.mozconfig . 2. Provide an "autogen.sh" - OR call it "configure" and have it move "configure" to "autogen.sh" before autoconf is run - that: o finds .mozconfig - if none exists, creates it from arguments - this command would actually work: $ function to_mozconfig \ { echo $* | tr ' ' '\n' | grep '^--' | sed 's,^,ac_add_options ,'; } $ to_mozconfig --prefix=/usr --libdir=/usr/lib64 ac_add_options --prefix=/usr ac_add_options --libdir=/usr/lib64 - INCLUDING adding a default $OBJDIR if none is specified : MOZ_OBJDIR=${OBJDIR:-`uname -m`} This will set the default OBJDIR to the architecture name - eg 'i686' for IA32 or 'x86_64' for x86_64 - Then, once a default .mozconfig has been verified to exist or has been created, the autogen.sh / first 'configure' runs client.mk which runs autoconf. This new script can actually be called 'configure' if it does something like: if [ "$0" = configure ]; then mv ./configure autogen.sh; fi BEFORE running : make -f client.mk So then, if autogen.sh succeeds, a new $OBJDIR/configure script is created, AND we are ready to build in $OBJDIR . 3. THEN, to actually conform to user's expectations of what the configure environment variables $LIBDIR, $BINDIR, $DATAROOTDIR, etc. are meant to be used for - eg. I, as a newbie mozilla developer, would have expected ONE SINGLE COPY of the nspr and nss and libxul libraries to be created in ${PREFIX}/${LIBDIR} - is a "make install phase" issue and perhaps should be the topic of new bug report.
This is an example of a script that could generate .mozconfig if non-existent to contain given autoconf args and default $MOZ_OBJDIR and correct $AUTOCONF setting, and setup a build in $OBJDIR by running 'make -f client.mk'
This script could be shipped as initial first script to run to setup build in $OBJDIR (set by default to machine name if unset) and setup default .mozconfig from arguments if no .mozconfig exists.
Attachment #526598 - Attachment is obsolete: true
Attachment #526599 - Attachment is obsolete: true
Example autogen.sh script that could be run as initial script to setup build in $OBJDIR (default: machine name) using $MOZCONFIG (created from arguments if non-existent) and running $topsrcdir/client.mk
Attachment #526601 - Attachment is obsolete: true
oops, problems with "content-type" submitting this file - fixed now - sorry.
better version
Attachment #526604 - Attachment is obsolete: true
Attachment #526607 - Attachment is obsolete: true
version with 'make -f client.mk' UNcommented !
Attachment #526608 - Attachment is obsolete: true
To test - COMMENT OUT line #51 - `make -f client.mk' ! with existing .mozconfig: $ ./autogen.sh Using default OBJDIR: /home/firefox/src/x86_64 Using existing .mozconfig: .mozconfig Configuration succeeded. Now, change directory (cd) to /home/firefox/src/x86_64 and type: 'make' . with existing $HOME/.mozconfig: $ mv .mozconfig $HOME $ ./autogen.sh Using default OBJDIR: /home/firefox/src/x86_64 Using existing .mozconfig: /home/firefox/.mozconfig Configuration succeeded. Now, change directory (cd) to /home/firefox/src/x86_64 and type: 'make' . With no .mozconfig or $HOME/.mozconfig, giving configure arguments: $ mv $HOME/.mozconfig $HOME/.mozconfig.save $ ./autogen.sh $(grep ac_add_options ~/.mozconfig.save | sed 's,ac_add_options ,,') Using default OBJDIR: /home/firefox/src/x86_64 Creating default .mozconfig: /home/firefox/src/.mozconfig Configuration succeeded. Now, change directory (cd) to /home/firefox/src/x86_64 and type: 'make' . $ cat .mozconfig MOZ_OBJDIR=/home/firefox/src/x86_64 AUTOCONF=/usr/bin/autoconf-2.13 ac_add_options --prefix=/usr ac_add_options --libdir=/usr/lib64 ac_add_options --disable-static ac_add_options --enable-shared ac_add_options --with-pic ac_add_options --enable-application=browser ac_add_options --enable-debug ac_add_options --enable-default-toolkit=cairo-gtk2 ac_add_options --enable-plugins ac_add_options --with-pthreads ac_add_options --with-libxul-sdk=/usr/lib64/xulrunner-devel-2.0b13pre ac_add_options --with-system-libxul ac_add_options --with-system-nspr ac_add_options --with-nspr-prefix=/usr ac_add_options --with-system-libevent=/usr ac_add_options --with-system-jpeg=/usr ac_add_options --with-system-zlib=/usr ac_add_options --with-system-bz2=/usr ac_add_options --with-java-bin-path=/usr/java/bin ac_add_options --with-java-include-path=/usr/java/include ac_add_options --disable-strip ac_add_options --disable-install-strip NOTE: ALL of the above should work when run in another directory, ie.: $ $path_to_mozilla_source/autogen.sh
final beta version
Attachment #526609 - Attachment is obsolete: true
Attachment #526614 - Flags: review+
more tests: Setup to build in my own objdir: $ cd /tmp; mkdir ff_build; cd ff_build; export OBJDIR=`pwd`; $ /home/firefox/src/autogen.sh $ /home/firefox/src/autogen.sh $(grep ac_add_options ~/.mozconfig.save | sed 's,ac_add_options ,,') Creating default .mozconfig: /home/firefox/src/.mozconfig make[1]: Entering directory `/home/firefox/src' cd x86_64 /home/firefox/src/configure Adding configure options from /home/firefox/src/.mozconfig: --prefix=/usr --libdir=/usr/lib64 --disable-static --enable-shared --with-pic --enable-application=browser --enable-debug --enable-default-toolkit=cairo-gtk2 --enable-plugins --with-pthreads --with-libxul-sdk=/usr/lib64/xulrunner-devel-2.0b13pre --with-system-libxul --with-system-nspr --with-nspr-prefix=/usr --with-system-libevent=/usr --with-system-jpeg=/usr --with-system-zlib=/usr --with-system-bz2=/usr --with-java-bin-path=/usr/java/bin --with-java-include-path=/usr/java/include --disable-strip --disable-install-strip loading cache ./config.cache checking host system type... x86_64-unknown-linux-gnu checking target system type... x86_64-unknown-linux-gnu checking build system type... x86_64-unknown-linux-gnu checking for mawk... (cached) gawk checking for perl5... (cached) /usr/bin/perl checking for gcc... (cached) /usr/bin/gcc checking whether the C compiler (/usr/bin/gcc -Wa,--compress-debug-sections ) works... yes ... $ cat /home/firefox/src/.mozconfig MOZ_OBJDIR=/tmp/ff_build AUTOCONF=/usr/bin/autoconf-2.13 ac_add_options --prefix=/usr ac_add_options --libdir=/usr/lib64 ac_add_options --disable-static ac_add_options --enable-shared ac_add_options --with-pic ac_add_options --enable-application=browser ac_add_options --enable-debug ac_add_options --enable-default-toolkit=cairo-gtk2 ac_add_options --enable-plugins ac_add_options --with-pthreads ac_add_options --with-libxul-sdk=/usr/lib64/xulrunner-devel-2.0b13pre ac_add_options --with-system-libxul ac_add_options --with-system-nspr ac_add_options --with-nspr-prefix=/usr ac_add_options --with-system-libevent=/usr ac_add_options --with-system-jpeg=/usr ac_add_options --with-system-zlib=/usr ac_add_options --with-system-bz2=/usr ac_add_options --with-java-bin-path=/usr/java/bin ac_add_options --with-java-include-path=/usr/java/include ac_add_options --disable-strip ac_add_options --disable-install-strip
That "cd x86_64" was because it had previously been configured for "x86_64" and it somehow "stuck" in the make environment - that was not the script - hmm... looking into that ...
Oops, no it was my fault: - Script MUST export MOZ_OBJDIR if $OBJDIR is set
Attachment #526614 - Attachment is obsolete: true
in 'plain-text'!
Attachment #526615 - Attachment is obsolete: true
Comment on attachment 526614 [details] Setup default OBJDIR , potentially default .mozconfig from arguments & make -f client.mk To ask for a review on a patch, you need to set the review flag to ? and give a reviewer name (such as :ted). I'm not going to do so for you, though, since you keep posting new versions.
Attachment #526614 - Flags: review+
Also you can change the content-type of a patch from the Details link; you don't need to reupload every time.
$ bash -xf /home/firefox/src/autogen.sh $(grep ac_add_options ~/.mozconfig.save | sed 's,ac_add_options ,,') + ME=/home/firefox/src/autogen.sh + ME=autogen.sh + SRC=/home/firefox/src/ ++ cd /home/firefox/src/ ++ pwd + SRC=/home/firefox/src + cd /home/firefox/src + HOME=/home/firefox ++ which autoconf-2.13 + AUTOCONF213=/usr/bin/autoconf-2.13 + '[' '!' -x /usr/bin/autoconf-2.13 ']' + /usr/bin/autoconf-2.13 --version + egrep -q '2\.13' + MOZCONFIG=.mozconfig + '[' x/tmp/ff_build = x ']' + MOZ_OBJDIR=/tmp/ff_build + export MOZ_OBJDIR + '[' '!' -d /tmp/ff_build ']' + '[' -r .mozconfig ']' + '[' -s .mozconfig ']' + echo 'Using existing .mozconfig: .mozconfig' Using existing .mozconfig: .mozconfig + export MOZCONFIG + make -f client.mk make[1]: Entering directory `/home/firefox/src' cd /tmp/ff_build /home/firefox/src/configure Adding configure options from /home/firefox/src/.mozconfig: --prefix=/usr --libdir=/usr/lib64 --disable-static --enable-shared --with-pic --enable-application=browser --enable-debug --enable-default-toolkit=cairo-gtk2 --enable-plugins --with-pthreads --with-libxul-sdk=/usr/lib64/xulrunner-devel-2.0b13pre --with-system-libxul --with-system-nspr --with-nspr-prefix=/usr --with-system-libevent=/usr --with-system-jpeg=/usr --with-system-zlib=/usr --with-system-bz2=/usr --with-java-bin-path=/usr/java/bin --with-java-include-path=/usr/java/include --disable-strip --disable-install-strip creating cache ./config.cache checking host system type... x86_64-unknown-linux-gnu checking target system type... x86_64-unknown-linux-gnu checking build system type... x86_64-unknown-linux-gnu checking for mawk... no checking for gawk... gawk checking for perl5... no checking for perl... /usr/bin/perl checking for gcc... /usr/bin/gcc checking whether the C compiler (/usr/bin/gcc -Wa,--compress-debug-sections ) works... yes checking whether the C compiler (/usr/bin/gcc -Wa,--compress-debug-sections ) is a cross-compiler... no checking whether we are using GNU C... yes checking whether /usr/bin/gcc accepts -g... yes checking for c++... /usr/bin/g++ checking whether the C++ compiler (/usr/bin/g++ -fpermissive ) works... ^C*** Fix above errors and then restart with "make -f client.mk build" make[1]: *** [configure] Error 1 make: *** [/tmp/ff_build/Makefile] Interrupt
Josh : oops, sorry - got the content-type set OK now, and this should be FINAL version .
FINAL version: set first $MOZCONFIG tried to ${SRC}/.mozconfig , not '.mozconfig', though both end up being OK also added 'reviewer please review' :ted - tag
Attachment #526616 - Attachment is obsolete: true
Attachment #526617 - Flags: review?(ted.mielczarek)
Depends on: 650756
No longer depends on: 650756
Thanks for the patch! I think having an autogen.sh is a good idea, but I'm not sure about the implementation: - do autogen.sh scripts generally call configure too? I don't think they do. - I'm not sure about creating a .mozconfig file - i dont think we want to duplicate functionality from client.mk into autogen.sh. One should rely on the other. Perhaps autogen.sh should be as simple as `make -f client.mk create-configure-file` (or whatever that target is called). The self-configuring I think should be moved to the top of the configure file, in that it moves to an objdir if it's execed from the srcdir. It should probably print a warning at the end that an objdir has been used, and to type 'make -C objdir).
Hi Paul - RE: > do autogen.sh scripts generally call configure too? I don't think they do. Well, the xorg / freedesktop.org ones do : $ cat /usr/src/xorg/everything/xserver/autogen.sh #! /bin/sh srcdir=`dirname $0` test -z "$srcdir" && srcdir=. ORIGDIR=`pwd` cd $srcdir autoreconf --force -v --install || exit 1 cd $ORIGDIR || exit $? $srcdir/configure --enable-maintainer-mode "$@" $ RE: > - I'm not sure about creating a .mozconfig file Well, if autogen.sh doesn't do it , what will ? It ONLY creates a .mozconfig from its arguments if 1) $SRC/.mozconfig does not exist 2) $HOME/.mozconfig doesn not exist Then , if BOTH conditions are satisfied, it passes its '--' prefixed configure arguments to the 'to_mozconfig' function to create $SRC/.mozconfig and then exports $MOZCONFIG to the full path of the newly created file. RE: > Perhaps autogen.sh should be as simple as `make -f client.mk create-configure-file` Yes, but at the moment, no such option exists, and it NEED NOT exist, if this autogen.sh is used. RE: > type 'make -C objdir' To me, this is far more "nonstandard and geeky" than the standard: $ ./autogen.sh <configure args ...> && cd $OBJDIR && make
(In reply to comment #42) > $srcdir/configure --enable-maintainer-mode "$@" Fair enough. > Well, if autogen.sh doesn't do it , what will ? We don't need .mozconfig files anymore (since --enable-application=browser was added. If the user doesn't know about them, no need to mention them. > RE: > Perhaps autogen.sh should be as simple as `make -f client.mk > create-configure-file` > > Yes, but at the moment, no such option exists, and it NEED NOT exist, if > this autogen.sh is used. There is definitely a rule in our Makefiles that runs autoconf2.13, though I don't know what it's called. We should be reusing code where possible. > RE: > type 'make -C objdir' > To me, this is far more "nonstandard and geeky" than the standard: > > $ ./autogen.sh <configure args ...> && cd $OBJDIR && make I think what we're aiming for in the long run is to include configure in mozilla-central. Then it can be the more idiomatic: ./configure && cd $OBJDIR && make Do we want to ship an autogen.sh in this case? Maybe, but if so, we want the logic to be in configure, not in autogen.sh.
OK, so the simplest solution is to first ship autogen.sh, then when the configure code is finished, ship configure . RE: > We don't need .mozconfig files anymore (since --enable-application=browser > was added) AHA ! I didn't know this - it isn't documented anywhere. So if I run the autoconf-2.13 generated configure script WITH ARGUMENTS, it will automatically create a .mozconfig to honor those arguments if one does not exist ? I'll try that and see. What about '--enable-application=xulrunner' ? Will that generate a default .mozconfig also ? I'll investigate some more.
RE: > Then it can be the more idiomatic: > ./configure && cd $OBJDIR && make My understanding of "the idiom" here is that if I want to build any autoconf package in a separate OBJDIR, I simply do: $ mkdir $my_build $ cd $my_build $ ${the_source_dir}/configure && make NOTE: NO setting of $OBJDIR . Perhaps the one change I'd make to the autogen.script I submitted would be to make the "default $OBJDIR" setting if one is not set be the initial `pwd` .
(In reply to comment #44) > So if I run the autoconf-2.13 generated configure script WITH ARGUMENTS, > it will automatically create a .mozconfig to honor those arguments if > one does not exist ? No, it doesn't create a .mozconfig, but it will honour them for that build/objdir. > What about '--enable-application=xulrunner' ? If that's passed as an argument to a configure, it will be used in the objdir.
(In reply to comment #45) > RE: > Then it can be the more idiomatic: > > > ./configure && cd $OBJDIR && make > > My understanding of "the idiom" here is that if I want to build any autoconf > package in a separate OBJDIR, I simply do: > > $ mkdir $my_build > $ cd $my_build > $ ${the_source_dir}/configure && make > > NOTE: NO setting of $OBJDIR . > > Perhaps the one change I'd make to the autogen.script I submitted would be > to make the "default $OBJDIR" setting if one is not set be the initial `pwd` . Yes, your understanding of the idiom is correct. We have a subtle change though, which is that when a user wants to do a srcdir build, we dont want to support that. So instead we want to automatically convert it into an objdir build. Note that things like generating the configure file, regenerating the configure file upon changes to configure.in, regenerating Makefiles upon changes to makefile.in, and the default name of the objdir, are all already handled in client.mk. So if possible, we should either: - make helper scripts defer back to `make -f client.mk` or - factor the work from client.mk into a different script, and make sure that its use in client.mk still works.
Hi - RE: > We have a subtle change though, which is that when a user wants to do a > srcdir build, we dont want to support that. So instead we want to > automatically convert it into an objdir build. The autogen.sh script already does that ; but I'm going to add a flag MOZ_BUILD_HERE=1 so that instead of the default $OBJDIR being $SRC/`uname -m` it will be `pwd` IFF [ "$MOZ_BUILD_HERE" -eq 1 ] && [ "`pwd`" != "$SRC" ]. RE: > we should - make helper scripts defer back to `make -f client.mk` Yes, that is what autogen.sh is doing, but there has to be something that actually does this with some sensible and user-configurable default settings ; there needs to be some "first command to enter" - I think running one shell script with no arguments is the only safe method to expect average users to perform reliably . I think autogen.sh, if no .mozconfig exists and "echo $* | grep '^--'" is empty, should refuse to proceed if no '--enable-application' argument is given, and if it is, should load a default .mozconfig into $SRC/.mozconfig : case ${*//*--enable-application=/} browser*) My understanding is that CURRENTLY, AS-IS , client.mk can't quite handle relocating to a new OBJDIR and operating without ANY .mozconfig straight out-of-the-box ; why make .client.mk any more complicated when what is required can be acheived with use of a well-know autoconf step like running autogen.sh ?
The way distro maintainer builds work is that they create some temporary directory ( eg. /var/rpm/build/XXXXXX-$rpm_version-$rpm_release ) , unpack the source to a SOURCE directory ( eg. /var/rpm/src/XXXXXX/ ) , and run $SRCDIR/configure - ( I know a bit about this because I used to be a redhat package maintainer ) - having ensured that all package dependencies have been built and installed in a unique chroot directory for each build. Wouldn't it be great if distro maintainers did not have to build the complete libxul et al @ 4 times over for: o the /usr/lib/{libnspr,nspr,libnss3,nss3}/* directories o firefox o xulrunner and could depend on your build to generate all the ${PKG}-devel and /usr/lib/lib*.so links required for all the above packages in one build ? It is very easy to incorporate a run of "autogen.sh" into a "%prep" RPM spec file section so that the "%build" section can just do "%configure" .
(In reply to comment #48) > Hi - > RE: > We have a subtle change though, which is that when a user wants to do a > > srcdir build, we dont want to support that. So instead we want to > > automatically convert it into an objdir build. > > The autogen.sh script already does that ; but I'm going to add a flag > MOZ_BUILD_HERE=1 so that instead of the default $OBJDIR being > $SRC/`uname -m` it will be `pwd` IFF [ "$MOZ_BUILD_HERE" -eq 1 ] && [ "`pwd`" > != "$SRC" ]. > > RE: > we should - make helper scripts defer back to `make -f client.mk` > > Yes, that is what autogen.sh is doing, autogen.sh should do nothing but call autoconf213, and then call configure $@. Whatever other logic there is needs to be in configure. > but there has to be something > that actually does this with some sensible and user-configurable > default settings ; there needs to be some "first command to enter" - I dont follow. > I think running one shell script with no arguments is the only > safe method to expect average users to perform reliably . I agree. The expected behaviour is that the user only knows to call configure and make. > I think autogen.sh, if no .mozconfig exists and "echo $* | grep '^--'" is > empty, should refuse to proceed if no '--enable-application' argument is given, > and if it is, should load a default .mozconfig into $SRC/.mozconfig : > case ${*//*--enable-application=/} > browser*) Why? It no longer needs a --enable-application, or a .mozconfig, and has sensible defaults for everything. > My understanding is that CURRENTLY, AS-IS , client.mk can't quite handle > relocating to a new OBJDIR and operating without ANY .mozconfig straight > out-of-the-box ; why make .client.mk any more complicated when what > is required can be acheived with use of a well-know autoconf step > like running autogen.sh ? All of our other logic is in client.mk or configure.in, including generating the objdir and operating without any .mozconfig. What doesn't work is the relocating. That needs to go to the same place as our existing logic. autogen.sh needs to do almost nothing, just run autoconf213 (and use the existing client.mk target for it, so that it doesnt need to duplicate the "is it called autoconf213 or autoconf-2.13" logic). configure can handle the objdir relocation (this is what doesnt work now). If we left that to autogen.sh, then when we started shipping configure, we'd lose the objdir relocation.
(In reply to comment #49) > Wouldn't it be great if distro maintainers did not have Sure, but we should do one bug at a time :)
AFAIK, the way distributions build Firefox nowadays is: 1) Build XULRunner --with-system-{nspr,nss,...}, which builds a libxul+xulrunner, but not the other system libraries. 2) Build Firefox --with-libxul-sdk to make it use the existing libxul from the XULRunner build, which just builds Firefox and not all the other bits. But I don't really see how any of this is relevant to this bug. Normal users probably don't care about any of this.
One other thing: the great thing about all "autogen.sh" scripts is that they cause a "config.cache" to be created in the SOURCE directory; the configure arguments you supply to them are meant to be "cached" as the defaults to use if new args are not supplied - so by providing an autogen.sh like the one I attached to this bug, that accepts arguments and writes them to a .mozconfig file, you're doing exactly that, via the .mozconfig file ; and it works as expected; users then don't have to enter ANY arguments to configure and the ones given to "autogen.sh" will be used; but if they do give arguments, they override the defaults, which is the job of your configure.in script.
RE: Comment #50: OK, so autogen.sh should be just: #/bin/bash export SRCDIR=${0%/*} export OBJDIR=`pwd` pushd $SRCDIR && autoconf-2.13 && ${SRCDIR}/configure $* && popd ??
Nope, that doesn't work - it configures only in $SRCDIR, not $OBJDIR .
(In reply to comment #54) > RE: Comment #50: OK, so autogen.sh should be just: > > #/bin/bash > export SRCDIR=${0%/*} > export OBJDIR=`pwd` > pushd $SRCDIR && autoconf-2.13 && ${SRCDIR}/configure $* && popd > > ?? I would use `make -f client.mk configure-files` instead of calling autoconf. And now I see why you wanted to create a .mozconfig file: so that `make -f client.mk configure` would get it's values from somewhere. Hmmm. Is it absolutely idiomatic that autogen.sh does the configure? It would be a lot simpler if not.
(In reply to comment #55) > Nope, that doesn't work - it configures only in $SRCDIR, not $OBJDIR . Right, because the logic to auto-relocate to OBJDIR doesnt exist yet, and needs to be added in this bug.
RE: Comment #52: > the way distributions build Firefox nowadays is: > 1) Build XULRunner --with-system-{nspr,nss,...}, which builds a > libxul+xulrunner, but not the other system libraries Yes, but there are many packages involved here: - XULRunner + libnspr + libnss3 + firefox each of which many have both ${PKG} and ${PKG}-devel sub-packages, with variants for each architecture. Most big packages such as xorg / glibc build their "runtime" libs - shipped in $PKG - in the same build as their $PKG-devel libs - but the way mozilla does its install makes this very difficult. So they probably do do complete separate builds to build each component - what a waste of time , when with a little tweaking, the mozilla build could do it all for them. And I think probably most "users" wouldn't go near any compiler and that "users" of the mozilla build system are PRIMARILY people like distro maintainers - I just like maintaining my own distro.
I disagree heartily with that last sentiment. We have lots of users who are developers who don't have experience developing with Mozilla. That does not mean they care one whit about system libraries vs. anything else, just that they want to build a working Firefox without much fuss. It may be well and good to make life easier for distro maintainers, but that is not this bug. If I understand you correctly now, you're proposing that a simple Firefox build should produce results that are installable as if you separately built and installed NSPR, NSS, etc as system libraries. Is that correct? That may be a useful goal (but I would want to hear feedback from current distro packagers), but it is not this bug.
RE: Comment #55 - AHA ! NOW I GET IT !! You want configure to enforce relocation into an $OBJDIR no matter what arguments it is given or what $OBJDIR is set to , based on defaults created by client.mk . Yes, but that is precisely what happens when you run the example autogen.sh and a .mozconfig is created for $OBJDIR - why is more client.mk code required ? Once you run the example autogen.sh (currently you must set $OBJDIR or $MOZ_OBJDIR, but this is easily rectified) , $SRC/config.cache and $SRC/.mozconfig are set up ONLY for building in $OBJDIR, and even if you 'cd $SRCDIR; make' the build will still take place in the $OBJDIR set during the autogen.sh run . So why modify configure.in script / client.mk at all for this bug when it can be handled entirely by something like autogen.sh ?
Hi Ted - RE: Comment #59 : > I disagree heartily with that last sentiment. We have lots of users who are > developers who don't have experience developing with Mozilla. Yes, but everyone would appreciate it if builds completed more quickly and used less disk space ? And anyone who builds a package from source and installs it is by definition a "distro maintainer", whether they realize it or want to be one or not - that is the problem - mozilla installs really need to maintain their distros more efficiently, and I think fixing this bug would be part of doing so . But yes, it is a digression from the specifics of this bug report. What would you recommend ? autoconf.sh or configure.in / client.mk script ?
sorry: last line should read: autogen.sh or configure.in / client.mk ? I've satisfied the needs of my build system with autogen.sh - maybe someone else could put in needed changes to configure.in for this bug, because I guess I still don't understand at the moment exactly what these should be ?
This version now honors a MOZ_BUILD_HERE=1 setting, so if $OBJDIR is NOT set, $OBJDIR and $MOZ_OBJDIR are set to `pwd` : $ mkdir /tmp/my_build; cd /tmp/my_build; $ MOZ_BUILD_HERE=1 $MOZILLA_SRC/autogen.sh OTHERWISE, if NO OBJDIR or $MOZ_OBJDIR setting exists, and NO $MOZCONFIG exists, then creates a ${SRC}/.mozconfig file with 'MOZ_OBJDIR='$OBJDIR which is set to $MOZILLA_SRC/`uname -m` if nothing else is specified.
Attachment #526617 - Attachment is obsolete: true
Attachment #526617 - Flags: review?(ted.mielczarek)
Attachment #526815 - Flags: review?(ted.mielczarek)
$ MOZ_BUILD_HERE=1 bash -xf ~/src/autogen.sh + ME=/home/firefox/src/autogen.sh + ME=autogen.sh + SRC=/home/firefox/src/ ++ cd /home/firefox/src/ ++ pwd + SRC=/home/firefox/src ++ tr -d '\n' ++ pwd + XDIR=/tmp/ff_build + cd /home/firefox/src + HOME=/home/firefox ++ which autoconf-2.13 + AUTOCONF213=/usr/bin/autoconf-2.13 + '[' '!' -x /usr/bin/autoconf-2.13 ']' + egrep -q '2\.13' + /usr/bin/autoconf-2.13 --version + MOZCONFIG=/home/firefox/src/.mozconfig + '[' x/tmp/ff_build = x ']' + MOZ_OBJDIR=/tmp/ff_build + export MOZ_OBJDIR + '[' '!' -d /tmp/ff_build ']' + '[' -r /home/firefox/src/.mozconfig ']' + MOZCONFIG=/home/firefox/.mozconfig + '[' -r /home/firefox/.mozconfig ']' + MOZCONFIG=/home/firefox/src/.mozconfig + echo 'Creating default .mozconfig: /home/firefox/src/.mozconfig' Creating default .mozconfig: /home/firefox/src/.mozconfig ++ to_mozconfig +++ tr ' ' '\n' +++ sed 's,^,ac_add_options ,;s/$/\\n/' +++ grep '^--' +++ echo ++ echo + echo -e 'MOZ_OBJDIR=/tmp/ff_build\n' 'AUTOCONF=/usr/bin/autoconf-2.13\n' '' + export MOZCONFIG + make -f client.mk make[1]: Entering directory `/home/firefox/src' cd /tmp/ff_build /home/firefox/src/configure creating cache ./config.cache checking host system type... x86_64-unknown-linux-gnu checking target system type... x86_64-unknown-linux-gnu checking build system type... x86_64-unknown-linux-gnu
I'm now working exclusively on bug #650756 .
Blocks: contrib-engagement
Whiteboard: [contrib-engagement]
OK, well I guess bug #650756 is a total non-starter from mozilla's perspective - I'll keep this as "my private mozilla installer patch" . I'm still happy to work on this bug, which, AFAICS , is: 1. The command: $ mkdir $OBJDIR; export OBJDIR; 'make -f client.mk $OBJDIR/configure' should ONLY run autoconf-2.13 to create the $OBJDIR/configure script, which should then require NO $MOZCONFIG or $HOME/.mozconfig, but should handle default argument setting and any default .mozconfig creation itself. 2. Provide an autogen.sh which does the above 'make -f client.mk ${OBJDIR:-`./new_objdir.sh`}/configure' command. I can do this now if you want.
I don't think we need that complexity: as noted earlier, we should just: * check in configure, add hg hooks to make sure it is always the "right one" * if you try to configure a srcdir build, use a default objdir Then we don't need/shouldn't have client.mk at all, except for the tinderboxes which continue to use it. (I haven't used client.mk in years.)
We'll continue to use client.mk since it handles our Universal Mac builds and PGO builds, but we could certainly fix those in other ways and deprecate it in the future.
So do you want me to have a look at testing / developing client.mk + configure.in patch for this or not ? NOTE: please have a look at bug #652807 - I think many configure affecting environment variables, such as $CXX and $OBJDIR, are NOT being correctly inherited by the make environment, which is why just doing : $ autoconf-2.13 && ( export OBJDIR=$my_objdir; ./configure ) does NOT pick up or use $OBJDIR .
Er, OBJDIR is not a variable that configure uses currently. I think we should do what bsmedberg suggests in comment 67. We've had a bug on committing the generated configure file for a while now, we should just do it. The desired build process should just be: "./configure && make" The only tricky bit is that we would like configure to create an objdir instead of configuring in the srcdir, which means that we need to do something to cause "make" in the srcdir to work.
Comment on attachment 526815 [details] Setup default OBJDIR , potentially default .mozconfig from arguments & make -f client.mk Thanks for looking at this, but I don't think this is the approach we want.
Attachment #526815 - Flags: review?(ted.mielczarek) → review-
Product: Core → Firefox Build System
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: