Make the build steps familiar to new contributors

NEW
Unassigned

Status

defect
8 years ago
a year ago

People

(Reporter: paul.biggar, Unassigned)

Tracking

(Depends on 1 bug, Blocks 1 bug)

Trunk
x86
All
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 11 obsolete attachments)

2.10 KB, text/plain
ted
: review-
Details
Reporter

Description

8 years ago
From the docs, I believe the build steps are:

   make -f client.mk build

We've removed the need to have a mozconfig file (yay!), and so newbies are not-too-far from just having to run |make|.

I tried omitting "build" and it seems to still work. Is is supposed to work?

Can we make it so that we can omit |-f client.mk|? The simple first attempt of copying client.mk to Makefile didn't work, as Makefile is generated from Makefile.in.

On the other hand, copying to GNUmakefile worked, though renaming didn't. Do we already require GNU make? If so, there may be a tiny GNUmakefile we can write which redirects to |make -f client.mk|. Or maybe there's a better solution.
(In reply to comment #0)
> From the docs, I believe the build steps are:
> 
>    make -f client.mk build
> 
> We've removed the need to have a mozconfig file (yay!), and so newbies are
> not-too-far from just having to run |make|.
> 
> I tried omitting "build" and it seems to still work. Is is supposed to work?

Yes.
 
> Can we make it so that we can omit |-f client.mk|? The simple first attempt of
> copying client.mk to Makefile didn't work, as Makefile is generated from
> Makefile.in.

Not easily.  client.mk runs autoconf and configure.

> On the other hand, copying to GNUmakefile worked, though renaming didn't. Do we
> already require GNU make? If so, there may be a tiny GNUmakefile we can write
> which redirects to |make -f client.mk|. Or maybe there's a better solution.

So the problem is that we already have a Makefile.in in the root directory of the source tree, and adding a Makefile in the root too might melt builds where the objdir and the srcdir are the same.
In theory we can hg mov |Makefile.in| to |main.mk| or some such, and create a |Makefile| shim that translates to client.mk if no target, or |main.mk| if a target and adjust other code as needed.

(I think my above will be the shortest path to victory for most of our tooling, dependant projects/repos and users; but it has not been tested and I'm not sure how well it will work)
Reporter

Comment 3

8 years ago
(In reply to comment #1)
> (In reply to comment #0)
> > On the other hand, copying to GNUmakefile worked, though renaming didn't. Do we
> > already require GNU make? If so, there may be a tiny GNUmakefile we can write
> > which redirects to |make -f client.mk|. Or maybe there's a better solution.
> 
> So the problem is that we already have a Makefile.in in the root directory of
> the source tree, and adding a Makefile in the root too might melt builds where
> the objdir and the srcdir are the same.

Yes, that appears to be the problem. FWIW, renaming client.mk to GNUmakefile, and doing s/\bclient\.mk\b/GNUmakefile/ gives me a successful build by just typing |make| (which includes running autoconf and automake).
Reporter

Comment 4

8 years ago
You know, I'm slightly rethinking this. The goal is to make a build familiar to new contributor. The most familiar thing is:

 ./configure && make
or
 mkdir objdir && cd objdir && ../configure && make

By which I mean, this is more familiar than a system that just uses |make|. So perhaps we should just check in configure?
Summary: Change the build steps to just "make" → Make the build steps familiar to new contributors
hm, my homemade build script (see below) has three main steps (checkout, build, package): IIUC this bug touches just the middle one?

Here it is; line 2 has to be customized for your objdir:

#!/bin/bash
export AJM_OBJDIR=obj-x86_64-unknown-linux-gnu
date && \
echo 'python client.py checkout' && \
python client.py checkout && \
date && \
echo 'make -f client.mk build' && \
make -f client.mk build && \
test -n "$AJM_OBJDIR" -a -d $AJM_OBJDIR && \
date && \
echo "make -C $AJM_OBJDIR package" && \
make -C $AJM_OBJDIR package
echo 'Exit status' $?
(In reply to comment #4)
> You know, I'm slightly rethinking this. The goal is to make a build familiar to
> new contributor. The most familiar thing is:
> 
>  ./configure && make
> or
>  mkdir objdir && cd objdir && ../configure && make
> 
> By which I mean, this is more familiar than a system that just uses |make|. So
> perhaps we should just check in configure?

I don't think expecting a user to run autoconf is much more of a burden beyond that.  The only tricky bit is that we're still on autoconf 2.13.  Today

autoconf213 && ./configure && make

should work.
Also, back in the ancient days we used to check in configure.  Somebody who was around then would know why we don't do that anymore.
So, back then we had a script that watched for checkins to configure.in and checked in the matching configure. We didn't have that when we moved to hg, so we didn't import configure.

I think the only issue is people forgetting to check in the matching configure.
It occurs to me that we could easily write a Hg hook to ensure that both configure and configure.in are changed if either one is changed. If we wanted to be super-paranoid, we could have it run autoconf-2.13 on the configure.in and make sure it matches the configure in the patch.
Also, our source tarballs for releases include configure, and there `./configure && make` should work just fine.
Having a hard time composing all my thoughts this morning, but I'd rather not encourage "configure && make" in hg checkouts if we can avoid it, since that creates a srcdir build and pollutes your src tree with a bunch of crap, which is probably not going to be helpful for new contributors.
Yeah, that's the problem here.  client.mk is better than running autoconf/configure/make by hand, just less discoverable.
Checking in configure is bug 464053, with the amusing result that if you develop on Windows or Mac, your generated configure will be just fine, but if you develop on the most common Linux distro, you'd be checking in a freaky crap configure. Dunno whether or not Ubuntu still has a hacky autoconf2.13, but somebody could check that, and write the hooks to check for a configure in any configure.in push and to check for #include <sys/types.h> in configure pushes, and land it.

Couldn't we solve the srcdir build problem by just copying the bits in client.mk that decide if you don't specify an objdir, then you'll get MOZ_OBJDIR = obj-$(CONFIG_GUESS), into /Makefile.in? We've said for years that we want to disallow srcdir builds: here's our chance!
Once you've run configure it's too late, you have a srcdir build.

Although, I wonder if we could add hacks in configure to check if it's being run from $topsrcdir, and do a mkdir, cd, and recursively invoke itself in that directory. I have no idea if that would work, but it might be interesting to try.
Reporter

Updated

8 years ago
Whiteboard: [contrib-engagement]
Reporter

Comment 15

8 years ago
Have we got consensus on how we'd like it to behave (ignoring the technical details of how we'd make that work).

Do we want:

1.   $ make

or

2.   $ ./configure && make (let's assume this makes an objdir)

or

3.   $ mkdir objdir && cd objdir && ../configure && make


Or do we want to support all 3 of the above?
I think #2 is familiar enough, since it's how 99% of open source projects are built. If we can make it create an objdir, then it ought to be basically perfect. (We should not break #3, since that's how client.mk builds work under the hood right now anyway.)
Many people currently do #3, so we should keep that working. Making #2 use an objdir would be good. It could just write a "Makefile" to the srcdir which shells out to the default objdir.

Comment 18

8 years ago
I suggest something similar to the following:

1. Ship a "configure" script that is just something like:

   #!/bin/bash
   HOME=${HOME:-`grep `whoami` /etc/passwd | sed 's,^[^:]*:[^:]*:[^:]*:[^:]*:[^:]*:,,;s,:.*$,,'`}
   MOZ_OBJDIR=${MOZ_OBJDIR:-${OBJDIR:`uname -m`}}
   if [ ! -rs ${HOME}/.mozconfig ]; then \
   echo -e 'mk_add_options AUTOCONF='${AUTOCONF:-autoconf2.13}'"\n" \
           'mk_add_options MOZ_OBJDIR='${MOZ_OBJDIR}"\n" \
           $(echo $* | tr ' ' '\n' | grep '^--' | sed 's,^,ac_add_options ') \
          > ${HOME}/.mozconfig
   fi
   MAKE=${MAKE:`if [ -x /usr/bin/gmake ]; then echo /usr/bin/gmake ; else echo /usr/bin/make;`}
   $MAKE -f client.mk && cd $MOZ_OBJDIR && ./configure $* && $MAKE
   
2. As discussed with Paul on email thread : 
   " Re: [Contribute] Inquiry about Mozilla Coding "
   (sorry, not a very descriptive subject line, but that's what it was -
    email headers: 
 In-Reply-To: <201104161654.p3GGsl02009993@pp-web01.phx.mozilla.com>
 References: <201104161654.p3GGsl02009993@pp-web01.phx.mozilla.com>
 From: Paul Biggar
 Date: Sat, 16 Apr 2011 19:04:36 +0100
 Message-ID: <BANLkTin1p5m5Z-HtTvUiN8dhEfuX=CKebA@mail.gmail.com>
 Subject: Re: [Contribute] Inquiry about Mozilla Coding
 To: jason.vas.dias@gmail.com
 Cc: contribute@mozilla.org,
  Ted Mielczarek
  )
 
   
  I think if the platform is Linux, then the "make install" phase should NOT
  run the installer ; instead, it should generate a make script that honors
  all autoconf / configure variables:
   $DESTDIR , $PREFIX , $LIBDIR , $BINDIR, $SBINDIR, $DATAROOTDIR , $SYSCONFDIR
  and will optionally install into $PREFIX/$LIBDIR/$MOZ_APP_INSTALL_DIR 
  (as it does currently, or, if some new $MOZ_USE_SYSTEM_TREE environment
  variable is set, will actually install the app correctly under
  $PREFIX/{${BINDIR},${LIBDIR},${DATAROOTDIR}} etc. like any "normal" package.
Reporter

Comment 19

8 years ago
JVD, I think your part 2 might be useful as a different bug. I don't think the install step is what we're talking about here.

I believe your step 1 creates a mozconfig file from the configure options, is that right? It's an interesting approach, but probably a little more complicated than necessary. I think we're looking for a special case in the configure file which:

- mkdir $(objdir)
- cd $(objdir)
- ...
- creates a makefile in the srcdir which does nothing but the equivalent of `make -C $(object) $*`

Comment 20

8 years ago
Paul - yes, maybe I was going off on my own hobby-horse with part 2 there - I'll raise a different bug about it.

But as I understand this bug :

1. You want the mozilla build to be "familiar" and "compatible" with how
   most users and package maintainers expect open-source software to be
   built and installed these days, which is autoconf - users expect to
   be able to unpack the source tar ball , then EITHER:
   $ cd src; ./configure --prefix=/usr --libdir=/usr/lib ... &&
     make && make install
   OR
   $ mkdir /tmp/my_obj_dir; cd /tmp/my_obj_dir; SRC=${dir_where_I_unpacked_src}
   $ $SRC/configure --prefix=/usr ... &&
     make && make install

But mozilla needs a .mozconfig and run of "client.mk" to run autoconf ; so you're shipping something that is very similar to an "autogen.sh" , as is
used by for example all the xorg / freedesktop.com git repositories - they
do not check in any "configure" script, only "autogen.sh" which basically runs
"automake ... && autoconf ..." to generate configure. 

So you could ship an autogen.sh - which users expect to see if configure does not exist - which is similar to the example fragment in Comment #18 - it will create a .mozconfig from its arguments if $MOZCONFIG is unset and no .mozconfig is found, create an "automatic OBJDIR" if one is not specified, and run client.mk - the last message it emits should be something like 
"now change directory to $OBJDIR and enter 'make'" .
 
At the moment, your make files correctly translate .mozconfig settings into
default configure arguments, but there seems nothing to do the reverse.

I suggest :
  1. Change the .mozconfig finding code to look for:
 $(topsrcdir)/.mozconfig

Comment 21

8 years ago
oops, comment #20 continued - as I was saying:

1. Change the .mozconfig finding code to look for:
 $(topsrcdir)/.mozconfig , $(HOME)/.mozconfig
   and set $HOME to $topsrcdir if unset,
   in that order, so the scripts do not have to bother looking up user's home
   directory and $(topsrcdir)/.mozconfig will override $(HOME)/.mozconfig .

2. Provide an "autogen.sh" - OR call it "configure" and have it move "configure"
   to "autogen.sh" before autoconf is run -
   that:
   o  finds .mozconfig - if none exists, creates it from arguments - this 
      command would actually work:
      
      $ function to_mozconfig \
        { echo $* | tr ' ' '\n' | grep '^--' | sed 's,^,ac_add_options ,'; }

      $ to_mozconfig --prefix=/usr --libdir=/usr/lib64
      ac_add_options --prefix=/usr
      ac_add_options --libdir=/usr/lib64

      - INCLUDING adding a default $OBJDIR if none is specified :
     
      MOZ_OBJDIR=${OBJDIR:-`uname -m`}

      This will set the default OBJDIR to the architecture name - eg 'i686'
      for IA32 or 'x86_64' for x86_64 -  

   Then, once a default .mozconfig has been verified to exist or has been
   created, the autogen.sh / first 'configure' runs client.mk which runs
   autoconf.

   This new script can actually be called 'configure' if it does something 
   like:  
         if [ "$0" = configure ]; then mv ./configure autogen.sh; fi
   BEFORE running :
         make -f client.mk

   So then, if autogen.sh succeeds, a new $OBJDIR/configure script is   
   created, AND we are ready to build in $OBJDIR .

3. THEN, to actually conform to user's expectations of what the configure
   environment variables $LIBDIR, $BINDIR, $DATAROOTDIR, etc. are meant
   to be used for - eg. I, as a newbie mozilla developer, would have expected
   ONE SINGLE COPY of the nspr and nss and libxul libraries to be created in   
   ${PREFIX}/${LIBDIR} - is a "make install phase" issue and perhaps should
   be the topic of new bug report.

Comment 22

8 years ago
This is an example of a script that could generate .mozconfig if non-existent to contain given autoconf args and default $MOZ_OBJDIR and correct $AUTOCONF setting, and setup a build in $OBJDIR by running 'make -f client.mk'

Comment 24

8 years ago
This script could be shipped as initial first script to run to setup build in $OBJDIR (set by default to machine name if unset) and setup default .mozconfig from arguments if no .mozconfig exists.
Attachment #526598 - Attachment is obsolete: true
Attachment #526599 - Attachment is obsolete: true

Comment 25

8 years ago
Example autogen.sh script that could be run as initial script to setup build in
$OBJDIR (default: machine name) using $MOZCONFIG (created from arguments if non-existent) and running $topsrcdir/client.mk
Attachment #526601 - Attachment is obsolete: true

Comment 26

8 years ago
oops, problems with "content-type" submitting this file - fixed now - sorry.

Comment 28

8 years ago
better version
Attachment #526604 - Attachment is obsolete: true
Attachment #526607 - Attachment is obsolete: true

Comment 29

8 years ago
version with 'make -f client.mk' UNcommented !
Attachment #526608 - Attachment is obsolete: true

Comment 30

8 years ago
To test - COMMENT OUT line #51 - `make -f client.mk' !

with existing .mozconfig:

 $ ./autogen.sh
Using default OBJDIR: /home/firefox/src/x86_64
Using existing .mozconfig: .mozconfig
Configuration succeeded.
Now, change directory (cd) to /home/firefox/src/x86_64 and type: 'make' .

with existing $HOME/.mozconfig:

 $ mv .mozconfig $HOME
 $ ./autogen.sh
Using default OBJDIR: /home/firefox/src/x86_64
Using existing .mozconfig: /home/firefox/.mozconfig
Configuration succeeded.
Now, change directory (cd) to /home/firefox/src/x86_64 and type: 'make' .

With no .mozconfig or $HOME/.mozconfig, giving configure arguments:

 $ mv $HOME/.mozconfig $HOME/.mozconfig.save

$  ./autogen.sh $(grep ac_add_options ~/.mozconfig.save | sed 's,ac_add_options ,,')
Using default OBJDIR: /home/firefox/src/x86_64
Creating default .mozconfig: /home/firefox/src/.mozconfig
Configuration succeeded.
Now, change directory (cd) to /home/firefox/src/x86_64 and type: 'make' .
$ cat .mozconfig
MOZ_OBJDIR=/home/firefox/src/x86_64
 AUTOCONF=/usr/bin/autoconf-2.13
 ac_add_options --prefix=/usr
 ac_add_options --libdir=/usr/lib64
 ac_add_options --disable-static
 ac_add_options --enable-shared
 ac_add_options --with-pic
 ac_add_options --enable-application=browser
 ac_add_options --enable-debug
 ac_add_options --enable-default-toolkit=cairo-gtk2
 ac_add_options --enable-plugins
 ac_add_options --with-pthreads
 ac_add_options --with-libxul-sdk=/usr/lib64/xulrunner-devel-2.0b13pre
 ac_add_options --with-system-libxul
 ac_add_options --with-system-nspr
 ac_add_options --with-nspr-prefix=/usr
 ac_add_options --with-system-libevent=/usr
 ac_add_options --with-system-jpeg=/usr
 ac_add_options --with-system-zlib=/usr
 ac_add_options --with-system-bz2=/usr
 ac_add_options --with-java-bin-path=/usr/java/bin
 ac_add_options --with-java-include-path=/usr/java/include
 ac_add_options --disable-strip
 ac_add_options --disable-install-strip


 
NOTE: ALL of the above should work when run in another directory, ie.:
 $ $path_to_mozilla_source/autogen.sh

Comment 31

8 years ago
final beta version
Attachment #526609 - Attachment is obsolete: true
Attachment #526614 - Flags: review+

Comment 32

8 years ago
more tests:

Setup to build in my own objdir:

 $ cd /tmp; mkdir ff_build; cd ff_build; export OBJDIR=`pwd`;
 $ /home/firefox/src/autogen.sh
$ /home/firefox/src/autogen.sh $(grep ac_add_options ~/.mozconfig.save | sed 's,ac_add_options ,,')
Creating default .mozconfig: /home/firefox/src/.mozconfig                                          
make[1]: Entering directory `/home/firefox/src'                                                    
cd x86_64                                                                                          
/home/firefox/src/configure                                                                        
Adding configure options from /home/firefox/src/.mozconfig:                                        
  --prefix=/usr                                                                                    
  --libdir=/usr/lib64                                                                              
  --disable-static                                                                                 
  --enable-shared                                                                                  
  --with-pic                                                                                       
  --enable-application=browser                                                                     
  --enable-debug                                                                                   
  --enable-default-toolkit=cairo-gtk2                                                              
  --enable-plugins                                                                                 
  --with-pthreads                                                                                  
  --with-libxul-sdk=/usr/lib64/xulrunner-devel-2.0b13pre                                           
  --with-system-libxul                                                                             
  --with-system-nspr                                                                               
  --with-nspr-prefix=/usr                                                                          
  --with-system-libevent=/usr                                                                      
  --with-system-jpeg=/usr                                                                          
  --with-system-zlib=/usr                                                                          
  --with-system-bz2=/usr                                                                           
  --with-java-bin-path=/usr/java/bin                                                               
  --with-java-include-path=/usr/java/include                                                       
  --disable-strip                                                                                  
  --disable-install-strip                                                                          
loading cache ./config.cache                                                                       
checking host system type... x86_64-unknown-linux-gnu                                              
checking target system type... x86_64-unknown-linux-gnu                                            
checking build system type... x86_64-unknown-linux-gnu                                             
checking for mawk... (cached) gawk                                                                 
checking for perl5... (cached) /usr/bin/perl                                                       
checking for gcc... (cached) /usr/bin/gcc                                                          
checking whether the C compiler (/usr/bin/gcc  -Wa,--compress-debug-sections ) works... yes        
 ...


$ cat /home/firefox/src/.mozconfig
MOZ_OBJDIR=/tmp/ff_build
 AUTOCONF=/usr/bin/autoconf-2.13
 ac_add_options --prefix=/usr
 ac_add_options --libdir=/usr/lib64
 ac_add_options --disable-static
 ac_add_options --enable-shared
 ac_add_options --with-pic
 ac_add_options --enable-application=browser
 ac_add_options --enable-debug
 ac_add_options --enable-default-toolkit=cairo-gtk2
 ac_add_options --enable-plugins
 ac_add_options --with-pthreads
 ac_add_options --with-libxul-sdk=/usr/lib64/xulrunner-devel-2.0b13pre
 ac_add_options --with-system-libxul
 ac_add_options --with-system-nspr
 ac_add_options --with-nspr-prefix=/usr
 ac_add_options --with-system-libevent=/usr
 ac_add_options --with-system-jpeg=/usr
 ac_add_options --with-system-zlib=/usr
 ac_add_options --with-system-bz2=/usr
 ac_add_options --with-java-bin-path=/usr/java/bin
 ac_add_options --with-java-include-path=/usr/java/include
 ac_add_options --disable-strip
 ac_add_options --disable-install-strip

Comment 33

8 years ago
That "cd x86_64" was because it had previously been configured for "x86_64" and it somehow "stuck" in the make environment - that was not the script - hmm...
looking into that ...

Comment 34

8 years ago
Oops, no it was my fault:
 - Script MUST export MOZ_OBJDIR if $OBJDIR is set
Attachment #526614 - Attachment is obsolete: true

Comment 35

8 years ago
in 'plain-text'!
Attachment #526615 - Attachment is obsolete: true
Comment on attachment 526614 [details]
Setup default OBJDIR , potentially default .mozconfig from arguments & make -f client.mk

To ask for a review on a patch, you need to set the review flag to ? and give a reviewer name (such as :ted).  I'm not going to do so for you, though, since you keep posting new versions.
Attachment #526614 - Flags: review+
Also you can change the content-type of a patch from the Details link; you don't need to reupload every time.

Comment 38

8 years ago
$ bash -xf /home/firefox/src/autogen.sh $(grep ac_add_options ~/.mozconfig.save | sed 's,ac_add_options ,,')
+ ME=/home/firefox/src/autogen.sh                                                                           
+ ME=autogen.sh                                                                                             
+ SRC=/home/firefox/src/                                                                                    
++ cd /home/firefox/src/                                                                                    
++ pwd                                                                                                      
+ SRC=/home/firefox/src                                                                                     
+ cd /home/firefox/src                                                                                      
+ HOME=/home/firefox                                                                                        
++ which autoconf-2.13                                                                                      
+ AUTOCONF213=/usr/bin/autoconf-2.13                                                                        
+ '[' '!' -x /usr/bin/autoconf-2.13 ']'                                                                     
+ /usr/bin/autoconf-2.13 --version                                                                          
+ egrep -q '2\.13'                                                                                          
+ MOZCONFIG=.mozconfig                                                                                      
+ '[' x/tmp/ff_build = x ']'                                                                                
+ MOZ_OBJDIR=/tmp/ff_build                                                                                  
+ export MOZ_OBJDIR                                                                                         
+ '[' '!' -d /tmp/ff_build ']'                                                                              
+ '[' -r .mozconfig ']'                                                                                     
+ '[' -s .mozconfig ']'                                                                                     
+ echo 'Using existing .mozconfig: .mozconfig'                                                              
Using existing .mozconfig: .mozconfig                                                                       
+ export MOZCONFIG                                                                                          
+ make -f client.mk                                                                                         
make[1]: Entering directory `/home/firefox/src'                                                             
cd /tmp/ff_build                                                                                            
/home/firefox/src/configure                                                                                 
Adding configure options from /home/firefox/src/.mozconfig:                                                 
  --prefix=/usr                                                                                             
  --libdir=/usr/lib64                                                                                       
  --disable-static
  --enable-shared
  --with-pic
  --enable-application=browser
  --enable-debug
  --enable-default-toolkit=cairo-gtk2
  --enable-plugins
  --with-pthreads
  --with-libxul-sdk=/usr/lib64/xulrunner-devel-2.0b13pre
  --with-system-libxul
  --with-system-nspr
  --with-nspr-prefix=/usr
  --with-system-libevent=/usr
  --with-system-jpeg=/usr
  --with-system-zlib=/usr
  --with-system-bz2=/usr
  --with-java-bin-path=/usr/java/bin
  --with-java-include-path=/usr/java/include
  --disable-strip
  --disable-install-strip
creating cache ./config.cache
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
checking build system type... x86_64-unknown-linux-gnu
checking for mawk... no
checking for gawk... gawk
checking for perl5... no
checking for perl... /usr/bin/perl
checking for gcc... /usr/bin/gcc
checking whether the C compiler (/usr/bin/gcc  -Wa,--compress-debug-sections ) works... yes
checking whether the C compiler (/usr/bin/gcc  -Wa,--compress-debug-sections ) is a cross-compiler... no
checking whether we are using GNU C... yes
checking whether /usr/bin/gcc accepts -g... yes
checking for c++... /usr/bin/g++
checking whether the C++ compiler (/usr/bin/g++  -fpermissive ) works... ^C*** Fix above errors and then restart with               "make -f client.mk build"
make[1]: *** [configure] Error 1
make: *** [/tmp/ff_build/Makefile] Interrupt

Comment 39

8 years ago
Josh : oops, sorry - got the content-type set OK now, and this should be FINAL version .

Comment 40

8 years ago
FINAL version: set first $MOZCONFIG tried to ${SRC}/.mozconfig , not '.mozconfig', though both end up being OK
also added 'reviewer please review' :ted - tag
Attachment #526616 - Attachment is obsolete: true
Attachment #526617 - Flags: review?(ted.mielczarek)

Updated

8 years ago
Depends on: 650756
Reporter

Updated

8 years ago
No longer depends on: 650756
Reporter

Comment 41

8 years ago
Thanks for the patch!

I think having an autogen.sh is a good idea, but I'm not sure about the implementation:

- do autogen.sh scripts generally call configure too? I don't think they do.

- I'm not sure about creating a .mozconfig file

- i dont think we want to duplicate functionality from client.mk into autogen.sh. One should rely on the other. Perhaps autogen.sh should be as simple as `make -f client.mk create-configure-file` (or whatever that target is called).


The self-configuring I think should be moved to the top of the configure file, in that it moves to an objdir if it's execed from the srcdir. It should probably print a warning at the end that an objdir has been used, and to type 'make -C objdir).

Comment 42

8 years ago
Hi Paul -
RE: 
> do autogen.sh scripts generally call configure too? I don't think they do.

Well, the xorg / freedesktop.org ones do :

$ cat /usr/src/xorg/everything/xserver/autogen.sh
#! /bin/sh

srcdir=`dirname $0`
test -z "$srcdir" && srcdir=.

ORIGDIR=`pwd`
cd $srcdir

autoreconf --force -v --install || exit 1
cd $ORIGDIR || exit $?

$srcdir/configure --enable-maintainer-mode "$@"

$

RE: > - I'm not sure about creating a .mozconfig file

Well, if autogen.sh doesn't do it , what will ?

It ONLY creates a .mozconfig from its arguments if
 1) $SRC/.mozconfig does not exist
 2) $HOME/.mozconfig doesn not exist
Then , if BOTH conditions are satisfied, it passes
its '--' prefixed configure arguments to the 'to_mozconfig'
function to create $SRC/.mozconfig and then exports $MOZCONFIG
to the full path of the newly created file.

RE: > Perhaps autogen.sh should be as simple as `make -f client.mk create-configure-file`

Yes, but at the moment, no such option exists, and it NEED NOT exist, if
this autogen.sh is used.

RE: > type 'make -C objdir'
To me, this is far more "nonstandard and geeky" than the standard:

$ ./autogen.sh <configure args ...> && cd $OBJDIR && make
Reporter

Comment 43

8 years ago
(In reply to comment #42)
> $srcdir/configure --enable-maintainer-mode "$@"

Fair enough.


> Well, if autogen.sh doesn't do it , what will ?

We don't need .mozconfig files anymore (since --enable-application=browser was added. If the user doesn't know about them, no need to mention them.


> RE: > Perhaps autogen.sh should be as simple as `make -f client.mk
> create-configure-file`
> 
> Yes, but at the moment, no such option exists, and it NEED NOT exist, if
> this autogen.sh is used.

There is definitely a rule in our Makefiles that runs autoconf2.13, though I don't know what it's called. We should be reusing code where possible.


> RE: > type 'make -C objdir'
> To me, this is far more "nonstandard and geeky" than the standard:
> 
> $ ./autogen.sh <configure args ...> && cd $OBJDIR && make

I think what we're aiming for in the long run is to include configure in mozilla-central. Then it can be the more idiomatic:

   ./configure && cd $OBJDIR && make

Do we want to ship an autogen.sh in this case? Maybe, but if so, we want the logic to be in configure, not in autogen.sh.

Comment 44

8 years ago
OK, so the simplest solution is to first ship autogen.sh, then when the configure code is finished, ship configure .

RE: 
> We don't need .mozconfig files anymore (since --enable-application=browser 
> was added)

AHA ! I didn't know this - it isn't documented anywhere.  

So if I run the autoconf-2.13 generated configure script WITH ARGUMENTS,
it will automatically create a .mozconfig to honor those arguments if
one does not exist ?

I'll try that and see.

What about '--enable-application=xulrunner' ?
Will that generate a default .mozconfig also ?
I'll investigate some more.

Comment 45

8 years ago
RE: >  Then it can be the more idiomatic:

    > ./configure && cd $OBJDIR && make

My understanding of "the idiom" here is that if I want to build any autoconf
package in a separate OBJDIR, I simply do:

   $ mkdir $my_build
   $ cd $my_build
   $ ${the_source_dir}/configure && make

NOTE: NO setting of $OBJDIR .

Perhaps the one change I'd make to the autogen.script I submitted would be
to make the "default $OBJDIR" setting if one is not set be the initial `pwd` .
Reporter

Comment 46

8 years ago
(In reply to comment #44)
> So if I run the autoconf-2.13 generated configure script WITH ARGUMENTS,
> it will automatically create a .mozconfig to honor those arguments if
> one does not exist ?

No, it doesn't create a .mozconfig, but it will honour them for that build/objdir.


> What about '--enable-application=xulrunner' ?

If that's passed as an argument to a configure, it will be used in the objdir.
Reporter

Comment 47

8 years ago
(In reply to comment #45)
> RE: >  Then it can be the more idiomatic:
> 
>     > ./configure && cd $OBJDIR && make
> 
> My understanding of "the idiom" here is that if I want to build any autoconf
> package in a separate OBJDIR, I simply do:
> 
>    $ mkdir $my_build
>    $ cd $my_build
>    $ ${the_source_dir}/configure && make
> 
> NOTE: NO setting of $OBJDIR .
> 
> Perhaps the one change I'd make to the autogen.script I submitted would be
> to make the "default $OBJDIR" setting if one is not set be the initial `pwd` .


Yes, your understanding of the idiom is correct.

We have a subtle change though, which is that when a user wants to do a srcdir build, we dont want to support that. So instead we want to automatically convert it into an objdir build.


Note that things like generating the configure file, regenerating the configure file upon changes to configure.in, regenerating Makefiles upon changes to makefile.in, and the default name of the objdir, are all already handled in client.mk. So if possible, we should either:

- make helper scripts defer back to `make -f client.mk`

or

- factor the work from client.mk into a different script, and make sure that its use in client.mk still works.

Comment 48

8 years ago
Hi -
RE: > We have a subtle change though, which is that when a user wants to do a 
    > srcdir build, we dont want to support that. So instead we want to  
    > automatically convert it into an objdir build.

The autogen.sh script already does that ;  but I'm going to add a flag 
MOZ_BUILD_HERE=1  so that instead of the default $OBJDIR being 
$SRC/`uname -m` it will be `pwd` IFF  [ "$MOZ_BUILD_HERE" -eq 1 ] && [ "`pwd`" != "$SRC" ].

RE: > we should - make helper scripts defer back to `make -f client.mk`

Yes, that is what autogen.sh is doing, but there has to be something
that actually does this with some sensible and user-configurable
default settings ; there needs to be some "first command to enter" -
I think running one shell script with no arguments is the only
safe method to expect average users to perform reliably .

I think autogen.sh, if no .mozconfig exists and "echo $* | grep '^--'" is empty, should refuse to proceed if no '--enable-application' argument is given,
and if it is, should load a default .mozconfig into $SRC/.mozconfig : 
 case ${*//*--enable-application=/}
 browser*)


My understanding is that CURRENTLY, AS-IS , client.mk can't quite handle
relocating to a new OBJDIR and operating without ANY .mozconfig straight
out-of-the-box ;  why make .client.mk any more complicated when what
is required can be acheived with use of a well-know autoconf step
like running autogen.sh ?

Comment 49

8 years ago
The way distro maintainer builds work is that they create some temporary
directory ( eg. /var/rpm/build/XXXXXX-$rpm_version-$rpm_release ) ,
unpack the source to a SOURCE directory ( eg. /var/rpm/src/XXXXXX/ ) ,
and run $SRCDIR/configure - ( I know a bit about this because I used
to be a redhat package maintainer ) - having ensured that all package
dependencies have been built and installed in a unique chroot directory
for each build. Wouldn't it be great if distro maintainers did not have
to build the complete libxul et al @ 4 times over for:
  o the /usr/lib/{libnspr,nspr,libnss3,nss3}/* directories
  o firefox 
  o xulrunner
and could depend on your build to generate all the ${PKG}-devel and /usr/lib/lib*.so links required for all the above packages in one build ?

It is very easy to incorporate a run of "autogen.sh" into a "%prep" RPM spec
file section so that the "%build" section can just do "%configure" .
Reporter

Comment 50

8 years ago
(In reply to comment #48)
> Hi -
> RE: > We have a subtle change though, which is that when a user wants to do a 
>     > srcdir build, we dont want to support that. So instead we want to  
>     > automatically convert it into an objdir build.
> 
> The autogen.sh script already does that ;  but I'm going to add a flag 
> MOZ_BUILD_HERE=1  so that instead of the default $OBJDIR being 
> $SRC/`uname -m` it will be `pwd` IFF  [ "$MOZ_BUILD_HERE" -eq 1 ] && [ "`pwd`"
> != "$SRC" ].
> 
> RE: > we should - make helper scripts defer back to `make -f client.mk`
> 
> Yes, that is what autogen.sh is doing,

autogen.sh should do nothing but call autoconf213, and then call configure $@. Whatever other logic there is needs to be in configure.

> but there has to be something
> that actually does this with some sensible and user-configurable
> default settings ; there needs to be some "first command to enter" -

I dont follow.

> I think running one shell script with no arguments is the only
> safe method to expect average users to perform reliably .

I agree. The expected behaviour is that the user only knows to call configure and make.
 

> I think autogen.sh, if no .mozconfig exists and "echo $* | grep '^--'" is
> empty, should refuse to proceed if no '--enable-application' argument is given,
> and if it is, should load a default .mozconfig into $SRC/.mozconfig : 
>  case ${*//*--enable-application=/}
>  browser*)

Why? It no longer needs a --enable-application, or a .mozconfig, and has sensible defaults for everything.


> My understanding is that CURRENTLY, AS-IS , client.mk can't quite handle
> relocating to a new OBJDIR and operating without ANY .mozconfig straight
> out-of-the-box ;  why make .client.mk any more complicated when what
> is required can be acheived with use of a well-know autoconf step
> like running autogen.sh ?

All of our other logic is in client.mk or configure.in, including generating the objdir and operating without any .mozconfig. What doesn't work is the relocating. That needs to go to the same place as our existing logic.

autogen.sh needs to do almost nothing, just run autoconf213 (and use the existing client.mk target for it, so that it doesnt need to duplicate the "is it called autoconf213 or autoconf-2.13" logic). configure can handle the objdir relocation (this is what doesnt work now). If we left that to autogen.sh, then when we started shipping configure, we'd lose the objdir relocation.
Reporter

Comment 51

8 years ago
(In reply to comment #49)
> Wouldn't it be great if distro maintainers did not have

Sure, but we should do one bug at a time :)
AFAIK, the way distributions build Firefox nowadays is:
1) Build XULRunner --with-system-{nspr,nss,...}, which builds a libxul+xulrunner, but not the other system libraries.
2) Build Firefox --with-libxul-sdk to make it use the existing libxul from the XULRunner build, which just builds Firefox and not all the other bits.

But I don't really see how any of this is relevant to this bug. Normal users probably don't care about any of this.

Comment 53

8 years ago
One other thing: the great thing about all "autogen.sh" scripts is that they
cause a "config.cache" to be created in the SOURCE directory; the 
configure arguments you supply to them are meant to be "cached" as
the defaults to use if new args are not supplied - so by providing
an autogen.sh like the one I attached to this bug, that accepts
arguments and writes them to a .mozconfig file, you're doing
exactly that, via the .mozconfig file ; and it works as expected;
users then don't have to enter ANY arguments to configure and the
ones given to "autogen.sh" will be used; but if they do give arguments,
they override the defaults, which is the job of your configure.in script.

Comment 54

8 years ago
RE:  Comment #50:  OK, so autogen.sh should be just:

#/bin/bash
export SRCDIR=${0%/*}
export OBJDIR=`pwd`
pushd $SRCDIR && autoconf-2.13 && ${SRCDIR}/configure $* && popd

??

Comment 55

8 years ago
Nope, that doesn't work - it configures only in $SRCDIR, not $OBJDIR .
Reporter

Comment 56

8 years ago
(In reply to comment #54)
> RE:  Comment #50:  OK, so autogen.sh should be just:
> 
> #/bin/bash
> export SRCDIR=${0%/*}
> export OBJDIR=`pwd`
> pushd $SRCDIR && autoconf-2.13 && ${SRCDIR}/configure $* && popd
> 
> ??

I would use `make -f client.mk configure-files` instead of calling autoconf.

And now I see why you wanted to create a .mozconfig file: so that `make -f client.mk configure` would get it's values from somewhere. Hmmm. Is it absolutely idiomatic that autogen.sh does the configure? It would be a lot simpler if not.
Reporter

Comment 57

8 years ago
(In reply to comment #55)
> Nope, that doesn't work - it configures only in $SRCDIR, not $OBJDIR .

Right, because the logic to auto-relocate to OBJDIR doesnt exist yet, and needs to be added in this bug.

Comment 58

8 years ago
RE: Comment #52: 
 > the way distributions build Firefox nowadays is:
 > 1) Build XULRunner --with-system-{nspr,nss,...}, which builds a
 > libxul+xulrunner, but not the other system libraries

Yes, but there are many packages involved here:
 - XULRunner + libnspr + libnss3 + firefox
each of which many have both ${PKG} and ${PKG}-devel sub-packages,
with variants for each architecture.

Most big packages such as xorg / glibc build their "runtime" libs -
shipped in $PKG - in the same build as their $PKG-devel libs -
but the way mozilla does its install makes this very difficult.
So they probably do do complete separate builds to build each component -
what a waste of time , when with a little tweaking, the mozilla build
could do it all for them.
  
And I think probably most "users" wouldn't go near any compiler and that
"users" of the mozilla build system are PRIMARILY people like distro
maintainers - I just like maintaining my own distro.
I disagree heartily with that last sentiment. We have lots of users who are developers who don't have experience developing with Mozilla. That does not mean they care one whit about system libraries vs. anything else, just that they want to build a working Firefox without much fuss. It may be well and good to make life easier for distro maintainers, but that is not this bug.

If I understand you correctly now, you're proposing that a simple Firefox build should produce results that are installable as if you separately built and installed NSPR, NSS, etc as system libraries. Is that correct? That may be a useful goal (but I would want to hear feedback from current distro packagers), but it is not this bug.

Comment 60

8 years ago
RE: Comment #55 - AHA ! NOW I GET IT !!

You want configure to enforce relocation into an $OBJDIR no matter what
arguments it is given or what $OBJDIR is set to , based on defaults 
created by client.mk .

Yes, but that is precisely what happens when you run the example autogen.sh
and a .mozconfig is created for $OBJDIR -  why is more client.mk code required ?

Once you run the example autogen.sh (currently you must set $OBJDIR or $MOZ_OBJDIR, but this is easily rectified) ,  $SRC/config.cache and $SRC/.mozconfig are set up ONLY for building in $OBJDIR, and even
if you 'cd $SRCDIR; make' the build will still take place in the
$OBJDIR set during the autogen.sh run .

So why modify configure.in script / client.mk at all for this bug when it can
be handled entirely by something like autogen.sh ?

Comment 61

8 years ago
Hi Ted -

RE: Comment #59 :
> I disagree heartily with that last sentiment. We have lots of users who are
> developers who don't have experience developing with Mozilla.

Yes, but everyone would appreciate it if builds completed more quickly and
used less disk space ?  And anyone who builds a package from source and
installs it is by definition a "distro maintainer", whether they realize it
or want to be one or not - that is the problem - mozilla installs really
need to maintain their distros more efficiently, and I think fixing this bug
would be part of doing so .

But yes, it is a digression from the specifics of this bug report.

What would you recommend ?  autoconf.sh or configure.in / client.mk script ?

Comment 62

8 years ago
sorry: last line should read: autogen.sh or configure.in / client.mk ?

I've satisfied the needs of my build system with autogen.sh - maybe
someone else could put in needed changes to configure.in for this
bug, because I guess I still don't understand at the moment exactly what
these should be ?

Comment 63

8 years ago
This version now honors a MOZ_BUILD_HERE=1 setting, so if $OBJDIR is NOT set,
$OBJDIR and $MOZ_OBJDIR are set to `pwd` :

 $ mkdir /tmp/my_build; cd /tmp/my_build; 
 $ MOZ_BUILD_HERE=1 $MOZILLA_SRC/autogen.sh

OTHERWISE, if NO OBJDIR or $MOZ_OBJDIR setting exists, and NO $MOZCONFIG exists,
then creates a ${SRC}/.mozconfig file with 'MOZ_OBJDIR='$OBJDIR which is set to $MOZILLA_SRC/`uname -m` if nothing else is specified.
Attachment #526617 - Attachment is obsolete: true
Attachment #526617 - Flags: review?(ted.mielczarek)
Attachment #526815 - Flags: review?(ted.mielczarek)

Comment 64

8 years ago
$ MOZ_BUILD_HERE=1 bash -xf ~/src/autogen.sh                                                           
+ ME=/home/firefox/src/autogen.sh                                                                      
+ ME=autogen.sh                                                                                        
+ SRC=/home/firefox/src/                                                                               
++ cd /home/firefox/src/                                                                               
++ pwd                                                                                                 
+ SRC=/home/firefox/src                                                                                
++ tr -d '\n'                                                                                          
++ pwd                                                                                                 
+ XDIR=/tmp/ff_build                                                                                   
+ cd /home/firefox/src                                                                                 
+ HOME=/home/firefox                                                                                   
++ which autoconf-2.13                                                                                 
+ AUTOCONF213=/usr/bin/autoconf-2.13                                                                   
+ '[' '!' -x /usr/bin/autoconf-2.13 ']'                                                                
+ egrep -q '2\.13'
+ /usr/bin/autoconf-2.13 --version
+ MOZCONFIG=/home/firefox/src/.mozconfig
+ '[' x/tmp/ff_build = x ']'
+ MOZ_OBJDIR=/tmp/ff_build
+ export MOZ_OBJDIR
+ '[' '!' -d /tmp/ff_build ']'
+ '[' -r /home/firefox/src/.mozconfig ']'
+ MOZCONFIG=/home/firefox/.mozconfig
+ '[' -r /home/firefox/.mozconfig ']'
+ MOZCONFIG=/home/firefox/src/.mozconfig
+ echo 'Creating default .mozconfig: /home/firefox/src/.mozconfig'
Creating default .mozconfig: /home/firefox/src/.mozconfig
++ to_mozconfig
+++ tr ' ' '\n'
+++ sed 's,^,ac_add_options ,;s/$/\\n/'
+++ grep '^--'
+++ echo
++ echo
+ echo -e 'MOZ_OBJDIR=/tmp/ff_build\n' 'AUTOCONF=/usr/bin/autoconf-2.13\n' ''
+ export MOZCONFIG
+ make -f client.mk
make[1]: Entering directory `/home/firefox/src'
cd /tmp/ff_build
/home/firefox/src/configure
creating cache ./config.cache
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
checking build system type... x86_64-unknown-linux-gnu

Comment 65

8 years ago
I'm now working exclusively on bug #650756 .
Reporter

Updated

8 years ago
Whiteboard: [contrib-engagement]

Comment 66

8 years ago
OK, well I guess bug #650756 is a total non-starter from mozilla's perspective -
I'll keep this as "my private mozilla installer patch" .

I'm still happy to work on this bug, which, AFAICS , is:

1. The command:

   $ mkdir $OBJDIR; export OBJDIR; 'make -f client.mk $OBJDIR/configure'
    
   should ONLY run autoconf-2.13 to create the $OBJDIR/configure script,
   which should then require NO $MOZCONFIG or $HOME/.mozconfig, but
   should handle default argument setting and any default .mozconfig
   creation itself.

2. Provide an autogen.sh which does the above 
   'make -f client.mk ${OBJDIR:-`./new_objdir.sh`}/configure'
   command.

I can do this now if you want.
I don't think we need that complexity: as noted earlier, we should just:

* check in configure, add hg hooks to make sure it is always the "right one"
* if you try to configure a srcdir build, use a default objdir

Then we don't need/shouldn't have client.mk at all, except for the tinderboxes which continue to use it. (I haven't used client.mk in years.)
We'll continue to use client.mk since it handles our Universal Mac builds and PGO builds, but we could certainly fix those in other ways and deprecate it in the future.

Comment 69

8 years ago
So do you want me to have a look at testing / developing client.mk + configure.in
patch for this or not ? 
NOTE: please have a look at bug #652807 - I think many configure affecting
environment variables, such as $CXX and $OBJDIR, are NOT being correctly
inherited by the make environment, which is why just doing :
   $ autoconf-2.13 && ( export OBJDIR=$my_objdir; ./configure )
does NOT pick up or use $OBJDIR .
Er, OBJDIR is not a variable that configure uses currently.

I think we should do what bsmedberg suggests in comment 67. We've had a bug on committing the generated configure file for a while now, we should just do it. The desired build process should just be:
"./configure && make"

The only tricky bit is that we would like configure to create an objdir instead of configuring in the srcdir, which means that we need to do something to cause "make" in the srcdir to work.
Comment on attachment 526815 [details]
Setup default OBJDIR , potentially default .mozconfig from arguments & make -f client.mk

Thanks for looking at this, but I don't think this is the approach we want.
Attachment #526815 - Flags: review?(ted.mielczarek) → review-

Updated

a year ago
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.