Closed Bug 1081682 Opened 10 years ago Closed 10 years ago

ccache "-with-ccache=/usr/bin/ccache" is not used properly any more in build (both C-C and M-C)

Categories

(Firefox Build System :: General, defect)

All
Linux
defect
Not set
major

Tracking

(Not tracked)

RESOLVED FIXED
mozilla36

People

(Reporter: ishikawa, Assigned: glandium)

References

Details

Attachments

(1 file)

As of Oct 12, 2014 during compilation of C-C TB locally, I noticed
that many files under M-C portion such as js/src (i.e., under
mozilla/js/src) are compiled WITHOUT USING CCACHE even though I have
--with-ccache=/usr/bin/ccache in my mozconfig file. 

Thus unnecessary code generation is required after timestamps of
source files are changed (but not the contents), as the result of
frequent hg qpush/qpop operations.

I have no idea of the extent of the failure to configure
to use ccache for C-C TB.
(As I write this memo, I notice mozilla/nsprpub is not configured to use
ccache as well. Hmm...)
Also, I found that M-C FF build is affected, too.

As a comparison target, I could dig up a log of C-C TB build (see the
excerpts in [1] below) circa the start of March 2014, and there
mozilla/js/src is explicitly configured with
--with-ccache=/usr/bin/ccache, while today's local build log shows
mozilla/jsr/src is not configured to use ccache at all(see [2] below.)

Becoming curious, I looked at the logs of typical compilation of TB
and FF available at https://tbpl.mozilla.org/
and found that other people's submissions
there also suffer from the lack of proper configuration to use ccache.

So this affects M-C build as well.

Someone in the know with the latest change in the BUILD infrastructure
ought to take a look into this issue.

Some investigation:

FF build using the M-C subdirectory portion of today's C-C tree
(somewhat a simulation of M-C FF build) was done, and again the use of
ccache was not configured very well (See [3].)  

I created excerpts of relevant configure lines from three local logs
for comparison purposes: from

1. -  C-C TB build log circa March 8th, 2014.
2. -  C-C TB build log Oct 12, 2014
3. -  FF build under M-C subtree of C-C source tree, Oct 12, 2014

Although they may differ slightly from the standard logs found in
https://tbpl.mozilla.org/ because I disabled a few modules for
local test purposes (or running valgrind), the 
excerpt  should give us the status of the problems.

Again, I appreciate that someone in the know can take a look at this
issue ASAP since this affect the compilation farms workload, I
suspect, and it definitely affects local developer time very much
especially if one has a large queue of hg patches and push/pop them
often.

TIA

PS: I am uploading the excerpt [1], [2], [3] as separate 
comments.
[1] Excerpt from C-C TB build log Circa March 8th, 2014 (with somewhat
older source tree):

I only excerpt the beginning part and
part where I find "configuring in DIRECTORY"  message and some lines
that follow the message.

To recap, in March 2014, ccache configuration/usage request was passed
to configure as follows: (Beware that that I disable certain
modules/subdirectories and so they don't show up in the log below.)

========================================
Target of configure     ccache information passed...
----------------------------------------
C-C as a whole		--with-ccache=/usr/bin/ccache added from my
       	 		mozconfig for TB 
----------------------------------------
./mozilla subdirectory	--with-ccache=/usr/bin/ccache added from my 
	  		mozconfig for TB
----------------------------------------
./mozilla/js/src/ctypes/libffi  CC and CCX variables: they start with
			/usr/bin/ccache already.
----------------------------------------
./ldap/sdks/c-sdk       --with-ccache_/usr/bin/ccache on the configure
                        command line
----------------------------------------
./mozilla/nsprpub	--with-ccache=/usr/bin/ccache on the configure
			command line
----------------------------------------	
./mozilla/js/src	--with-ccache=/usr/bin/ccache on the configure
			command line
========================================


Exceprts:

 ...[beginning]...
 /usr/bin/make -f client.mk -s
 Adding client.mk options from /REF-COMM-CENTRAL/comm-central/mozconfig-tb3:
	    MOZ_OBJDIR=/REF-OBJ-DIR/objdir-tb3
	    MOZ_CO_PROJECT=mail
	    BUILD_OFFICIAL=1
	    MOZILLA_OFFICIAL=1
	    BUILD_MODULES=all
	    MOZ_MAKE_FLAGS=-j4
	    MOZ_MAKE_FLAGS=-s
	    FOUND_MOZCONFIG := /REF-COMM-CENTRAL/comm-central/mozconfig-tb3
 cd /REF-OBJ-DIR/objdir-tb3
 /REF-COMM-CENTRAL/comm-central/configure
 Adding configure options from /REF-COMM-CENTRAL/comm-central/mozconfig-tb3:
	  --enable-application=mail
	  --enable-gc-zeal
	  --enable-debug-symbols=-gsplit-dwarf
	  --disable-webrtc
	  --disable-necko-wifi
	  --enable-tests
	  --disable-gstreamer
	  --enable-debug
	  --enable-shared
	  --enable-official-branding
	  --enable-crashreporter
	  --disable-libjpeg-turbo
	  --enable-default-toolkit=cairo-gtk2
	  --with-system-ply
	  --disable-necko-wifi
	  --enable-valgrind
	  --enable-optimize=-g -O2 -freorder-blocks
	  --disable-jemalloc
	  --with-ccache=/usr/bin/ccache
	  --disable-unified-compilation
 loading cache ./config.cache
 checking host system type... x86_64-unknown-linux-gnu
 checking target system type... x86_64-unknown-linux-gnu
 checking build system type... x86_64-unknown-linux-gnu

		[...]

	after the configuring top-level directctory,
	M-C portion of the subtree of C-C is now configured.
	Note that --with-ccache=/usr/bin/ccache is clearly
	visible in the configure command line

 configuring in mozilla
 running /bin/sh /REF-COMM-CENTRAL/comm-central/mozilla/configure  --enable-application=mail --enable-gc-zeal --enable-debug-symbols=-gsplit-dwarf --disable-webrtc --disable-necko-wifi --enable-tests --disable-gstreamer --enable-debug --enable-shared --enable-official-branding --enable-crashreporter --disable-libjpeg-turbo --enable-default-toolkit=cairo-gtk2 --with-system-ply --disable-necko-wifi --enable-valgrind '--enable-optimize=-g -O2 -freorder-blocks' --disable-jemalloc --with-ccache=/usr/bin/ccache --disable-unified-compilation --enable-application=../mail --with-external-source-dir=/REF-COMM-CENTRAL/comm-central --disable-official-branding --with-branding=../other-licenses/branding/thunderbird --cache-file=.././config.cache --srcdir=/REF-COMM-CENTRAL/comm-central/mozilla
 Adding configure options from /REF-COMM-CENTRAL/comm-central/mozconfig-tb3:
	  --enable-application=mail
	  --enable-gc-zeal
	  --enable-debug-symbols=-gsplit-dwarf
	  --disable-webrtc
	  --disable-necko-wifi
	  --enable-tests
	  --disable-gstreamer
	  --enable-debug
	  --enable-shared
	  --enable-official-branding
	  --enable-crashreporter
	  --disable-libjpeg-turbo
	  --enable-default-toolkit=cairo-gtk2
	  --with-system-ply
	  --disable-necko-wifi
	  --enable-valgrind
	  --enable-optimize=-g -O2 -freorder-blocks
	  --disable-jemalloc
	  --with-ccache=/usr/bin/ccache
	  --disable-unified-compilation
 loading cache .././config.cache
 checking host system type... x86_64-unknown-linux-gnu
 checking target system type... x86_64-unknown-linux-gnu

	[...]
Now I see mozilla/js/src/ctypes/libffi is configured
specially.
Here ccache is indirectly specified in the variables
|CC| and |CXX|: note that the string value of these variables
begin with "/usr/bin/ccache".

 configuring in js/src/ctypes/libffi
 running /bin/sh /REF-COMM-CENTRAL/comm-central/mozilla/js/src/ctypes/libffi/configure  --disable-shared --enable-static --disable-raw-api --enable-debug --with-pic AS='$(CC)' CC='/usr/bin/ccache /usr/bin/gcc-4.8 -fno-builtin-strlen -Wl,--gdb-index -DDEBUG=1 -DDEBUG_4GB_CHECK -DUSEHELGRIND=1' CXX='/usr/bin/ccache /usr/bin/g++-4.8 -fno-builtin-strlen -Wl,--gdb-index -DDEBUG=1 -DDEBUG_4GB_CHECK -DUSEHELGRIND=1' CPP='/usr/bin/gcc-4.8 -fno-builtin-strlen -Wl,--gdb-index -DDEBUG=1 -DDEBUG_4GB_CHECK -DUSEHELGRIND=1 -E' LD='ld' AR='ar' RANLIB='ranlib' STRIP='strip' --build=x86_64-unknown-linux-gnu --host=x86_64-unknown-linux-gnu --cache-file=/REF-OBJ-DIR/objdir-tb3/mozilla/js/src/ctypes/libffi/config.cache --srcdir=/REF-COMM-CENTRAL/comm-central/mozilla/js/src/ctypes/libffi
configure: creating cache /REF-OBJ-DIR/objdir-tb3/mozilla/js/src/ctypes/libffi/config.cache
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu

	[...]
Now comes configuring  of mozilla/../ldap/sdks/c-sdk
(I have no idea why the ordering of configure is like this. It
could be "-j4" to make may invoke the configure in
somewhat unpredictable order.
This configure uses --with-ccache_/usr/bin/ccache on
the command line.
	(As of Oct 12, 2014, I have
	ac_add_options --disable-ldap
	in my mozconfig and so this configuration step does not show
	up in Today's local build log.)


 configuring in ../ldap/sdks/c-sdk
 configuring in ../ldap/sdks/c-sdk
 running /bin/sh /REF-COMM-CENTRAL/comm-central/mozilla/../ldap/sdks/c-sdk/configure  --enable-application=mail --enable-gc-zeal --enable-debug-symbols=-gsplit-dwarf --disable-webrtc --disable-necko-wifi --enable-tests --disable-gstreamer --enable-debug --enable-shared --enable-official-branding --enable-crashreporter --disable-libjpeg-turbo --enable-default-toolkit=cairo-gtk2 --with-system-ply --disable-necko-wifi --enable-valgrind '--enable-optimize=-g -O2 -freorder-blocks' --disable-jemalloc --with-ccache=/usr/bin/ccache --disable-unified-compilation --enable-application=mail --enable-gc-zeal --enable-debug-symbols=-gsplit-dwarf --disable-webrtc --disable-necko-wifi --enable-tests --disable-gstreamer --enable-debug --enable-shared --enable-official-branding --enable-crashreporter --disable-libjpeg-turbo --enable-default-toolkit=cairo-gtk2 --with-system-ply --disable-necko-wifi --enable-valgrind '--enable-optimize=-g\ -O2\ -freorder-blocks' --disable-jemalloc --with-ccache=/usr/bin/ccache --disable-unified-compilation --enable-application=../mail --with-external-source-dir=/REF-COMM-CENTRAL/comm-central --disable-official-branding --with-branding=../other-licenses/branding/thunderbird --prefix=/REF-OBJ-DIR/objdir-tb3/mozilla/dist --with-dist-prefix=/REF-OBJ-DIR/objdir-tb3/mozilla/dist --without-nss --with-mozilla --enable-64bit --cache-file=/REF-OBJ-DIR/objdir-tb3/config.cache --srcdir=/REF-COMM-CENTRAL/comm-central/mozilla/../ldap/sdks/c-sdk
 loading cache /REF-OBJ-DIR/objdir-tb3/config.cache

 [...]
Now comes the configure in mozilla/nsprpub
this uses --with-ccache=/usr/bin/ccache on the command line.

 configuring in nsprpub
 running /bin/sh /REF-COMM-CENTRAL/comm-central/mozilla/nsprpub/configure  --enable-application=mail --enable-gc-zeal --enable-debug-symbols=-gsplit-dwarf --disable-webrtc --disable-necko-wifi --enable-tests --disable-gstreamer --enable-debug --enable-shared --enable-official-branding --enable-crashreporter --disable-libjpeg-turbo --enable-default-toolkit=cairo-gtk2 --with-system-ply --disable-necko-wifi --enable-valgrind '--enable-optimize=-g -O2 -freorder-blocks' --disable-jemalloc --with-ccache=/usr/bin/ccache --disable-unified-compilation --enable-application=mail --enable-gc-zeal --enable-debug-symbols=-gsplit-dwarf --disable-webrtc --disable-necko-wifi --enable-tests --disable-gstreamer --enable-debug --enable-shared --enable-official-branding --enable-crashreporter --disable-libjpeg-turbo --enable-default-toolkit=cairo-gtk2 --with-system-ply --disable-necko-wifi --enable-valgrind '--enable-optimize=-g\ -O2\ -freorder-blocks' --disable-jemalloc --with-ccache=/usr/bin/ccache --disable-unified-compilation --enable-application=../mail --with-external-source-dir=/REF-COMM-CENTRAL/comm-central --disable-official-branding --with-branding=../other-licenses/branding/thunderbird --with-dist-prefix=/REF-OBJ-DIR/objdir-tb3/mozilla/dist --with-mozilla --enable-debug --enable-64bit --cache-file=/REF-OBJ-DIR/objdir-tb3/config.cache --srcdir=/REF-COMM-CENTRAL/comm-central/mozilla/nsprpub
 loading cache /REF-OBJ-DIR/objdir-tb3/config.cache

Now comes the configuring of mozilla/js/src

 configuring in js/src
 running /bin/sh /REF-COMM-CENTRAL/comm-central/mozilla/js/src/configure	 --enable-application=mail --enable-gc-zeal --enable-debug-symbols=-gsplit-dwarf --disable-webrtc --disable-necko-wifi --enable-tests --disable-gstreamer --enable-debug --enable-shared --enable-official-branding --enable-crashreporter --disable-libjpeg-turbo --enable-default-toolkit=cairo-gtk2 --with-system-ply --disable-necko-wifi --enable-valgrind '--enable-optimize=-g -O2 -freorder-blocks' --disable-jemalloc --with-ccache=/usr/bin/ccache --disable-unified-compilation --enable-application=mail --enable-gc-zeal --enable-debug-symbols=-gsplit-dwarf --disable-webrtc --disable-necko-wifi --enable-tests --disable-gstreamer --enable-debug --enable-shared --enable-official-branding --enable-crashreporter --disable-libjpeg-turbo --enable-default-toolkit=cairo-gtk2 --with-system-ply --disable-necko-wifi --enable-valgrind '--enable-optimize=-g\ -O2\ -freorder-blocks' --disable-jemalloc --with-ccache=/usr/bin/ccache --disable-unified-compilation --enable-application=../mail --with-external-source-dir=/REF-COMM-CENTRAL/comm-central --disable-official-branding --with-branding=../other-licenses/branding/thunderbird --enable-threadsafe --without-intl-api --enable-ctypes --disable-shared-js --disable-exact-rooting --with-nspr-cflags='-I/REF-OBJ-DIR/objdir-tb3/mozilla/dist/include/nspr' --with-nspr-libs='-L/REF-OBJ-DIR/objdir-tb3/mozilla/dist/lib -lnspr4 -lplc4 -lplds4' --prefix=/REF-OBJ-DIR/objdir-tb3/mozilla/dist --cache-file=/REF-OBJ-DIR/objdir-tb3/config.cache --srcdir=/REF-COMM-CENTRAL/comm-central/mozilla/js/src
 loading cache /REF-OBJ-DIR/objdir-tb3/config.cache
[2] Excerpt from C-C TB build log Oct 12, 2014
    Source was refereshed within last 12 hours.

To recap, as of Oct 12, 2014, ccache configuration/usage request was
passed to configure as follows: (Beware that that I disable certain
modules/subdirectories and so they don't show up in the log below.)

========================================
Target of configure     
----------------------------------------
C-C as a whole		(Same as before [March, 2014])
       	 		--with-ccache=/usr/bin/ccache added from my
       	 		mozconfig 
----------------------------------------
./mozilla subdirectory	(Hmm, I don't see this configure line
	  		indepently any more in Oct 12 build log.)
	  		???
----------------------------------------
./mozilla/intl/icu/target  (this was not in March 2014 log?)

			   I think it is OK although there is no
			   mention of ccache since `icu' seems to
			   handle ccache on its own.
----------------------------------------
./mozilla/js/src/ctypes/libffi  (same as before [March, 2014])
			CC and CCX variables: they start with
			/usr/bin/ccache already.
----------------------------------------
./ldap/sdks/c-sdk       [this was not configured in Oct 12 build since
  		        I disabled ldap recently.]
			n/a
----------------------------------------
./mozilla/nsprpub	VERY BAD: there is no mention of ccache.
			In March log, it had
			--with-ccache=/usr/bin/ccache on the configure 
			command line
----------------------------------------	
./mozilla/js/src	VERY BAD: there is no mention of ccache (!).
			In March log, it had
			--with-ccache=/usr/bin/ccache on the configure  
			command line
========================================


...[beginning] ...
Adding client.mk options from /REF-COMM-CENTRAL/comm-central/mozconfig-tb3:
    AUTOCLOBBER=1
    MOZ_OBJDIR=/REF-OBJ-DIR/objdir-tb3
    MOZ_CO_PROJECT=mail
    BUILD_OFFICIAL=1
    MOZILLA_OFFICIAL=1
    BUILD_MODULES=all
    MOZ_MAKE_FLAGS=-j1
    FOUND_MOZCONFIG := /REF-COMM-CENTRAL/comm-central/mozconfig-tb3
make[1]: Entering directory '/REF-COMM-CENTRAL/comm-central'
cd /REF-OBJ-DIR/objdir-tb3
/REF-COMM-CENTRAL/comm-central/configure
creating cache ./config.cache
Adding configure options from /REF-COMM-CENTRAL/comm-central/mozconfig-tb3
  --enable-application=mail
  --enable-gc-zeal
  --enable-debug-symbols=-gsplit-dwarf
  --disable-ldap
  --disable-necko-wifi
  --enable-tests
  --disable-gstreamer
  --enable-debug
  --enable-shared
  --enable-official-branding
  --enable-crashreporter
  --disable-libjpeg-turbo
  --enable-default-toolkit=cairo-gtk2
  --with-system-ply
  --disable-necko-wifi
  --enable-valgrind
  --enable-optimize=-g -O2 -freorder-blocks
  --disable-jemalloc
  --with-ccache=/usr/bin/ccache
  --disable-unified-compilation
  --enable-profiling
loading cache ./config.cache
checking host system type... x86_64-unknown-linux-gnu

	 Configuring mozilla/intl/icu/target
	 I think it is OK not to pass ccache configuration info
	 since `icu' seems to handle ccache on its own.

configuring in intl/icu/target
running /bin/sh /REF-COMM-CENTRAL/comm-central/mozilla/intl/icu/source/configure --build=x86_64-unknown-linux-gnu --host=x86_64-unknown-linux-gnu --enable-debug --enable-static --disable-shared --disable-extras --disable-icuio --disable-layout --disable-tests --disable-samples --cache-file=/REF-OBJ-DIR/objdir-tb3/intl/icu/target/config.cache
configure: creating cache /REF-OBJ-DIR/objdir-tb3/intl/icu/target/config.cache


 Configuring js/src/ctypes/libffi
 OK, ccache is handled indirectly by having variables CC and
 CXX that have values that start with "/usr/bin/ccache".

configuring in js/src/ctypes/libffi
running /bin/sh /REF-COMM-CENTRAL/comm-central/mozilla/js/src/ctypes/libffi/configure --build=x86_64-unknown-linux-gnu --host=x86_64-unknown-linux-gnu --disable-shared --enable-static --disable-raw-api --enable-debug --with-pic AS=$(CC) CC=/usr/bin/ccache /usr/bin/gcc-4.9 -fno-builtin-strlen -Wl,--gdb-index -Dfdatasync=fdatasync -DDEBUG=1 -DDEBUG_4GB_CHECK -DUSEHELGRIND=1 CXX=/usr/bin/ccache /usr/bin/g++-4.9 -fno-builtin-strlen -Wl,--gdb-index -Dfdatasync=fdatasync -DDEBUG=1 -DDEBUG_4GB_CHECK -DUSEHELGRIND=1 CPP=/usr/bin/gcc-4.9 -fno-builtin-strlen -Wl,--gdb-index -Dfdatasync=fdatasync -DDEBUG=1 -DDEBUG_4GB_CHECK -DUSEHELGRIND=1 -E LD=ld AR=ar RANLIB=ranlib STRIP=strip --cache-file=/REF-OBJ-DIR/objdir-tb3/js/src/ctypes/libffi/config.cache
configure: creating cache /REF-OBJ-DIR/objdir-tb3/js/src/ctypes/libffi/config.cache

	 [...]
	 Now comes the configuring of mozilla/nsprpub:
	 Ouch. Very bad. It seems it is not configured with ccache.
	 There is no reference to ccache on the command line :-(

	 Also, DO NOTE THE WARNING after the configure command line
	 configure: WARNING: unrecognized options:
	 These may suggest that some things may not work as expected.



configuring in nsprpub
running /bin/sh /REF-COMM-CENTRAL/comm-central/mozilla/nsprpub/configure --build=x86_64-unknown-linux-gnu --enable-application=mail --enable-gc-zeal --enable-debug-symbols=-gsplit-dwarf --disable-ldap --disable-necko-wifi --enable-tests --disable-gstreamer --enable-debug --enable-shared --enable-official-branding --enable-crashreporter --disable-libjpeg-turbo --enable-default-toolkit=cairo-gtk2 --with-system-ply --disable-necko-wifi --enable-valgrind --enable-optimize=-g -O2 -freorder-blocks --disable-jemalloc --disable-unified-compilation --enable-profiling --with-external-source-dir=/REF-COMM-CENTRAL/comm-central --with-dist-prefix=/REF-OBJ-DIR/objdir-tb3/dist --with-mozilla --enable-debug --enable-64bit AS=$(CC) CC=/usr/bin/gcc-4.9 -fno-builtin-strlen -Wl,--gdb-index -Dfdatasync=fdatasync -DDEBUG=1 -DDEBUG_4GB_CHECK -DUSEHELGRIND=1 CXX=/usr/bin/g++-4.9 -fno-builtin-strlen -Wl,--gdb-index -Dfdatasync=fdatasync -DDEBUG=1 -DDEBUG_4GB_CHECK -DUSEHELGRIND=1 CPP=/usr/bin/gcc-4.9 -fno-builtin-strlen -Wl,--gdb-index -Dfdatasync=fdatasync -DDEBUG=1 -DDEBUG_4GB_CHECK -DUSEHELGRIND=1 -E LD=ld AR=ar RANLIB=ranlib STRIP=strip --cache-file=/REF-OBJ-DIR/objdir-tb3/nsprpub/config.cache
configure: WARNING: unrecognized options: --enable-application, --enable-gc-zeal, --disable-ldap, --disable-necko-wifi, --enable-tests, --disable-gstreamer, --enable-shared, --enable-official-branding, --enable-crashreporter, --disable-libjpeg-turbo, --enable-default-toolkit, --with-system-ply, --disable-necko-wifi, --enable-valgrind, --disable-jemalloc, --disable-unified-compilation, --enable-profiling, --with-external-source-dir
configure: creating cache /REF-OBJ-DIR/objdir-tb3/nsprpub/config.cache
checking build system type... x86_64-unknown-linux-gnu


configuring of mozilla/js/src
	    Note the lack of reference to ccache on the command line.
	    The lines are exactly the same as in [2]
	    Oct 12.

configuring in js/src
running /bin/sh /REF-COMM-CENTRAL/comm-central/mozilla/js/src/configure --enable-application=mail --enable-gc-zeal --enable-debug-symbols=-gsplit-dwarf --disable-ldap --disable-necko-wifi --enable-tests --disable-gstreamer --enable-debug --enable-shared --enable-official-branding --enable-crashreporter --disable-libjpeg-turbo --enable-default-toolkit=cairo-gtk2 --with-system-ply --disable-necko-wifi --enable-valgrind --enable-optimize=-g -O2 -freorder-blocks --disable-jemalloc --disable-unified-compilation --enable-profiling --with-external-source-dir=/REF-COMM-CENTRAL/comm-central --enable-threadsafe --enable-ctypes --disable-shared-js --disable-export-js --disable-gcgenerational --with-nspr-cflags=-I/REF-OBJ-DIR/objdir-tb3/dist/include/nspr --with-nspr-libs=-L/REF-OBJ-DIR/objdir-tb3/dist/lib -lnspr4 -lplc4 -lplds4 --prefix=/REF-OBJ-DIR/objdir-tb3/dist --cache-file=/REF-OBJ-DIR/objdir-tb3/config.cache
loading cache /REF-OBJ-DIR/objdir-tb3/config.cache
checking host system type... x86_64-unknown-linux-gnu

	 [...]
[3] Excerpt from the log of F-F local build using M-C subtree
   (./mozilla) of C-C source tree.

For comparison purposes, I created firefox from the 
M-C portion of C-C TB tree.

    Only relevant parts are excerpted as in [1], [2]

To recap, is ccache, which I would like to use by having
--with-ccache=/usr/bin/ccache in my mozconfig for FF, handled when FF
under M-C portion is configured?

========================================
Target of configure     ccache configuration info
----------------------------------------
./mozilla subdirectory	  OK: --with-ccache=/usr/bin/ccache added from
	  		  my mozconfig for FF browser.
----------------------------------------
./mozilla/intl/icu/target  (this was not visible in March 2014 log?)
			   It is OK although there is no mention of ccache
			   since `icu' seems to handle ccache on its
			   own.
----------------------------------------
./mozilla/js/src/ctypes/libffi  (same as before [1], [2])
			OK: CC and CCX variables: they start with
			/usr/bin/ccache already.
----------------------------------------
./ldap/sdks/c-sdk       [this was not configured in Oct 12 build.]
			n/a
----------------------------------------
./mozilla/nsprpub	VERY BAD: there is no mention of ccache.
----------------------------------------	
./mozilla/js/src	VERY BAD: there is no mention of ccache.
========================================





 ...[beginning ] ...
Adding client.mk options from /REF-COMM-CENTRAL/comm-central/mozconfig-ff:
    MOZ_CO_PROJECT=browser
    BUILD_OFFICIAL=1
    MOZILLA_OFFICIAL=1
    BUILD_MODULES=all
    MOZ_MAKE_FLAGS=-s
    MOZ_OBJDIR=/home/ishikawa/ff-objdir-tb3
    OBJDIR=/home/ishikawa/ff-objdir-tb3
    FOUND_MOZCONFIG=/REF-COMM-CENTRAL/comm-central/mozconfig-ff
cd /home/ishikawa/ff-objdir-tb3
/REF-COMM-CENTRAL/comm-central/mozilla/configure
Adding configure options from /REF-COMM-CENTRAL/comm-central/mozconfig-ff
  --enable-application=browser
  --enable-gc-zeal
  --enable-debug-symbols=-gsplit-dwarf
  --disable-necko-wifi
  --enable-tests
  --disable-gstreamer
  --enable-debug
  --enable-shared
  --enable-official-branding
  --enable-crashreporter
  --disable-libjpeg-turbo
  --enable-default-toolkit=cairo-gtk2
  --with-system-ply
  --disable-necko-wifi
  --enable-valgrind
  --enable-optimize=-g -O2 -freorder-blocks
  --disable-jemalloc
  --with-ccache=/usr/bin/ccache
  --disable-unified-compilation
creating cache ./config.cache
checking host system type... x86_64-unknown-linux-gnu

	 [...]
	 Now comes the configuration of intl/icu/target

	 Except for the configure cache location (that is placed
	 inside the different MOZ_OBJ directories specified during build/config)
	 the line look the same as in [2].

configuring in intl/icu/target
running /bin/sh /REF-COMM-CENTRAL/comm-central/mozilla/intl/icu/source/configure --build=x86_64-unknown-linux-gnu --host=x86_64-unknown-linux-gnu --enable-debug --enable-static --disable-shared --disable-extras --disable-icuio --disable-layout --disable-tests --disable-samples --cache-file=/home/ishikawa/ff-objdir-tb3/intl/icu/target/config.cache
configure: creating cache /home/ishikawa/ff-objdir-tb3/intl/icu/target/config.cache
checking for ICU version numbers... release 52.1, library 52.1, unicode version 6.3

	 [...]
	 Now comes the js/src/ctypes/libffi configuration
	 Except for the configure cache location, and
	 the following options that are specified only for TB compilation, the lines
	 are the same as in [2]: the intentionally changed compiler options  are
	 -fno-builtin-strlen
	 -Dfdatasync=fdatasync
	 -DDEBUG=1
	 (they are only in TB build.)

configuring in js/src/ctypes/libffi
running /bin/sh /REF-COMM-CENTRAL/comm-central/mozilla/js/src/ctypes/libffi/configure --build=x86_64-unknown-linux-gnu --host=x86_64-unknown-linux-gnu --disable-shared --enable-static --disable-raw-api --enable-debug --with-pic AS=$(CC) CC=/usr/bin/ccache /usr/bin/gcc-4.9 -Wl,--gdb-index -DDEBUG_4GB_CHECK -DUSEHELGRIND=1 CXX=/usr/bin/ccache /usr/bin/g++-4.9 -Wl,--gdb-index -DDEBUG_4GB_CHECK -DUSEHELGRIND=1 CPP=/usr/bin/gcc-4.9 -Wl,--gdb-index -DDEBUG_4GB_CHECK -DUSEHELGRIND=1 -E LD=ld AR=ar RANLIB=ranlib STRIP=strip --cache-file=/home/ishikawa/ff-objdir-tb3/js/src/ctypes/libffi/config.cache
configure: creating cache /home/ishikawa/ff-objdir-tb3/js/src/ctypes/libffi/config.cache
checking build system type... x86_64-unknown-linux-gnu

	 [...]
	 Now comes the configuration of	 nsprpub
	 Except for intentional changes of build options noted below
	 as well as the compiler options noted above in the libffi
	 section,
	 the lines look the same.
	 Changes of build paramters:	 
	 --enable-applicaton=mail -> browser
	 --disable-ldap (only in TB build)
	 --enabled-profiling (only in TB build)
	 binary positions: --with-dist-prefix

	 BUT DO NOTE THE WARNING after the configure command line
	 configure: WARNING: unrecognized options:
	 These may suggest that some things may not work as expected.

configuring in nsprpub
running /bin/sh /REF-COMM-CENTRAL/comm-central/mozilla/nsprpub/configure --build=x86_64-unknown-linux-gnu --enable-application=browser --enable-gc-zeal --enable-debug-symbols=-gsplit-dwarf --disable-necko-wifi --enable-tests --disable-gstreamer --enable-debug --enable-shared --enable-official-branding --enable-crashreporter --disable-libjpeg-turbo --enable-default-toolkit=cairo-gtk2 --with-system-ply --disable-necko-wifi --enable-valgrind --enable-optimize=-g -O2 -freorder-blocks --disable-jemalloc --disable-unified-compilation --with-dist-prefix=/home/ishikawa/ff-objdir-tb3/dist --with-mozilla --enable-debug --enable-64bit AS=$(CC) CC=/usr/bin/gcc-4.9 -Wl,--gdb-index -DDEBUG_4GB_CHECK -DUSEHELGRIND=1 CXX=/usr/bin/g++-4.9 -Wl,--gdb-index -DDEBUG_4GB_CHECK -DUSEHELGRIND=1 CPP=/usr/bin/gcc-4.9 -Wl,--gdb-index -DDEBUG_4GB_CHECK -DUSEHELGRIND=1 -E LD=ld AR=ar RANLIB=ranlib STRIP=strip --cache-file=/home/ishikawa/ff-objdir-tb3/nsprpub/config.cache
configure: WARNING: unrecognized options: --enable-application, --enable-gc-zeal, --disable-necko-wifi, --enable-tests, --disable-gstreamer, --enable-shared, --enable-official-branding, --enable-crashreporter, --disable-libjpeg-turbo, --enable-default-toolkit, --with-system-ply, --disable-necko-wifi, --enable-valgrind, --disable-jemalloc, --disable-unified-compilation
configure: creating cache /home/ishikawa/ff-objdir-tb3/nsprpub/config.cache
checking build system type... x86_64-unknown-linux-gnu

	 [...]  Now comes the js/src configure.  Except for the object
	 directory locations, intentionally changed options between TB
	 and FF build, the lines look the same as in [2].

configuring in js/src
running /bin/sh /REF-COMM-CENTRAL/comm-central/mozilla/js/src/configure --enable-application=browser --enable-gc-zeal --enable-debug-symbols=-gsplit-dwarf --disable-necko-wifi --enable-tests --disable-gstreamer --enable-debug --enable-shared --enable-official-branding --enable-crashreporter --disable-libjpeg-turbo --enable-default-toolkit=cairo-gtk2 --with-system-ply --disable-necko-wifi --enable-valgrind --enable-optimize=-g -O2 -freorder-blocks --disable-jemalloc --disable-unified-compilation --enable-threadsafe --enable-ctypes --disable-shared-js --disable-export-js --with-nspr-cflags=-I/home/ishikawa/ff-objdir-tb3/dist/include/nspr --with-nspr-libs=-L/home/ishikawa/ff-objdir-tb3/dist/lib -lnspr4 -lplc4 -lplds4 --prefix=/home/ishikawa/ff-objdir-tb3/dist --cache-file=/home/ishikawa/ff-objdir-tb3/config.cache
loading cache /home/ishikawa/ff-objdir-tb3/config.cache
--with-ccache is not the whole story to enable ccache. Check the actual command lines used to build.
It however looks like it is truly failing to pass ccache down on tbpl mac builds. Can you try --with-compiler-wrapper="/usr/bin/ccache"?
(In reply to Mike Hommey [:glandium] from comment #4)
> --with-ccache is not the whole story to enable ccache. Check the actual
> command lines used to build.

I have not changed my command lines at all since March (regarding ccache as far as I can tell) That is why I reported the issue.

(In reply to Mike Hommey [:glandium] from comment #5)
> It however looks like it is truly failing to pass ccache down on tbpl mac
> builds. Can you try --with-compiler-wrapper="/usr/bin/ccache"?

Will try by putting this --with-compiler-wrapper=... in my mozconfig.
[I suspect mac builds as well as x86_64 builds and others as well if my analysis is correct.]

TIA
I am running configure, but please note

https://developer.mozilla.org/ja/docs/ccache

--- begin quote ---
Configuring Mozilla Builds

To configure Mozilla builds to use ccache, you'll need to configure some make and configure options. In your .mozconfig, add the following:

 ac_add_options --with-ccache=/usr/local/bin/ccache

Be sure to specify the appropriate path to ccache on your system!

Now, run a build and verify the cache is being utilized:
--- end quote ---

The above is what I have been using for the last 2-3 yeas.
Maybe the above needs rewriting.
A progress:
I have configured C-C TB with --with-compiler-wrapper=/usr/bin/ccache,
and saw the compilation using ccache of jsmath.cpp in ccache logfile now (!) 
Maybe with this option in mozconfig, ccache compilation can be enabled.

But be carefull. I would still like those who know the build infrastructure to figure out what happened to --with-ccache=...

This is because I am not sure if ALL the files that are supposed to be compiled with
ccache are compiled in that manner (even with --with-compiler-wrapper=...),
AND more importantly
if there are files that were excluded from ccache for a reason (I can't figure out if there are such valid reasons, though) are still compiled without ccache.

I know there were files compiled WITHOUT ccache before.
For example, some utility routines and such that were compiled near the beginning of
fresh new build were compiled without ccache.

In any case, maybe someone needs to rewrite
https://developer.mozilla.org/ja/docs/ccache
unless "--with-ccache=..." is not honored any more in the new build infrastructure.

TIA

PS: Despite my passing --disable-unified-compilation, files under some directories
seem to be compiled in unified-compilation fashion.
(Individual files in these unified compilation need to be factored in
when we count/verify if all the files that can benefit from ccache 
are indeed compiled with ccache.)
 
UnifiedProtcols0.o ... UnifiedProtocols17.o: these seem to be
IPC which may not change often so unified build is justifiable.
UnifiedBindings0.o ... UnifiedBindings20.o: these seem to be
dom/binding  and so may not change often and unified build is justifiable.

PPS:
Oh, it might be instructive to check if jsmath.o is in ccache log of compile farm machines of mozilla. I suspect it and other files under js/src are not in the log (if my analysis is correct.). If it is the case, these files were compiled every time a compilation request was received even if they [and included header files] are not touched and could use ccache. Hmm...
This just points to how to fix the problem: implement bug 867750 comment 2. IOW, have --with-ccache essentially set --with-compile-wrapper.
FWIW, this was "broken" by bug 1067248
Blocks: 1067248
Also, avoid removing --with-ccache from all subconfigure calls. Only remove it
from NSPR's.

Technically, only the latter is required, but while I was here...
Attachment #8504439 - Flags: review?(mshal)
Assignee: nobody → mh+mozilla
Status: NEW → ASSIGNED
(In reply to Mike Hommey [:glandium] from comment #11)
> Created attachment 8504439 [details] [diff] [review]
> Make --with-ccache use --with-compiler-wrapper under the hood
> 
> Also, avoid removing --with-ccache from all subconfigure calls. Only remove
> it
> from NSPR's.
> 
> Technically, only the latter is required, but while I was here...

I don't understand the make magic used in mozilla source tree very well, but
here is what found out today, and I hope the above patch takes care of the issue.

Basically, yesterday I modified mozconfig for C-C TB to include --with-compiler-wrapper=/usr/bin/ccache 
in addition to the original --with-enable-ccache=/usr/bin/ccache.

Since the names of jsobj.c and other files that had not been recorded in the logfile of ccache
began to appear in the log, I thought the change was OK.

But today,  I analyzed the logfile today to see cache hits/misses statistics and other
anomalies, and and found so many instances of 
"Result: multiple source files" warning messages.

This warning essentially means that /usr/bin/ccache was given multiple source files to the compiler it
handles and 
it does not know how to handle such multiple source files, it just runs the original compiler instead.
So many of these warnings.

So I investigated:
It turns out that, having both --enable-ccache=/usr/bin/ccache AND
--with-compiler-wrapper=/usr/bin/ccache in mozconfig for C-C TB can result in compilation command line that
start with two instances of ccache binary as follows for many files (but not all of them !).

/usr/bin/ccache /usr/bin/ccache /usr/bin/g++-4.9 options ... source file

Note the repetition of /usr/bin/ccache twice in a row at the start.

This command line is given up by the first /usr/bin/ccache since the case of having more than
one source files on the command line does not work.
The first instance of /usr/bin/ccache thinks
/usr/bin/g++-4.9 is the FIRST source file given on the command line to the "compiler", the second instance of /usr/bin/ccache as above. Counting this FIRST source file AND the real source file at the end 
makes the first instance of /usr/bin/ccache to think that the compiler is given  two source files!

But this means then, the "REAL" compiler command line as seen
from the view point of the first instance of /usr/bin/ccache is invoked from within the first
ccache: The REAL compiler line for the first /usr/bin/ccache is as follows:

/usr/bin/ccache /usr/bin/g++-4.9 options ... real source file

Now the first instance of /usr/bin/ccache is removed and
the second instance of /usr/bin/ccache becomes the program name on the command line.
By sheer luck, this turns out to be the proper /usr/bin/ccache invocation to 
compile a single source file.

So eventually the object is produced as expected by the invocation of the correct command line, but
there are a few extra steps and disturbing log lines before
/usr/bin/ccache is invoked properly.

I don't know whether the patch proposed above takes care of the duplicated
/usr/bin/ccache issue, but I hope so. [ Reading  bug 867750 and
bug 1067248 makes me think someone knows for sure :-) 
The use of [[ ... ]] in a regular expression was a novel usage example. That is surely one way
of taking care of the quoting issue nicely, I think.

I *think* this duplicated /usr/bin/ccache issue affected the files under M-C
except for js/src, and a few other modules/subdirectories that are configured
explicitly as in the table of comment 2:
The files that seem to have duplicate /usr/bin/ccache in the command line are generally speaking
in the following category. Other directory specially handled do not have the duped /usr/bin/ccache.:

> ./mozilla subdirectory	(Hmm, I don't see this configure lin?
> 
>	  		indepently any more in Oct 12 build log.)
>	  		???


Anyway,I think we need to check the log of ccache until this patch lands and thereafter for some time
to make sure everything is in order.

TIA
Attachment #8504439 - Flags: review?(mshal) → review+
https://hg.mozilla.org/mozilla-central/rev/f7e31a757487
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla36
Depends on: 1083487
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.