Closed Bug 559964 (gcc4.5) Opened 12 years ago Closed 11 years ago

switch to gcc-4.5 and enable PGO for linux builds

Categories

(Release Engineering :: General, defect, P4)

All
Linux
defect

Tracking

(firefox6-, blocking2.0 -)

RESOLVED FIXED
Tracking Status
firefox6 - ---
blocking2.0 --- -

People

(Reporter: taras.mozilla, Assigned: glandium)

References

(Blocks 1 open bug)

Details

(Whiteboard: [puppet][buildslaves] nominated in comment 163)

Attachments

(7 files, 29 obsolete files)

5.77 KB, patch
Details | Diff | Splinter Review
7.80 KB, patch
bear
: review+
Details | Diff | Splinter Review
2.32 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
1.71 KB, patch
Details | Diff | Splinter Review
12.88 KB, patch
catlee
: review+
Details | Diff | Splinter Review
21.83 KB, patch
catlee
: review+
Details | Diff | Splinter Review
3.26 KB, text/plain
ted
: review+
Details
PGO provides a huge(30%) startup improvement on Linux. It's the biggest easy win available right now.
A few questions:

- Any special instructions for compiling gcc 4.4?  We had some patches for gcc4.3 when we compiled it.

- Which branches should this be enabled for?

- Any known compatibility issues between 4.3 and 4.4?  ABI changes?  libstdc++ changes?-

- What are the impacts of users of the binaries?  Will they need the gcc 4.4 and associated libstdc++ libraries installed?
(In reply to comment #1)
> A few questions:
> 
> - Any special instructions for compiling gcc 4.4?  We had some patches for
> gcc4.3 when we compiled it.

No such patches for 4.4.
> 
> - Which branches should this be enabled for?

trunk.

> 
> - Any known compatibility issues between 4.3 and 4.4?  ABI changes?  libstdc++
> changes?-

Libstdc++ is gcc-dependent. So users will have to be using a distributing with gcc4.4

> - What are the impacts of users of the binaries?  Will they need the gcc 4.4
> and associated libstdc++ libraries installed?

they need the 4.4 libstdc++
I don't think we actually *do* require a newer version of libstdc++: it's still libstdc++.so.6, and the ABI hasn't really changed since 4.2. If we don't run on Ubuntu 08.04 LTS, that would be a problem.
The only change I'm aware of in 4.4 is the va_list ABI on ARM, but mozilla has no externally available API using it, I believe.

Please note, as I said in bug 418866, and since mozilla support for linux x86-64 is being added, that gcc 4.4 fails to generate working coverage builds (1st pass) on x86-64, which makes it impossible to build pgo. I haven't tried with 4.5 yet.
(In reply to comment #3)
> I don't think we actually *do* require a newer version of libstdc++: it's still
> libstdc++.so.6, and the ABI hasn't really changed since 4.2. If we don't run on
> Ubuntu 08.04 LTS, that would be a problem.

You are right. I just confirmed 4.4-produced binaries run great on 4.2-based ubuntu LTS. So there are no abi concerns from me anymore. 

So how soon can we get a linux PGO tinderbox?
(In reply to comment #5)
> (In reply to comment #3)
> > I don't think we actually *do* require a newer version of libstdc++: it's still
> > libstdc++.so.6, and the ABI hasn't really changed since 4.2. If we don't run on
> > Ubuntu 08.04 LTS, that would be a problem.
> 
> You are right. I just confirmed 4.4-produced binaries run great on 4.2-based
> ubuntu LTS. So there are no abi concerns from me anymore. 
> 
> So how soon can we get a linux PGO tinderbox?

Will this replace the existing linux build, or be in addition to it?

If you want to change how the existing build is done, I think this merits some newsgroup discussion first.
(In reply to comment #6)
> Will this replace the existing linux build, or be in addition to it?
> 
> If you want to change how the existing build is done, I think this merits some
> newsgroup discussion first.

I would like to replace the current build as it would be insufferably slow compared to a PGO one. I sent a notice to http://groups.google.com/group/mozilla.dev.platform/browse_thread/thread/4037844abc6ef1f5#

I don't see why anyone would disagree with a faster firefox, but lets proceed on this bug in parallel to the discussion. In the worst case we can settle for an additional build.
(In reply to comment #7)
> I would like to replace the current build as it would be insufferably slow
> compared to a PGO one.

FWIW, I didn't get any really noticeable improvement with pgo builds, even on startup time, though i haven't formally verified the latter, and I have a SSD, so i cheat for startup. But that would suggest that there is no cpu-bound improvement in your faster startups, which in turn suggests there might be better ways to get these improvements (such as your reordering work).

But benchmarks such as sunspider didn't end up running significantly faster even when running it during the profiling phase. Is it worth doubling the build time ?
(In reply to comment #8)
> (In reply to comment #7)
> > I would like to replace the current build as it would be insufferably slow
> > compared to a PGO one.
> 
> FWIW, I didn't get any really noticeable improvement with pgo builds, even on
> startup time, though i haven't formally verified the latter, and I have a SSD,
> so i cheat for startup. But that would suggest that there is no cpu-bound
> improvement in your faster startups, which in turn suggests there might be
> better ways to get these improvements (such as your reordering work).

Actually the good numbers produced by pgo in http://blog.mozilla.com/tglek/2010/04/12/squeezing-every-last-bit-of-performance-out-of-the-linux-toolchain/ have little to do with cpu time. -freorder-blocks-and-partition option changed the layout of the binary for the better. Any minor cpu-time benefits are just icing on my startup cake.
 
> 
> But benchmarks such as sunspider didn't end up running significantly faster
> even when running it during the profiling phase. Is it worth doubling the build
> time ?

For a 30-40% improvement in startup time, yes.
(In reply to comment #9)
> For a 30-40% improvement in startup time, yes.

Even when you can get the same improvement with tools that don't require to double the build time ?
With most of our Linux builds happening on very fast hardware I don't think we should overly concern ourselves with doubling the build time. The last mozilla-central nightly on Linux took 15 minutes to compile. Even at 30 minutes to compile, Linux is not going to be our long-pole platform.
Depends on: 561236
After Benjamin grumbled about the 4.4 switch, I figured out what needs to be done to do PGO builds on x86/x86_64 with gcc 4.5 such that the resulting binaries run fine on older distributions.
So lets get GCC 4.5.0 installed and pgo turned on.
Summary: Install GCC 4.4 on linux VMs to enable PGO on Linux → Install GCC 4.5 on linux VMs to enable PGO on Linux
From Taras on IRC:

- Build with same switches as we did gcc 4.3.3: https://wiki.mozilla.org/ReferencePlatforms/Linux-CentOS-5.0#GCC_4.3.3

- No need to keep gcc headers around; just need binaries and libs (whatever 'make install' creates)
You really want to switch our builds to a compiler that was released not quite two weeks ago? Brave!
(In reply to comment #14)
> You really want to switch our builds to a compiler that was released not quite
> two weeks ago? Brave!

Yup. This way we avoid another compiler change in the near future. This way we have a much better chance of having any fresh issues fixed quickly by gcc guys if they arise.
Assignee: nobody → rail
Status: NEW → ASSIGNED
Priority: -- → P2
Hardware: x86 → All
Attached file compile-gcc-4.5.0.sh (obsolete) —
Attaching the current GCC 4.5.0 compile script which prepares Puppet friendly tarball.

GCC 4.5 requires new version of gmp, mpfr and mpc libraries. To avoid any possible conflicts with te current toolchain I installed all of tools with --prefix=/tools/gcc-4.5.0 prefix and add wrappers for gcc/g++ with injected LD_LIBRARY_PATH. Any ideas how to avoid this hack without touching /etc/ld.so.conf and setting LD_LIBRARY_PATH?

Normal build (using wrappers) runs fine, but without the new libstdc++.so.6.0.14 linked to libstdc++.so.6.0 fails with:

./firefox-bin: /usr/lib32/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by ./libxul.so)

Taras, is there any ETA for updated profiledbuild target for linux? IIRC it hasn't landed yet.
(In reply to comment #16)
> Created an attachment (id=443099) [details]
> compile-gcc-4.5.0.sh
> 
> Attaching the current GCC 4.5.0 compile script which prepares Puppet friendly
> tarball.
> 
> GCC 4.5 requires new version of gmp, mpfr and mpc libraries. To avoid any
> possible conflicts with te current toolchain I installed all of tools with
> --prefix=/tools/gcc-4.5.0 prefix and add wrappers for gcc/g++ with injected
> LD_LIBRARY_PATH. Any ideas how to avoid this hack without touching
> /etc/ld.so.conf and setting LD_LIBRARY_PATH?
> 
> Normal build (using wrappers) runs fine, but without the new
> libstdc++.so.6.0.14 linked to libstdc++.so.6.0 fails with:
> 
> ./firefox-bin: /usr/lib32/libstdc++.so.6: version `GLIBCXX_3.4.14' not found
> (required by ./libxul.so)

Yeah, that's because of the new libstd++ dependency. Bug 561236 aims to address this. I need to investigate alternative workarounds. Will try to wrap that up this week.

> 
> Taras, is there any ETA for updated profiledbuild target for linux? IIRC it
> hasn't landed yet.

Ted, waiting for your review on this in bug 418866.
Attached file compile-gcc-4.5.0.sh (obsolete) —
Create tarballs ready for Puppet.
The binaries can be downloaded from http://people.mozilla.org/~raliiev/puppet/
Attachment #443099 - Attachment is obsolete: true
Attachment #443309 - Flags: review?(catlee)
Attached patch GCC 4.5.0 Puppet manifests (obsolete) — Splinter Review
Install gcc450 as a secondary compiler for Centos platform. No symlink required.
Attachment #443310 - Flags: review?(bhearsum)
Attached patch Combined patch for profiledbuild (obsolete) — Splinter Review
Putting the combined patch from bugs mentioned by Taras just for the history. :)

I "tested" (just ran it :) ) the profiled build on i386 arch only (will try on amd64 these days). Worked fine on Ubuntu 8.04 (i386) and 10.04 (amd64 with lib32) with default libstdc++. Visually starts faster.

I used default nightly .mozconfig with the following changes:

CC="ccache /tools/gcc-4.5.0/bin/gcc"
CXX="ccache /tools/gcc-4.5.0/bin/g++"
mk_add_options  PROFILE_GEN_SCRIPT='$(PYTHON) @TOPSRCDIR@/_profile/pgo/profileserver.py'
amd64 build failed with the following error:

/tools/gcc-4.5.0/bin/g++ -o xptcstubs_x86_64_linux.o -c -I../../../../../../dist/stl_wrappers -I../../../../../../dist/system_wrappers -include ../../../../../../config/gcc_hidden.h -DMOZILLA_INTERNAL_API -DOSTYPE=\"Linux2.6.18-53.1.14\" -DOSARCH=Linux -DEXPORT_XPTC_API -D_IMPL_NS_COM -I./../.. -I./../../../../xptinfo/src  -I. -I. -I../../../../../../dist/include -I../../../../../../dist/include/nsprpub  -I/builds/slave/mozilla-central-linux64-nightly/build/dist/include/nspr -I/builds/slave/mozilla-central-linux64-nightly/build/dist/include/nss       -fPIC  -fno-rtti -fno-exceptions -Wall -Wpointer-arith -Woverloaded-virtual -Wsynth -Wno-ctor-dtor-privacy -Wno-non-virtual-dtor -Wcast-align -Wno-invalid-offsetof -Wno-variadic-macros -Werror=return-type -Wno-long-long -pedantic -gdwarf-2 -fno-strict-aliasing -fshort-wchar -pthread -pipe  -DNDEBUG -DTRIMMED -gdwarf-2 -fprofile-use -Wcoverage-mismatch -freorder-blocks-and-partition -fprofile-correction  -Os -freorder-blocks -fomit-frame-pointer    -DMOZILLA_CLIENT -include ../../../../../../mozilla-config.h -Wp,-MD,.deps/xptcstubs_x86_64_linux.pp xptcstubs_x86_64_linux.cpp
{standard input}: Assembler messages:
{standard input}:7765: Error: can't resolve `.text.unlikely' {.text.unlikely section} - `.LCOLDB0' {.text section}
{standard input}:7927: Error: can't resolve `.text.unlikely' {.text.unlikely section} - `.LCOLDB0' {.text section}
Attachment #443309 - Flags: review?(catlee) → review+
GCC 4.5.0 tarballs for linux and linux64 platforms are in /N/centos5/dist.
(In reply to comment #16)
> GCC 4.5 requires new version of gmp, mpfr and mpc libraries. To avoid any
> possible conflicts with te current toolchain I installed all of tools with
> --prefix=/tools/gcc-4.5.0 prefix and add wrappers for gcc/g++ with injected
> LD_LIBRARY_PATH. Any ideas how to avoid this hack without touching
> /etc/ld.so.conf and setting LD_LIBRARY_PATH?

No. Another unfun alternative is to make a mozg++ wrapper script with
LD_LIBRARY_PATH=... /path/to/g++ $@

> 
> Normal build (using wrappers) runs fine, but without the new
> libstdc++.so.6.0.14 linked to libstdc++.so.6.0 fails with:
> 
> ./firefox-bin: /usr/lib32/libstdc++.so.6: version `GLIBCXX_3.4.14' not found
> (required by ./libxul.so)

Apply the gcc patch in bug 561236 to fix this.


(In reply to comment #21)
> {standard input}: Assembler messages:
> {standard input}:7765: Error: can't resolve `.text.unlikely' {.text.unlikely
> section} - `.LCOLDB0' {.text section}
> {standard input}:7927: Error: can't resolve `.text.unlikely' {.text.unlikely
> section} - `.LCOLDB0' {.text section}

That's weird. I don't have a 64bit centos vm here. This *might* be due to having an ancient linker. Newer binutils might fix this.
Adding LDFLAGS -static-libstdc++ to LDFLAGS seems to work fine(as an alternative to patching gcc). Infact, I don't even see any significant increase in size.
Attached file compile-gcc-4.5.0.sh (obsolete) —
(In reply to comment #24)
> Adding LDFLAGS -static-libstdc++ to LDFLAGS seems to work fine(as an
> alternative to patching gcc). Infact, I don't even see any significant increase
> in size.

Worked fine for i386. Increase in size is 417380 (11702238 vs 12119618).

I removed all vector/list changes from the previous "Combined patch for profiledbuild" (next one) for this build.

Snippet from .mozcofig:
export LDFLAGS="-static-libstdc++"

(In reply to comment #23)
> (In reply to comment #16)
> > GCC 4.5 requires new version of gmp, mpfr and mpc libraries. To avoid any
> > possible conflicts with te current toolchain I installed all of tools with
> > --prefix=/tools/gcc-4.5.0 prefix and add wrappers for gcc/g++ with injected
> > LD_LIBRARY_PATH. Any ideas how to avoid this hack without touching
> > /etc/ld.so.conf and setting LD_LIBRARY_PATH?

Setting LDFLAGS and BOOT_LDFLAGS to "-L/tools/gcc-4.5.0/lib -Wl,-rpath,/tools/gcc-4.5.0/lib" fixed the problem. No need to set LD_LIBRARY_PATH now. Tarballs for linux and linux64 put on /N/ and deployed using Pupet in staging.

> That's weird. I don't have a 64bit centos vm here. This *might* be due to
> having an ancient linker. Newer binutils might fix this.

i386 is more or less ready, let's try to hack this one. :)
Attachment #443309 - Attachment is obsolete: true
Attached patch Combined patch for profiledbuild (obsolete) — Splinter Review
Attachment #443311 - Attachment is obsolete: true
Attached patch buildbot-configs changes (obsolete) — Splinter Review
Use GCC 4.5.5 as default compiler for mozilla-central linux (32bit only) builds.
Adjust PROFILE_GEN_SCRIPT for nightly builds and add it for the release builds.

profiledbuild is not turned on by default, will be enabled later by tuning config.py.
Attachment #443721 - Flags: review?(bhearsum)
Attached patch buildbot-configs changes (obsolete) — Splinter Review
Added LDFLAGS for statical linking
Attachment #443721 - Attachment is obsolete: true
Attachment #443736 - Flags: review?(bhearsum)
Attachment #443721 - Flags: review?(bhearsum)
Comment on attachment 443736 [details] [diff] [review]
buildbot-configs changes

I think we should convert the addonsmgr, places, electrolysis, and tracemonkey repos over too, since they're derived from mozilla-central.

Beware of the symlinks!
Attachment #443310 - Flags: review?(bhearsum) → review+
Attached patch buildbot-configs changes (obsolete) — Splinter Review
s,@TOPSRCDIR@/build,@MOZ_OBJDIR@,
Attachment #443736 - Attachment is obsolete: true
Attachment #443967 - Flags: review?(bhearsum)
Attachment #443736 - Flags: review?(bhearsum)
Depends on: 564381
Found a small problem with above.

Need to add PIC flags BOOT_CFLAGS=-fPIC and BOOT_LDFLAGS=-fPIC in order to produce useful binary with -static-libstdc++.

Otherwise libstdc++ causes libxul to contain text relocations which piss off selinux and prelink.
Comment on attachment 443599 [details] [diff] [review]
Combined patch for profiledbuild

landed to mozilla-central
Attachment #443599 - Attachment is obsolete: true
Summary: Install GCC 4.5 on linux VMs to enable PGO on Linux → Install GCC 4.5 on linux buildslaes to enable PGO on Linux
Whiteboard: [puppet][buildslaves]
Summary: Install GCC 4.5 on linux buildslaes to enable PGO on Linux → Install GCC 4.5 on linux buildslaves to enable PGO on Linux
Attached patch compile-gcc-4.5.0.sh (obsolete) — Splinter Review
GCC 4.5.0 build script with updated flags.

Taras, could you take a look and check if the parameters and flags are OK.

The builds using this compiler pass all unittests (profiledbuild and debug build) in staging. If there is no stopper I'd like to deploy the current compiler build to the build slaves and enable it and profiledbuild for mozilla-central.
Attachment #443596 - Attachment is obsolete: true
Attachment #445094 - Flags: review?(tglek)
Comment on attachment 445094 [details] [diff] [review]
compile-gcc-4.5.0.sh

>mkdir -p $pkg-$ver/gcc-obj-release
>export CFLAGS="-fPIC"
>export BOOT_CFLAGS="$CFLAGS"
>export LDFLAGS="-L$PREFIX/lib -Wl,-rpath,$PREFIX/lib -fPIC"
>export BOOT_LDFLAGS="$LDFLAGS"
>cd $pkg-$ver/gcc-obj-release
>../configure --prefix=$PREFIX --enable-__cxa_atexit --enable-languages=c,c++ \
>    --with-gmp=$PREFIX \
>    --with-mpfr=$PREFIX \
>    --with-mpc=$PREFIX
>make BOOT_LDFLAGS="$BOOT_LDFLAGS" LDFLAGS="$LDFLAGS" \
>    CFLAGS="$CFLAGS" BOOT_CFLAGS="$BOOT_CFLAGS" bootstrap
>make BOOT_LDFLAGS="$BOOT_LDFLAGS" LDFLAGS="$LDFLAGS" bootstrap

Don't need to pass these arguments to make, but it probably doesn't hurt. 

Looks good other than that.
Attachment #445094 - Flags: review?(tglek) → review+
Attached patch compile-gcc-4.5.0.sh (obsolete) — Splinter Review
Bump version (moz1) to make puppet happy.

Keeping r+ by Taras
Attachment #445094 - Attachment is obsolete: true
Attachment #445161 - Flags: review+
Comment on attachment 443310 [details] [diff] [review]
GCC 4.5.0 Puppet manifests

http://hg.mozilla.org/build/puppet-manifests/rev/367074f7877a
64bit part ignored
Attachment #443310 - Flags: checked-in+
Attached patch GCC 4.5.0_moz1 Puppet manifests (obsolete) — Splinter Review
Bump version, obsolete previous build, maintain gcc-4.5 symlink
Attachment #445167 - Flags: review?(bhearsum)
Comment on attachment 445161 [details] [diff] [review]
compile-gcc-4.5.0.sh


>mkdir -p $pkg-$ver/gcc-obj-release
>export CFLAGS="-fPIC"
>export BOOT_CFLAGS="$CFLAGS"
>export LDFLAGS="-L$PREFIX/lib -Wl,-rpath,$PREFIX/lib -fPIC"
>export BOOT_LDFLAGS="$LDFLAGS"
>cd $pkg-$ver/gcc-obj-release
>../configure --prefix=$PREFIX \
>    --enable-__cxa_atexit --enable-languages=c,c++ \
>    --with-gmp=$PREFIX \
>    --with-mpfr=$PREFIX \
>    --with-mpc=$PREFIX
>make BOOT_LDFLAGS="$BOOT_LDFLAGS" LDFLAGS="$LDFLAGS" \
>    CFLAGS="$CFLAGS" BOOT_CFLAGS="$BOOT_CFLAGS" bootstrap

Instead of passing -fPIC above, the correct way is to add --with-pic to configure options. This fixes the full gcc build here.
(In reply to comment #38)

> 
> Instead of passing -fPIC above, the correct way is to add --with-pic to
> configure options. This fixes the full gcc build here.

Scratch that. --with-pic produces a weird libstdc++. export CXXFLAGS=-fPIC is the way to go.
4.5 fixes a build failure with 4.3 in  bug 415563
taras: do you have the details of the gcc bug in bug 415563
that is fixed in gcc 4.5?  I'd like to know under what conditions
that bug affects us, and whether the fix has been backported
to a gcc 4.3.x or 4.4.x release.  This info will enable me
to implement a workaround in NSPR.  Thanks.
(In reply to comment #41)
> taras: do you have the details of the gcc bug in bug 415563
> that is fixed in gcc 4.5?  I'd like to know under what conditions
> that bug affects us, and whether the fix has been backported
> to a gcc 4.3.x or 4.4.x release.  This info will enable me
> to implement a workaround in NSPR.  Thanks.

No, all I did was try two different gcc versions(4.3.3 and 4.5). The stackoverflow link that blassey posted in bug 415563 indicates that -march=i486 on relevant files is a workaround.
Taras, sorry for the delay, I'm a bit busy by other tasks right now. :(

Would you mind if I ask you to try compiling gcc and test it on one of the our builders? I could ask IT to clone one.
(In reply to comment #43)
> Taras, sorry for the delay, I'm a bit busy by other tasks right now. :(
> 
> Would you mind if I ask you to try compiling gcc and test it on one of the our
> builders? I could ask IT to clone one.

I've been testing in the buildvm, CXXFLAGS=-fPIC is all we need to get this going. Just need to confirm/deploy in production now
Build fails with CXXFLAGS=-fPIC set (for GCC). Tried to use the current version of binutils (2.20) but the result is the same.

/tools/gcc-4.5/bin/g++ -o xptcinvoke_gcc_x86_unix.o -c -I../../../../../../dist/stl_wrappers -I../../../../../../dist/system_wrappers -include /builds/slave/linux_build/build/config/gcc_hidden.h -DMOZILLA_INTERNAL_API -DOSTYPE=\"Linux2.6.18-53.1.19\" -DOSARCH=Linux -DEXPORT_XPTC_API -D_IMPL_NS_COM -I/builds/slave/linux_build/build/xpcom/reflect/xptcall/src/md/unix/../.. -I/builds/slave/linux_build/build/xpcom/reflect/xptcall/src/md/unix/../../../../xptinfo/src  -I/builds/slave/linux_build/build/xpcom/reflect/xptcall/src/md/unix -I. -I../../../../../../dist/include -I../../../../../../dist/include/nsprpub  -I/builds/slave/linux_build/build/obj-firefox/dist/include/nspr -I/builds/slave/linux_build/build/obj-firefox/dist/include/nss       -fPIC  -fno-rtti -fno-exceptions -Wall -Wpointer-arith -Woverloaded-virtual -Wsynth -Wno-ctor-dtor-privacy -Wno-non-virtual-dtor -Wcast-align -Wno-invalid-offsetof -Wno-variadic-macros -Werror=return-type -Wno-long-long -pedantic -gdwarf-2 -fno-strict-aliasing -fshort-wchar -pthread -pipe  -DNDEBUG -DTRIMMED -gdwarf-2 -fprofile-use -fprofile-correction -Wcoverage-mismatch -freorder-blocks-and-partition -Os -freorder-blocks -fomit-frame-pointer    -DMOZILLA_CLIENT -include ../../../../../../mozilla-config.h -Wp,-MD,.deps/xptcinvoke_gcc_x86_unix.pp /builds/slave/linux_build/build/xpcom/reflect/xptcall/src/md/unix/xptcinvoke_gcc_x86_unix.cpp
{standard input}: Assembler messages:
{standard input}:200: Error: can't resolve `.text.unlikely' {.text.unlikely section} - `.LCOLDB0' {.text section}
make[10]: *** [xptcinvoke_gcc_x86_unix.o] Error 1
Depends on: 569137
Sorry forgot about that, I had a workaround patch in my queue for this. I filed bug 569137 to fix this.
That's not a binutils problem, it's something to do with how gcc layouts the sections. I'll file a gcc bug eventually.
A bit ugly spec:
* I included GMP, MPFR and MPC libraries into the package, so I need to install them to the target (/tools/gcc...) before gcc build. The libraries could be packaged separately, but we aren't going to use them for other packages.
* %install just copies files from --prefix to rpm's "right" directory to make rpmbuild happy

A binary produced by this package got OK from Taras.
Attachment #445161 - Attachment is obsolete: true
Attachment #448687 - Flags: review?(bhearsum)
Attached patch GCC 4.5.0 Puppet manifests (obsolete) — Splinter Review
The manifest also removes the previous buggy gcc-4.5.0 version.
Attachment #443310 - Attachment is obsolete: true
Attachment #445167 - Attachment is obsolete: true
Attachment #448692 - Flags: review?(bhearsum)
Attachment #445167 - Flags: review?(bhearsum)
Attachment #448692 - Flags: review?(bhearsum) → review?(bear)
Comment on attachment 448692 [details] [diff] [review]
GCC 4.5.0 Puppet manifests

Bear, can you review this?
Comment on attachment 448687 [details] [diff] [review]
GCC 4.5.0 spec (including GMP, MPFR, MPC libraries)

Taras tells me the build instructions look good.

Looks OK to me, too.
Attachment #448687 - Flags: review?(bhearsum) → review+
Comment on attachment 443967 [details] [diff] [review]
buildbot-configs changes

I think you need to use /tools/gcc-4.5 now, based on my reading of the spec and puppet manifests.
Attachment #443967 - Flags: review?(bhearsum) → review-
Attachment #448692 - Flags: review?(bear) → review+
Is this ready for testing on staging - do you have the rpm ready?
No longer blocks: android
* Fixed --prefix (/tools/gcc-4.5) to eliminate Puppet symlink manipulation.
* Used 0moz1 release instead of moz1
Attachment #448687 - Attachment is obsolete: true
Attachment #449699 - Flags: review?(bear)
Attached patch GCC 4.5.0 Puppet manifests (obsolete) — Splinter Review
Use install_rpm
Attachment #448692 - Attachment is obsolete: true
Attachment #449703 - Flags: review?(bhearsum)
Attachment #449699 - Flags: review?(bear) → review+
Comment on attachment 449699 [details] [diff] [review]
GCC 4.5.0 spec (including GMP, MPFR, MPC libraries)

just realized your asking me to review the spec but I was seeing the puppet changes
Attachment #449699 - Flags: review+ → review-
Oops. Here is the spec.
Attachment #449699 - Attachment is obsolete: true
Attachment #449710 - Flags: review?(bear)
Attachment #449710 - Flags: review?(bear) → review+
Attachment #449703 - Flags: review?(bhearsum) → review+
Comment on attachment 449703 [details] [diff] [review]
GCC 4.5.0 Puppet manifests

Last NFS remnant is being removed here, yay!
Comment on attachment 449703 [details] [diff] [review]
GCC 4.5.0 Puppet manifests

changeset:   177:d7b96acef7ad
Attachment #449703 - Flags: checked-in+
Comment on attachment 449710 [details]
GCC 4.5.0 spec (including GMP, MPFR, MPC libraries)

changeset:   3:026e7ec61437
Attachment #449710 - Flags: checked-in+
I watched a 32 and 64-bit slave sync up successfully.
Blocks: 570994
Attached patch buildbot-configs changes (obsolete) — Splinter Review
Taras, could you take a look at these mozconfigs for linux and linux64?

Some notes.

* I synced linux and linux64 mozconfigs, they are the same (next step will be removing linux64 configs and symlinking them to linux? :) ).

* As a result there are some changes:
** debug build for linux64 got --enable-libxul
** release build for linux64 got --enable-tests
** Dropped "export ac_cv_have_visibility_class_bug=yes" (per comments set for gcc-4.2.2)
Attachment #443967 - Attachment is obsolete: true
Attachment #450609 - Flags: review?(tglek)
FYI:

(In reply to comment #61)
> ** debug build for linux64 got --enable-libxul
> ** release build for linux64 got --enable-tests

These are both default settings. The --enable-libxul thing changed when we merged the electrolysis code, because it doesn't build without libxul, so we changed debug builds to default to libxul. --enable-tests has been the default for a long time.
Attachment #450609 - Flags: review?(tglek) → review?(ted.mielczarek)
Comment on attachment 450609 [details] [diff] [review]
buildbot-configs changes

I think Ted is the domain expert here.
Comment on attachment 450609 [details] [diff] [review]
buildbot-configs changes

>diff --git a/mozilla2-staging/linux/mozilla-central/codecoverage/mozconfig b/mozilla2-staging/linux/mozilla-central/codecoverage/mozconfig
>--- a/mozilla2-staging/linux/mozilla-central/codecoverage/mozconfig
>+++ b/mozilla2-staging/linux/mozilla-central/codecoverage/mozconfig
>@@ -1,20 +1,18 @@
> mk_add_options MOZ_MAKE_FLAGS="-j3"
>+mk_add_options PROFILE_GEN_SCRIPT='$(PYTHON) @MOZ_OBJDIR@/_profile/pgo/profileserver.py'

You don't really need this in the debug and codecoverage mozconfigs, since we're not going to do PGO builds there, but it also won't hurt anything by being there.

Looks fine otherwise. I should really file a bug about cleaning these up a bit. Since we got --enable-debug-symbols working correctly on trunk, we don't need the export CFLAGS="-gdwarf-2" bits, or the export MOZ_DEBUG_SYMBOLS=1 bits. Plus, as I mentioned before, --enable-tests is the default.
Attachment #450609 - Flags: review?(ted.mielczarek) → review+
Attached patch buildbot-configs changes (obsolete) — Splinter Review
(In reply to comment #64)
> (From update of attachment 450609 [details] [diff] [review])
> >diff --git a/mozilla2-staging/linux/mozilla-central/codecoverage/mozconfig b/mozilla2-staging/linux/mozilla-central/codecoverage/mozconfig
> >--- a/mozilla2-staging/linux/mozilla-central/codecoverage/mozconfig
> >+++ b/mozilla2-staging/linux/mozilla-central/codecoverage/mozconfig
> >@@ -1,20 +1,18 @@
> > mk_add_options MOZ_MAKE_FLAGS="-j3"
> >+mk_add_options PROFILE_GEN_SCRIPT='$(PYTHON) @MOZ_OBJDIR@/_profile/pgo/profileserver.py'
> 
> You don't really need this in the debug and codecoverage mozconfigs, since
> we're not going to do PGO builds there, but it also won't hurt anything by
> being there.

Removed it. Let's keep the mosconfigs clean. :) 

Keeping r+.
Attachment #450609 - Attachment is obsolete: true
Attachment #451648 - Flags: review+
Blocks: 571571
Attached patch buildbot-configs changes (obsolete) — Splinter Review
Applied to mozilla-2.0 too. Keeping r+.
Attachment #451648 - Attachment is obsolete: true
Attachment #453331 - Flags: review+
No longer blocks: 571571
Attachment #453331 - Flags: checked-in+
Attachment #453331 - Flags: checked-in-
Blocks: 575620
Depends on: 576399
Just landed a 4.5 workaround that should avoid the regressions we saw. When is the next time we can try a 4.5 switch?
I have been building Firefox (32-bit) on Linux using PGO successfully since Firefox version 3.0.

I have no problem with my most recent build of Firefox 3.6.6.

I have just had a preliminary attempt to build Firefox 4.0b1 and run into a minor problem:

Details:
source: pulled 'tar' downloaded from 'release candidate' folder on 'ftp site';
OS: Slackware Linux 12.1;
GCC version 4.2.3

Observation:
The PGO build process now proceed using '-fprofile-arcs/-fbranch-probabilities. instead of the correct process for GCC 4.2.3 which is '-fprofile-generate/fprofile-use'.

This has occurred because of the patch applied to 'configure.in' in BUG 418866 resulting in addition of GCC options (-fprofile-correction specifically) which is not compatible with GCC 4.2.3, therefore resulting in the process falling back to the option meant for GCC 3.3

See excerpt from 'configure.in':
dnl ========================================================
dnl Profile guided optimization
dnl ========================================================
dnl Test for profiling options
dnl Under gcc 3.3, use -fprofile-arcs/-fbranch-probabilities
dnl Under gcc 3.4+, use -fprofile-generate/-fprofile-use

I appreciate that future Mozilla builds will use GCC 4.5.x but if no amendment is made to 'configure'in' to allow for correct processing under GCC 4.2.3 then private builders will have problems as that GCC compiler will be treated like GCC 3.3.

Can anything be done to patch 'configure.in' to allow GCC 3.4+ to 4.2.3 to work as previously?.
Attached patch gcc patchSplinter Review
We'd like this patch applied to help static analysis builds
on mv-moz2-linux-ix-slave01:
Wrote: /home/cltbld/rpmbuild/SRPMS/gcc45-4.5.0-0moz2.src.rpm
Wrote: /home/cltbld/rpmbuild/RPMS/i386/gcc45-4.5.0-0moz2.i386.rpm

moz2-linux64-slave07:
Wrote: /usr/src/redhat/RPMS/x86_64/gcc45-4.5.0-0moz2.x86_64.rpm
Attachment #449710 - Attachment is obsolete: true
Attachment #457806 - Flags: review?(bear)
Attached patch GCC 4.5.0 Puppet manifests (obsolete) — Splinter Review
Bump RPM version
Attachment #449703 - Attachment is obsolete: true
Attachment #457809 - Flags: review?(bhearsum)
Attachment #457806 - Flags: review?(bear) → review+
Attachment #457809 - Flags: review?(bhearsum) → review+
Comment on attachment 457806 [details] [diff] [review]
GCC 4.5.0 spec (including plugin_finish_decl.diff)

http://hg.mozilla.org/build/rpm-sources/rev/c8fe426b3bdb
Attachment #457806 - Flags: checked-in+
Updated RPM package is being deployed and seen on some production and try machines. It should be available on all machines after the weekend.
Is there an eta for setting 4.5 as the default compiler yet?
Have you done new builds with it yet?  It should be available on try now.  What are the performance impacts of it?
(In reply to comment #77)
> Have you done new builds with it yet?  It should be available on try now.  What
> are the performance impacts of it?

I landed a workaround in bug 576399 which seems to bring perf back to 4.3 levels(according to try). Hard to say for sure until we try it.
Ok, let's try it early next week, assuming that works with the beta schedule.
(In reply to comment #79)
> Ok, let's try it early next week, assuming that works with the beta schedule.

So my plan is to try my workaround. Hope that it works like it did on try server.

If it turns out to not work, I'd like to switch our linux compilation to -O2, take the regression on startup(win on everything else), then turn on PGO and hopefully win on startup too.

It will all be simple if 4.5 works as expected(with 576399 workaround) and no hard decisions on accepting regressions have to be taken.
Attached patch buildbot-configs changes (obsolete) — Splinter Review
Refreshed mozconfigs.
Attachment #453331 - Attachment is obsolete: true
Attachment #460452 - Flags: review?(catlee)
4.5.1rc came out last week. Lets wait a few days for 4.5.1 final.
Attachment #460452 - Flags: review?(catlee) → review+
bug 582593 indicates that gcc 4.5 is broken :)
(In reply to comment #82)
> 4.5.1rc came out last week. Lets wait a few days for 4.5.1 final.

Ok, lets roll this out http://ftp.gnu.org/pub/gnu/gcc/gcc-4.5.1
Blocks: 584472
(In reply to comment #84)
> (In reply to comment #82)
> > 4.5.1rc came out last week. Lets wait a few days for 4.5.1 final.
> 
> Ok, lets roll this out http://ftp.gnu.org/pub/gnu/gcc/gcc-4.5.1

Taras, do you still need the patch (attachment 457606 [details] [diff] [review])?
(In reply to comment #85)

> Taras, do you still need the patch (attachment 457606 [details] [diff] [review])?

Yes please.
Attached patch GCC 4.5.1 spec (obsolete) — Splinter Review
Attachment #457806 - Attachment is obsolete: true
Attachment #463080 - Flags: review?(bear)
Attached patch GCC 4.5.1 Puppet manifests (obsolete) — Splinter Review
Attachment #457809 - Attachment is obsolete: true
Attachment #463082 - Flags: review?(bhearsum)
Attachment #463080 - Flags: review?(bear) → review+
Taras, please compile GCC 4.5.1 + patches and do some testing with it before we deploy it. We're happy to do compiler updates with good reason, but rolling them out to the entire build pool is not the right way to do initial testing.
(In reply to comment #89)
> Taras, please compile GCC 4.5.1 + patches and do some testing with it before we
> deploy it. We're happy to do compiler updates with good reason, but rolling
> them out to the entire build pool is not the right way to do initial testing.

I've run tests on 4.5.1 now, works as well as 4.3 for me.
Thanks Taras
Attachment #463082 - Flags: review?(bhearsum) → review+
Blocks: 585235
Blocks: 585605
No longer blocks: 585605
Comment on attachment 463082 [details] [diff] [review]
GCC 4.5.1 Puppet manifests

http://hg.mozilla.org/build/puppet-manifests/rev/4f06048ffe74

mpt-production-puppet, mv-production-puppet and staging-puppet servers are
updated as well. GCC 4.5.1 will be available on all slaves in day or two.
Attachment #463082 - Flags: checked-in+
This would be nice to have for firefox4 for the overall speedup and to facilitate jar reordering in bug 559961.
blocking2.0: --- → ?
Attached patch buildbot-configs changes (obsolete) — Splinter Review
Attachment #460452 - Attachment is obsolete: true
Attachment #468316 - Flags: review?(catlee)
Attachment #468316 - Flags: review?(catlee) → review+
Comment on attachment 468316 [details] [diff] [review]
buildbot-configs changes

Backed out due to failed "make test":
http://hg.mozilla.org/build/buildbot-configs/rev/9e22ddd81fa1
Attachment #468316 - Flags: checked-in+ → checked-in-
Rail, are there instructions for turning your spec files to rpms? Alternatively, are there some src rpms for the 4.5.0/4.5.1 specs above?
(In reply to comment #98)
> Rail, are there instructions for turning your spec files to rpms?

Very short instructions are here: https://wiki.mozilla.org/ReleaseEngineering:Puppet:Usage#Building_RPMs

> Alternatively, are there some src rpms for the 4.5.0/4.5.1 specs above?

The latest one is here:
http://people.mozilla.org/~raliiev/gcc/gcc45-4.5.1-0moz1.src.rpm

If you want I can regenerate src.rpm for 4.5.0.
Depends on: 594611
Depends on: 590683
Rail, please add the patch from http://gcc.gnu.org/bugzilla/show_bug.cgi?id=45623#c24 to our gcc 4.5.1. This appears to conclude our 4.5.1 issues on my machine. I would like to ensure that the same is true on try and switch to 4.5 asap. It's blocking bug 590181(otherwise we have to change how we build 4.3, but i'd rather not waste time on that if we can avoid it) and other perf fixes.
Blocks: 590181
(In reply to comment #9)
> Actually the good numbers produced by pgo in
> http://blog.mozilla.com/tglek/2010/04/12/squeezing-every-last-bit-of-performance-out-of-the-linux-toolchain/
> have little to do with cpu time. -freorder-blocks-and-partition option changed
> the layout of the binary for the better. Any minor cpu-time benefits are just
> icing on my startup cake.

This was PGO plus -Os, right?  I wonder if PGO would help more with -O3 (patch in bug 590181).
(In reply to comment #101)
> (In reply to comment #9)
> > Actually the good numbers produced by pgo in
> > http://blog.mozilla.com/tglek/2010/04/12/squeezing-every-last-bit-of-performance-out-of-the-linux-toolchain/
> > have little to do with cpu time. -freorder-blocks-and-partition option changed
> > the layout of the binary for the better. Any minor cpu-time benefits are just
> > icing on my startup cake.
> 
> This was PGO plus -Os, right?  I wonder if PGO would help more with -O3 (patch
> in bug 590181).

-O3 would hurt startup due to more bloated code causing more pagefaults.


Rail, http://gcc.gnu.org/viewcvs/branches/gcc-4_5-branch/gcc/tree-ssa-structalias.c?r1=164430&r2=164429&view=patch&pathrev=164430 is the official gcc fix. It supersedes the patch I linked in comment 100.
(In reply to comment #102)
> > This was PGO plus -Os, right?  I wonder if PGO would help more with -O3 (patch
> > in bug 590181).
> 
> -O3 would hurt startup due to more bloated code causing more pagefaults.

Right, this was the behavior we saw in bug 590181; -O3 was a Ts_cold regression, but sped everything else up.  But in that bug, we said we wanted -O3 despite the cold startup regression; my understanding is that we still intend to land it, once it's unblocked.

My suspicion is that PGO plus -O3 might give a bigger gain over -O3 than PGO plus -Os gives over -Os.  (Consider that adding -freorder-blocks to -Os is moving you closer to -O3.)  It's also possible that PGO plus -O3 might not hurt Ts_cold as much as non-PGO -O3, since perhaps PGO's optimizations reduce the cost of -O3's code bloat.

So my question is: Can we try this and collect data, perhaps in a separate bug?
Attached patch GCC 4.5.1 specSplinter Review
Updated spec file + the official patch
Attachment #463080 - Attachment is obsolete: true
Attachment #477257 - Flags: review?(bear)
Attachment #463082 - Attachment is obsolete: true
Attachment #477259 - Flags: review?(bhearsum)
Attachment #477257 - Flags: review?(bear) → review+
Attachment #477259 - Flags: review?(bhearsum) → review+
The package with the specified patch is being deployed and will be available on the build hosts (including try machines) within a couple of days. Happy testing! :)
Ok, try confirms that gcc4.5 fixes did their job. Lets make 4.5 the default compiler.

Don't worry about a perf hit, once 4.5 sticks. Bug 590181 will make up for any -Os perf regressions in 4.5.
(In reply to comment #109)
> Don't worry about a perf hit, once 4.5 sticks. Bug 590181 will make up for any
> -Os perf regressions in 4.5.
But we should probably post about the fact that a regression will happen to tree management and platform to make sure people are aware of what is going on.
And this should also include a reply to the automated Talos postings when it sees the regressions, otherwise people will get jumpy.
So 4.5 passed try except for opengl failures. I got
the following failing in an optimized build 
opengl
4998/80/270


and these in debug:
opengl
4998/125/280
I hope this is linux gl being broken and not.
Is that over and above the existing RGL failures? Linux/Linux64 are both pretty pooched on RGL on the standard build.
(In reply to comment #112)

> I hope this is linux gl being broken

Joe says "ignore", this means gcc 4.5 got a clean bill of health on try.
So now we need an eta from Build&Release.
Blocks: 599530
As per today's development meeting, this is not going to happen for Beta 7.

Taras, can you talk with beltzner to schedule having this possibly included in Beta 8?  If you want to plead your case, come to the next Tuesday development meeting.

Also, Talos numbers are needed before this can land.
jlebar, can you post your -O3 numbers with gcc 4.5?
Yes, I'll kick off a run on try.
(In reply to comment #117)
> Yes, I'll kick off a run on try.
We should get a few runs and then compare to a few good runs of parent changesets on mozilla-central with https://services.forerunnerdesigns.com/compare-talos/
Attachment #468316 - Attachment is obsolete: true
Attachment #480587 - Flags: review?(catlee)
Does this bug actually cover enabling PGO on our Linux builds, or just getting everything ready on the build slave side? If the former, we'll need to block on bug 602245.
(In reply to comment #120)
> Does this bug actually cover enabling PGO on our Linux builds, or just getting
> everything ready on the build slave side? If the former, we'll need to block on
> bug 602245.

AFAIK, we want to do both but in 2 steps: 1) use gcc-4.5.x for non profiled builds, wait some days and see if there are unexpected failures; 2) enable PGO
So now that b7 has branched when can we deploy 4.5?
Blocks: 603368
The b7 branch and trunk build with the same mozconfig. Can't do it at least until b7 ships.
Also, still missing talos numbers from comment 115.
For Talos results, see bug 590181 comment 68 and comment 69.  No ts_cold regression, which is a bit strange.  Its absence may be worth looking into.
Attachment #480587 - Flags: review?(catlee) → review+
Priority: P2 → P4
catlee, howcome this got bumped from P2 to P4?

I think we want this ASAP. Ideally, it'd be ready to go as soon as we ship b7. There is other stuff depending on this that we really want to get done.
(In reply to comment #126)
> catlee, howcome this got bumped from P2 to P4?
> 
> I think we want this ASAP. Ideally, it'd be ready to go as soon as we ship b7.
> There is other stuff depending on this that we really want to get done.

Because there's nothing actionable for us to do right now.  Everything is ready to go once we get the go-ahead.
Aha -- awesome. Thanks. :)
Bug 578880 seems to depend on this -- marking this a blocker.
blocking2.0: ? → betaN+
We've shipped b7. Are we good to go?
(In reply to comment #130)
> We've shipped b7. Are we good to go?

Hasn't quite shipped yet.  Build 1 is still in QA.
Er, sorry. Meant tagged. :/
(132 comments, really?)

So...since we've got a b7 relbranch, and any future b7 fixes will go on the relbranch, I think we're ok to make the switch on trunk.

Taras, want to bring this up at Tuesday's platform meeting to schedule the switch-over?
Flags: needs-treeclosure?
Depends on: 609543
Flags: needs-treeclosure?
Is there an ETA for this?  This is currently blocking blocker work for Firefox 4 on Linux, not just turning on PGO.
This needs bug 609543 to be fixed, at least, before we can try again.
Is gcc 4.5 already deployed to the build machines?  That is, can a mozconfig switch over to test whether bug 609543 is really being caused by gcc 4.5?
(In reply to comment #136)
> Is gcc 4.5 already deployed to the build machines?  That is, can a mozconfig
> switch over to test whether bug 609543 is really being caused by gcc 4.5?

Yes, it's been deployed for ages.  You can do a try build with a custom mozconfig to select gcc 4.5.
OK, bug 609543 is fixed. It sounds like jlebar has found another couple failures though: https://bugzilla.mozilla.org/show_bug.cgi?id=590181#c107. Do those need to block deployment here?
blocking2.0: betaN+ → -
No longer depends on: 609543
No longer blocks: 578880
Blocks: 578880
Depends on: 609543
Comment on attachment 480587 [details] [diff] [review]
buildbot-configs changes (mozconfigs)

>+export LDFLAGS="-static-libstdc++"

You will also need to set HOST_LDFLAGS to the same value. (but that'll need bug 628303 to land first)
This is currently a problem on seamonkey, which switched to 4.5 with this patch, since elfhack (bug 606145) landed.
gcc-4.5 is already installed on the linux machines; this bug is now tracking enabling it by default for building firefox.
Summary: Install GCC 4.5 on linux buildslaves to enable PGO on Linux → switch to gcc-4.5 for linux builds
I figured a hackish way to avoid g++ 4.5 linking against libstdc++ symbols that are only available in libstdc++.so.6 from gcc 4.5. I went even further and got rid of symbols from gcc 4.2, making the resulting binaries work with the system-provided libstdc++.so.6 on buildbots, allowing to get rid of these LD_LIBRARY_PATHs:
http://hg.mozilla.org/build/buildbot-configs/file/ccf475f8cf9d/mozilla/config.py#l1301

It also avoids requiring to use -static-libstdc++, so that could taken off attachment 480587 [details] [diff] [review].

I'll try to build with g++ 4.6 to see if there will be any further symbols to hack around in the future with the current codebase. (Future changes to the codebase may require other symbols, so the use of STL should still be discouraged)

Try builds available here:
http://ftp.mozilla.org/pub/mozilla.org/firefox/tryserver-builds/mh@glandium.org-69d6499b5c8a/

As that was only for a test, I tucked the hack in a random file and didn't protect it in some #ifdef. Integration details should be figured out before switching to gcc 4.5, obviously.
When bug 641842 is fixed, we should try rolling this out only on the build-system branch.
Depends on: 643690
Let's make both switches (gcc 4.5 + PGO) at the same time. I've heavily tested PGO on try and with gcc 4.5, it works great. We'll need to land bug 590181 first to get the best out of it (and I verified this doesn't make problems with gcc 4.3, except for a big size bump, that PGO greatly removes)
No longer blocks: 590181
Depends on: 641842, 602245
Summary: switch to gcc-4.5 for linux builds → switch to gcc-4.5 and enable PGO for linux builds
Depends on: 590181
So, we're almost set. All I need is a r+ on bug 643690, which should be pretty quick, and we're ready to go.

mozconfig-wise, we need to change CC and CXX, we don't need to add -static-libstdc++ to LDFLAGS, and we need to add the following to enable PGO:
  mk_add_options MOZ_PGO=1
and refresh PROFILE_GEN_SCRIPT where it makes sense.

We need to land bug 643690 before or during the switch, and bug 590181 preferably after or during.
How and when can we plan this switch?
(In reply to comment #144)
> So, we're almost set. All I need is a r+ on bug 643690, which should be pretty
> quick, and we're ready to go.

... and 44 seconds later, the r+ was there. So we're ready :)
Blocks: 650304
And now that I landed bug 650467, we'll also need:
  ac_add_options --enable-stdcxx-compat
Attached patch buildbot-config changes (obsolete) — Splinter Review
This is a refreshed patch as per comment 144 and comment 146.

I'm however unsure we want to enable PGO on unittest and debug builds.

Note that these changes will need to propagate to branches as the current mozilla-central propagates to aurora, beta and release.

At some point, some branches like tracemonkey and build-system will also need to switch. I'm however wondering how to handle try...
Attachment #480587 - Attachment is obsolete: true
Attachment #526668 - Flags: review?(catlee)
Comment on attachment 526668 [details] [diff] [review]
buildbot-config changes

Debug builds probably shouldn't use PGO. We don't on Windows at least.
No need to update the unittest mozconfigs, they're not used any more.

At the least try should be updated at the same time. Other branches should probably all happen at the same time as well. Maybe a post to dev.planning warning folks this is happening would be appropriate?
Attachment #526668 - Flags: review?(catlee) → review-
I think it would be best to give this a test run on build-system first. If we break anything, then it won't affect work on m-c.
The problem with try is that if something before the landing of bug 643690 is pushed, it won't work.
I had a nasty idea for try:

if hg log -r 98094f3afb93... -l1 | grep -q ^$; then
CC=/tools/gcc-4.5/bin/gcc
(...)
else
CC=/tools/gcc-4.3.3/installed/bin/gcc
(...)
fi

there's maybe a better way to do this, but I don't know mercurial well enough. It checks that revision 98094f3afb93 is an ancestor of the current revision.
(In reply to comment #149)
> I think it would be best to give this a test run on build-system first. If we
> break anything, then it won't affect work on m-c.

Note that it doesn't look like build-system has a separate mozconfig at the moment.
There appears to be a whole lot of stuff that are symbolic links to mozconfig-central configurations (e.g. mozilla-1.9.1, electrolysis...), we need to break these symbolic links first.
Attachment #526668 - Attachment is obsolete: true
Attachment #528560 - Flags: review?(catlee)
These are the actual changes to the buildbot configs.
Attachment #528561 - Flags: review?(catlee)
(In reply to comment #154)
> Created attachment 528561 [details] [diff] [review]
> buildbot-config changes
> 
> These are the actual changes to the buildbot configs.

Note, I was wondering if we also need to change the "generic" ones.
Attached file Mercurial hook (obsolete) —
This is a mercurial hook that checks whether something that is pushed to try contains changeset 3a38a70b0e12 as an ancestor, which is necessary for builds to work properly with gcc 4.5. It does so only if the try push is targetted at linux, linux64 or all platforms.
When it detects a push that doesn't contain that changeset, it issues a warning, currently only "Warning message" ; this obviously needs appropriate wording. For instance, do we want it to be verbose and give the workaround, or simply link to some documentation?
Attachment #528579 - Attachment mime type: text/x-python → text/plain
Attachment #528560 - Flags: review?(catlee) → review+
Attachment #528561 - Flags: review?(catlee) → review+
Attached file Mercurial hook
I will update the wiki accordingly (if I can)
Attachment #528579 - Attachment is obsolete: true
Attachment #528841 - Flags: review?(ted.mielczarek)
Attachment #528841 - Attachment mime type: text/x-python → text/plain
Comment on attachment 528841 [details]
Mercurial hook

Seems reasonable enough. You'll need to file a Server Ops bug to get the hook deployed into the try repo. You can commit it to the hghooks repo to have a place to put it: http://hg.mozilla.org/hgcustom/hghooks . You should ask someone in IT to verify that argparse will work there first, I'm not particularly sure about that.
Attachment #528841 - Flags: review?(ted.mielczarek) → review+
Depends on: 653437
It looks like it worked well (once the mozconfig were actually used on the
buildbots)
Assignee: rail → mh+mozilla
Status: ASSIGNED → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
This might have broken Linux crash reports.  See bug 654595.
Depends on: 655020
Depends on: 652459
No longer depends on: 655020
Depends on: 655020
Depends on: 655121
Depends on: 655124
Alias: gcc4.5
No longer depends on: 655124
Depends on: 656500
Depends on: 658167
We forgot to track it for Firefox 6.
Whiteboard: [puppet][buildslaves] → [puppet][buildslaves] nominated in comment 163
Just to clarify, we're tracking this for 6 just to watch what happens here and possibly backing out this change for 6? Or is there additional work that we're still waiting on here beyond possibly backing things out?
Is there any configuration that needs to migrate from mozilla-central to mozilla-aurora to mozilla-beta to mozilla-release in order for this to apply to releases?
(In reply to comment #165)
> Is there any configuration that needs to migrate from mozilla-central to
> mozilla-aurora to mozilla-beta to mozilla-release in order for this to apply
> to releases?

Looks like the beta buildbot-config [1] will need to be updated to match the aurora/m-c config [2].

[1] http://hg.mozilla.org/build/buildbot-configs/file/e2da95567950/mozilla2/linux/mozilla-beta/release/mozconfig

[2] http://hg.mozilla.org/build/buildbot-configs/file/e2da95567950/mozilla2/linux/mozilla-aurora/release/mozconfig
The same thing as bug 658167 will need to happen on mozilla-beta and then on mozilla-release. That's why bug 558180 would really be helpful.
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.