633269 - Use gold for linking on linux

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P5)

Description

[Jeff Muizelaar [:jrmuizel]]

gold is faster at linking

Comment 1

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

I have been using it for trying LTO, so far all the problem I have found were LTO related :-)

From my experience gold is faster than gnu ld, specially if built in 64 bits. Another nice thing is you can use a 64 bit gold for linking both 32 and 64 bits binaries by default.

Two easy ways to use it for just some builds:
*) use "export PATH=/path/to/gold:$PATH". This assumes that our gcc is searching the path.
*) use CC="gcc -B/path/to/gold" CXX="g++ -B/path/to/gold"

I think there is a newer option in gcc for doing the selection in a cleaner way, but I am not sure.

Comment 2

[Mike Hommey [:glandium]]

Wasn't that planned with gcc 4.5 in bug 559964 ?

Comment 3

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Well, the ld used is independent of the gcc version, so we can change to gold before or after the change of gcc.

Comment 4

[Mike Hommey [:glandium]]

But there is not much benefit from gold without a gcc change.

Comment 5

[Jeff Muizelaar [:jrmuizel]]

(In reply to comment #4)
> But there is not much benefit from gold without a gcc change.

Why not? How does the gcc version impact link time?

Comment 6

[Mike Hommey [:glandium]]

(In reply to comment #5)
> (In reply to comment #4)
> > But there is not much benefit from gold without a gcc change.
> 
> Why not? How does the gcc version impact link time?

Well, I don't see much benefit in changing the linker for the sake of a marginal win in the whole build time. That can help locally when you rebuild after changing one file, but you don't need mozilla to switch to gold for that.

Comment 7

[Chris AtLee [:catlee]]

(In reply to comment #6)
> (In reply to comment #5)
> > (In reply to comment #4)
> > > But there is not much benefit from gold without a gcc change.
> > 
> > Why not? How does the gcc version impact link time?
> 
> Well, I don't see much benefit in changing the linker for the sake of a
> marginal win in the whole build time. That can help locally when you rebuild
> after changing one file, but you don't need mozilla to switch to gold for that.

Except that we do a few hundred builds every day, so speeding up _any_ part of the process helps everybody out.

Comment 8

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

gold also has a much better tested support for LTO (both LLVM and GCC plugins).

Comment 9

[Mike Hommey [:glandium]]

(In reply to comment #8)
> gold also has a much better tested support for LTO (both LLVM and GCC plugins).

... which you don't get until you change gcc...

(In reply to comment #7)
> Except that we do a few hundred builds every day, so speeding up _any_ part of
> the process helps everybody out.

How long does libxul.so linking take on buildbots ? On my machine it takes a few seconds...

Comment 10

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

(In reply to comment #9)
> (In reply to comment #8)
> > gold also has a much better tested support for LTO (both LLVM and GCC plugins).
> 
> ... which you don't get until you change gcc...

Or clang :-)

There are features we get from having gold, there are features we get from having a new compiler and there are features we get from having both.

I don't see why the linker update should wait for the compiler update (or the other way if updating gcc is easier than changing the linker).

Comment 11

[Mike Hommey [:glandium]]

(In reply to comment #10)
> (In reply to comment #9)
> > (In reply to comment #8)
> > > gold also has a much better tested support for LTO (both LLVM and GCC plugins).
> > 
> > ... which you don't get until you change gcc...
> 
> Or clang :-)
> 
> There are features we get from having gold, there are features we get from
> having a new compiler and there are features we get from having both.
> 
> I don't see why the linker update should wait for the compiler update (or the
> other way if updating gcc is easier than changing the linker).

I'm not sure I want to handle binary layout changes due to toolchain changes twice. Especially seeing how different versions of gold give completely different layouts.

Comment 12

[(dormant account)]

Mike,
gold is a significant win in terms of footprint. it's not tied to gcc4.5 at all.

We are going to use a single version of gold, that's all that elfhack has to work with. Perhaps we could even integrate elfhack into gold someday :)

Comment 13

[John Ford [:jhford] CET/CEST Berlin Time]

This seems like a nice thing to have.  We should probably do some timing to see how this affects build time.  As this touches the product, we should probably also have people verify that it doesn't negatively impact our binaries.  Until that work is done, I don't think we can proceed here.  Please feel free to file bugs that block this one for timing and product verification.

Comment 14

[Jeff Muizelaar [:jrmuizel]]

(In reply to comment #13)
> This seems like a nice thing to have.  We should probably do some timing to see
> how this affects build time.  

The timing that I've done shows that it improves build time. To what detail do you need information?

> As this touches the product, we should probably
> also have people verify that it doesn't negatively impact our binaries.  Until
> that work is done, I don't think we can proceed here.

I'm not sure what you want done here. Is there much we can do beyond running it through our tests?

Comment 15

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to comment #14)
> (In reply to comment #13)
> > This seems like a nice thing to have.  We should probably do some timing to see
> > how this affects build time.  
> 
> The timing that I've done shows that it improves build time. To what detail do
> you need information?

How much does this improve the build time on a cold file cache clobber build vs the standard ld compared to total build time (including make package, make package-tests, make buildsymbols)?  How much does this improve build time on a cold file cache incremental build?

Until we are able to support LTO (which comment 9 suggests we don't currently), build time seems like the biggest win of switching to gold.  Lets make sure that we have numbers to back up this choice.

We are also in the endgame of Firefox 4 and it feels like any toolchain changes should be made after it ships.

Comment 16

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

I was able to build firefox with LTO yesterday. I will write a bit about it as soon as I have time.

Comment 17

[Chris AtLee [:catlee]]

To get this working with our build system, I need to add this to my mozconfig:

mk_add_options PATH=/tools/binutils-2.21/bin:$PATH

Using -B /tools/binutils-2.21 didn't work (gcc looked for real-ld in there, but didn't find it, so fell back to /usr/bin/ld)
Using GCC_EXEC_PREFIX didn't work either.

Comment 18

[Chris AtLee [:catlee]]

Attachment: rpm spec for building / packging gold from binutils-2.21

Comment 19

[Chris AtLee [:catlee]]

Attachment: puppet manifests for gold

Comment 20

[Chris AtLee [:catlee]]

Attachment: rpm spec for building / packging gold from binutils-2.21

Comment 21

[Mike Hommey [:glandium]]

It would be useful to have a directory where ld is bfd ld and another one where ld is gold, so that we can easily switch from one to the other manually (for example, on try) if necessary.

Please make sure to use binutils 2.21.1, too.

Comment 22

[Chris AtLee [:catlee]]

(In reply to comment #21)
> It would be useful to have a directory where ld is bfd ld and another one
> where ld is gold, so that we can easily switch from one to the other
> manually (for example, on try) if necessary.
> 
> Please make sure to use binutils 2.21.1, too.

This won't replace the existing binutils. Do you mean you want a different directory where the 2.21.1 bfd ld is the default?

What's changed in 2.21.1?

Comment 23

[Mike Hommey [:glandium]]

(In reply to comment #22)
> (In reply to comment #21)
> > It would be useful to have a directory where ld is bfd ld and another one
> > where ld is gold, so that we can easily switch from one to the other
> > manually (for example, on try) if necessary.
> > 
> > Please make sure to use binutils 2.21.1, too.
> 
> This won't replace the existing binutils. Do you mean you want a different
> directory where the 2.21.1 bfd ld is the default?

The existing binutils is old and should be upgraded, so yes, we'd need a directory where a recent bfd ld is available.
I know gold has many advantages, but it also has its own set of problems, and we'd better make sure to have a viable alternative if we have too many of them, or if we have good reasons to require bfd ld at some point. Our old syste bfd ld is not a viable alternative.

> What's changed in 2.21.1?

Supposedly, there are fixes and improvements in gold.

Comment 24

[Chris AtLee [:catlee]]

(In reply to comment #23)
> (In reply to comment #22)
> > (In reply to comment #21)
> > > It would be useful to have a directory where ld is bfd ld and another one
> > > where ld is gold, so that we can easily switch from one to the other
> > > manually (for example, on try) if necessary.
> > > 
> > > Please make sure to use binutils 2.21.1, too.
> > 
> > This won't replace the existing binutils. Do you mean you want a different
> > directory where the 2.21.1 bfd ld is the default?
> 
> The existing binutils is old and should be upgraded, so yes, we'd need a
> directory where a recent bfd ld is available.
> I know gold has many advantages, but it also has its own set of problems,
> and we'd better make sure to have a viable alternative if we have too many
> of them, or if we have good reasons to require bfd ld at some point. Our old
> syste bfd ld is not a viable alternative.

I can't replace the existing system bfd ld without extensive testing. I'm planning on making the new tools available at /tools/binutils-2.21.1/bin (with gold as the default). Perhaps /tools/binutils-2.21.1/bin.bfd could have the bfd ld as the default?

Comment 25

[Mike Hommey [:glandium]]

(In reply to comment #24)
> (In reply to comment #23)
> > (In reply to comment #22)
> > > (In reply to comment #21)
> > > > It would be useful to have a directory where ld is bfd ld and another one
> > > > where ld is gold, so that we can easily switch from one to the other
> > > > manually (for example, on try) if necessary.
> > > > 
> > > > Please make sure to use binutils 2.21.1, too.
> > > 
> > > This won't replace the existing binutils. Do you mean you want a different
> > > directory where the 2.21.1 bfd ld is the default?
> > 
> > The existing binutils is old and should be upgraded, so yes, we'd need a
> > directory where a recent bfd ld is available.
> > I know gold has many advantages, but it also has its own set of problems,
> > and we'd better make sure to have a viable alternative if we have too many
> > of them, or if we have good reasons to require bfd ld at some point. Our old
> > syste bfd ld is not a viable alternative.
> 
> I can't replace the existing system bfd ld without extensive testing.

I wasn't suggesting to do that :)

> I'm planning on making the new tools available at /tools/binutils-2.21.1/bin
> (with gold as the default). Perhaps /tools/binutils-2.21.1/bin.bfd could
> have the bfd ld as the default?

Sounds fair enough.

Comment 26

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Having two bin directories and passing -B to gcc to select the linker is probably the best option. Another somewhat hackish one is to compile gcc with --with-plugin-ld and pass a dummy plugin to it.

Comment 27

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

where "it" is the linker (-fuse-linker-plugin), not gcc :-)

Comment 28

[Chris AtLee [:catlee]]

Rafael, do we need to build with --enable-lto? Any other configure options? I've currently got:

./configure --enable-gold=default --enable-lto --prefix=%{install_dir} CC="/tools/gcc-4.5/bin/gcc -static-libgcc" CXX="/tools/gcc-4.5/bin/g++ -static-libgcc -static-libstdc++"

Comment 29

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

I think you would need to, both gold and gcc. It is probably a good thing to do anyway so that we can try LTO at some point.

One problem with using -fuse-linker-plugin just to select the linker is that it would slow down the link a bit as the plugin would be tried on every object file.

It is a pity gcc's support for selecting the linker at runtime is attached to LTO. I guess my suggestion then is

* build binutils with --enable-plugins --enable-gold=default --enable-lto
* copy the linkers, so that we have gold in <inst>/bin/ld and the old linker in <inst>/bin.bfd/ld
* build gcc with --enable-lto (I think it is the default)
* when building firefox, use -B (or export PATH) to select the linker

Comment 30

[Mike Hommey [:glandium]]

(In reply to comment #27)
> where "it" is the linker (-fuse-linker-plugin), not gcc :-)

AFAIK -fuse-linker-plugin doesn't work to select a linker at runtime with gcc. It's only used for lto.

Comment 31

[Mike Hommey [:glandium]]

(In reply to comment #29)
> * build gcc with --enable-lto (I think it is the default)

Which is pointless until we start building gcc 4.7

Comment 32

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

(In reply to comment #30)
> (In reply to comment #27)
> > where "it" is the linker (-fuse-linker-plugin), not gcc :-)
> 
> AFAIK -fuse-linker-plugin doesn't work to select a linker at runtime with
> gcc. It's only used for lto.

-fuse-linker-plugin causes gcc to use the linker defined with --with-plugin-ld instead of the default linker.

Comment 33

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

(In reply to comment #31)
> (In reply to comment #29)
> > * build gcc with --enable-lto (I think it is the default)
> 
> Which is pointless until we start building gcc 4.7

Or clang :-)

Comment 34

[Chris AtLee [:catlee]]

Attachment: rpm spec for building / packging gold from binutils-2.21.1

Comment 35

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 547098 [details] [diff] [review]
rpm spec for building / packging gold from binutils-2.21.1

># HG changeset patch
># User Chris AtLee <[e-mail]>
># Date 1311173911 14400
>
>diff --git a/binutils/centos5-i686/binutils.spec b/binutils/centos5-i686/binutils.spec
>new file mode 100644
>--- /dev/null
>+++ b/binutils/centos5-i686/binutils.spec
>@@ -0,0 +1,50 @@
>+# Named with the moz_ prefix so it doesn't conflict with the existing binutils
>+# package. Include the version in the same so we can have multiple versions of
>+# it installed.

I would define the real name somewhere up top

%define real_name binutils

>+Name: moz_binutils-2.21.1

Name: moz_%{binutils}_%{version}

>+Summary: binutils
>+Version: 2.21.1
>+Release: 0moz1

I don't think we should put in a moz release number unless we are modifying an upstream specfile, where the moz1 would represent the first moz release of upstream's 0 release </bikeshed>

>+License: GPL
>+Group: C
>+# Get source from http://ftp.gnu.org/gnu/binutils/binutils-2.21.1.tar.bz2
>+Source: binutils-2.21.1.tar.bz2

We should put the url as the source tag, as well as using an explicit number

Source0: http://ftp.gnu.org/gnu/%{real_name}/%{real_name}-%{version}.tar.bz2

>+BuildRoot: %{_tmppath}/%{name}-%{version}-root
>+
>+%define toplevel_dir binutils-2.21.1

I'd use macros,

%define toplevel_dir %{real_name}-%{version}

>+%define install_dir /tools/%{toplevel_dir}
>+
>+%description
>+binutils packaged for mozilla build machines
>+
>+%prep
>+rm -rf $RPM_BUILD_DIR/%{toplevel_dir}
>+tar -jxf %{SOURCE0} >/dev/null

This %prep section shouldn't be used.  Instead, we should be using the %setup macro (http://www.rpm.org/max-rpm/s1-rpm-inside-macros.html).  The %setup macro takes care of deleting the BUILD subdir as well as extracting the sources and changing to the correct working directory.

General purpose RPMs would use "%setup" but because we are prepending moz_ and appending _%{version} to the package name, we need to do

%setup -n %{real_name}-%{version}

>+%build
>+cd %{toplevel_dir}
>+# CC and CXX here are specific to our build machines
>+# For testing you may want to use the commented out line below
>+#./configure --enable-gold=default --prefix=%{install_dir} CC="ccache gcc" CXX="ccache g++"
>+./configure --enable-gold=default --enable-lto --enable-plugins --prefix=%{install_dir} CC="/tools/gcc-4.5/bin/gcc -static-libgcc" CXX="/tools/gcc-4.5/bin/g++ -static-libgcc -static-libstdc++"
>+make
>+
>+%install
>+install -d -m 755 $RPM_BUILD_ROOT
>+mkdir -p $RPM_BUILD_ROOT/%{install_dir}
>+cd %{toplevel_dir}
>+make install prefix=$RPM_BUILD_ROOT/%{install_dir}

Every RPM i have written or read outside of our repository uses DESTDIR instead of setting the prefix for the install target, http://www.gnu.org/prep/standards/html_node/DESTDIR.html.  Setting the prefix as aconfigure option should make the files install to the correct path under $RPM_BUILD_ROOT.  The upstream binutils package uses the DESTDIR variable and I think we should to.  http://pkgs.fedoraproject.org/gitweb/?p=binutils.git;a=blob;f=binutils.spec;h=a9fc7872defdf9988822d64800e136664c609ccb;hb=HEAD#l229 .

make install DESTDIR=$RPM_BUILD_ROOT

>+# Make another directory available where ld.bfd is the default
>+cp -al $RPM_BUILD_ROOT/%{install_dir}/bin $RPM_BUILD_ROOT/%{install_dir}/bin.bfd
>+ln -f $RPM_BUILD_ROOT/%{install_dir}/bin.bfd/ld.bfd $RPM_BUILD_ROOT/%{install_dir}/bin.bfd/ld

Cool!  I am not sure about the internals of binutils, so I assume that the /bin.bfd/ld binary won't reference anything in /bin/.  Is that correct?

>+
>+%clean
>+rm -fr $RPM_BUILD_ROOT
>+
>+%files
>+%defattr(-, root, root)
>+%{install_dir}
>+
>+%changelog
>+* Thu Feb 17 2011 Chris AtLee <[e-mail]>

Lets include the version and release as well

* Thu Feb 17 2011 Chris AtLee <[e-mail]> 2.21.1-0moz1

>+- Initial spec
>diff --git a/binutils/centos5-x86_64 b/binutils/centos5-x86_64
>new file mode 120000
>--- /dev/null
>+++ b/binutils/centos5-x86_64
>@@ -0,0 +1,1 @@
>+centos5-i686
>\ No newline at end of file

r- because:
  -%setup macro should be used
  -DESTDIR is preferable for the install target
  -we should use macros instead of strings for real package name and version to make upgrades easier in future

Comment 36

[Chris AtLee [:catlee]]

Attachment: rpm spec for building / packging gold from binutils-2.21.1

Comment 37

[Chris AtLee [:catlee]]

Attachment: refreshed manifest

Comment 38

[Chris AtLee [:catlee]]

Attachment: refreshed manifest

the real one!

Comment 39

[Chris AtLee [:catlee]]

Attachment: rpm spec for building / packging gold from binutils-2.21.1

Comment 40

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 547277 [details] [diff] [review]
rpm spec for building / packging gold from binutils-2.21.1

Review of attachment 547277 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good to me

Comment 41

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 547273 [details] [diff] [review]
refreshed manifest

stamp

Comment 42

[Chris AtLee [:catlee]]

being deployed to the slaves now, should be everywhere tomorrow.

you should be able to use it using

mk_add_options PATH=/tools/binutils-2.21.1/bin:$PATH

in your mozconfig-extra-linux on try.