683975 - Need infra for developer contributed compilers

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris AtLee [:catlee]]

Some use cases:

* dev is doing clang work, he wants to do a try build with a new version of clang
* dev patches gcc to fix a compiler bug, he wants to do a try build with it and then update the 'real' compilers once the try build is good

These are currently impossible or very difficult without a lot of back and forth with releng.

One idea was to have a pair of svn repos, one for try and one for production. Compilers would be checked in to e.g.
/linux32/gcc45/...
/macosx64/clangXX/...

the mozconfigs for builds would specify the path to repo, as well as a revision

the build machines would then look for e.g. /path/to/local/compilers/gcc45-r1 and if it doesn't exist, check it out from svn.

this prevents having to do updates across revisions with svn.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

From earlier conversations, a key point here is that this needs to be hands-free for releng -- a developer with try access should be able to test a custom compiler with no human interaction.

A few questions:
 - are any of these compilers secret, or can they be publicly accessible?
 - how large are the compilers?
 - how often do you expect new compilers to be uploaded?  Are we looking at some percentage of the try runs, or one or two per week?
 - do the compilers need to be kept around after the try run(s)?

I'd like to avoid adding additional systems or moving parts to support this, if possible.

Other implementation options:

 - drop files somewhere on FTP -- we'd have to figure out the auth for that, and data lifetime

 - user hg repos -- auth for free, and the dev controls the data lifetime (by deleting the repo)

Either of these options could be combined with changes to the build system, and no changes to the buildbot configs.  Also, they both follow existing network flows.

Comment 2

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Is this bug the same as 678088?

I like the idea of using mozconfigs to specify both repo and revision. Once they are in m-c they will be a very convenient. This should be used for all compilers and other tools, including the ones we build m-c with. Advantages:

*) releng doesn't need to be involved in testing or deploying a new compiler or tool (let say we start depending on bison for some crazy reason).
*) Test compiler and production compilers use the same infrastructure (test what you ship)
*) Tests can be done with the regular try servers.
*) Production changes (switch to clang on mac, gcc 4.6 on linux, etc) show up as a regular push on tbpl and can be monitored and reverted by regular devs.

Comment 3

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

>  - how often do you expect new compilers to be uploaded?  Are we looking at
> some percentage of the try runs, or one or two per week?

Even less than that. Since I joined, I think all the pushes would have been

* Two attempts to switch to gcc 4.5
* Two try versions of clang
* some pushes (less than 5) trying to fix gcc bugs

Do note that while there are not many of these, they do have a tendency to cluster, so there can be a week with 3 or 4 try pushes, which is a lot more than what existing infrastructure is convenient for.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Still looking for answers to the other questions, but it seems like this could be done with almost no change to releng infra:

First, add support for selecting/downloading tools in mozconfigs, using an arbitrary URL.  Any downloads should be part of the objdir, and not re-downloaded unnecessarily[1].

When you want to *test* a new tool, add it to a user repo and pull directly from that repo, since build machines don't have access to arbitrary internet URLs.  However, once the tool is in production this would put quite a load on hg, so at that point file a bug with releng to get the tool installed poolwide, and adjust the mozconfigs to point to the installed version.

Thoughts?

[1] Actually, this may be a lot of complexity for little benefit, particularly since try always clobbers.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

So from an IRC conversation in #build, the above proposal has the downside of requiring releng intervention in the deployment phase.  It also makes it hard to keep old tools around until their branches die, and age them out at that time.  Here's a new proposal, as discussed:

Build a system for downloading tools during the build process, but download them somewhere *outside* the objdir, so that they will persist between builds even across clobbers.  Store them using hashes of some sort to prevent collisions, and build a system to age out unused tools.

For the let's-try-a-new-tool phase, this system would be pointed at tools in a user hg repo.  This is a place that devs can access easily and iterate quickly.  Since it's content-addressable, parallel builds can even use different versions of the tools.

For the deploy-a-new-tool phase, the tool would be uploaded to a well-known directory on ftp.m.o, and downloaded from there.  Different branches can still use different versions of tools -- they'll just download the required version.  So this upload to FTP could happen as soon as the new tool is deemed likely to be deployed -- there's no need to coordinate it with mozconfig changes in m-c or m-i or the like.

The FTP upload is still a question.  Options:

 * releng bug: "Bug XXXXXX: Please add hg:/path/to/some/tool/1.2.3 to /tools/some/tool/1.2.3 on FTP".  This is a simple scp, so the bug would be quick to take care of.

 * SSH key access to the directory for a wider audience: Maybe release-drivers? select devs working on new tools?

 * Automation: a web-based uploader, or some other automatic tool (some security concerns there)

For the record, my opposition to SVN is based on two things:
 1. We don't actively use SVN in build right now, so starting to use it would add another moving piece to break
 2. SVN, while better than a DVCS, is not the right tool for handling large binary files

P.S. We briefly discussed using a CAS for this purpose.  Theoretically it's a good solution, but we have no such infrastructure set up at this point, and no concrete plans to do so.

Comment 6

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Since the proposed system can point to a different repo, it should be possible to do try by uploading compilers to people.mozilla.com or a user repo (they also support http anyway).

With try jobs being so easy, I think it is reasonable to ask releng to copy an yet unused package to a ftp server once it is ready to be used in production.

I will try to build something/copy it from rust.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

Rafael, access from the build network is pretty severely restricted (well, will be soon), so this would have to be in a fairly limited number of places.  I suspect that will *not* include people, but will include user repos.

Comment 8

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

That should be fine. A user repo can include a 100MB tar.bz2, no?

Comment 9

[(no longer active)]

(In reply to Rafael Ávila de Espíndola (:espindola) from comment #8)
> That should be fine. A user repo can include a 100MB tar.bz2, no?

Yes.

Comment 10

[[e-mail] (:bhearsum)]

(In reply to Dustin J. Mitchell [:dustin] from comment #5)
> The FTP upload is still a question.  Options:
> 
>  * releng bug: "Bug XXXXXX: Please add hg:/path/to/some/tool/1.2.3 to
> /tools/some/tool/1.2.3 on FTP".  This is a simple scp, so the bug would be
> quick to take care of.

RelEng already has access to FTP through cltbld & ffxbld accounts on surf. Would it be enough to create, eg, /pub/mozilla.org/buildtools owned by cltbld?

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

That's my thinking, yes.

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

I think this is a releng task from here on out, then?

Comment 13

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

I can do the first bits: Converting the spec file into a shell script to build a relocatable tar.br2.

Comment 14

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: gcc build script

I have had this bug as an idle task for some time. I have started by creating a build script for gcc. The final objective it that running the script on any linux machine should produce

* a .tar.bz2 file with a 32 bit compiler
* a .tar.bz2 file with a 64 bit compiler
* a pair of rpm files with the same contents as the .tar.bz2 files

In addition, the content of those files should be reproducible.

What I have so far is a script that works both on a new fedora 16 and in the very old centos 5 that we use. So far it only produces a 64 bit toolchain and the results are partially deterministic. Running the script twice will produce the exact same contents, but running it on two different distros will not.

I would like to check this in somewhere in m-c while I work on finishing it up if possible.

Comment 15

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: deterministic ar (helper script)

Comment 16

[Rail Aliiev [:rail]]

Comment on attachment 589030 [details]
gcc build script

Looks good to me.

Comment 17

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/a363fe2b2d3f

Comment 18

[Marco Bonardo [:mak]]

https://hg.mozilla.org/mozilla-central/rev/a363fe2b2d3f

Comment 19

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: bootstrap the linker too.

Comment 20

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=8c71c2afb684

Comment 21

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/8c71c2afb684

Comment 22

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: build glibc

We this we should be producing identical binaries everywhere, but I haven't tested that yet.

Comment 23

[Rail Aliiev [:rail]]

Comment on attachment 590936 [details] [diff] [review]
build glibc

Review of attachment 590936 [details] [diff] [review]:
-----------------------------------------------------------------

::: build/unix/build-toolchain/build-gcc.py
@@ +103,5 @@
>  def build_source_dir(prefix, version):
>      return source_dir + '/' + prefix + version
>  
>  binutils_version = "2.21.1"
> +glibc_version = "2.13" #FIXME: should probably use 2.5.1

Yup. 2.13 sounds too old. The slaves have glibc-2.5-12 installed. r+ with this fixed unless you explicitly want to use 2.13

Comment 24

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

You mean too *new*, right?
My hope was to push this first since it works and then backport any patches needed to build 2.5 with currentish gcc.

Comment 25

[Rail Aliiev [:rail]]

(In reply to Rafael Ávila de Espíndola (:espindola) from comment #24)
> You mean too *new*, right?
> My hope was to push this first since it works and then backport any patches
> needed to build 2.5 with currentish gcc.

I mean I prefer 2.5, because it used by our current infrastructure. Using 2.13 shouldn't be a problem though.

Comment 26

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/3ca66b666f85

Yes, we will probably have to switch to 2.5. This is just a step in the right direction.

Comment 27

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: Always extract source and build in the same directory

A particular issue in making reproducible builds is that the source and build directories show up in the debug info.

While it might be possible to have a more elegant solution, this patch implement the simple one: always use the same directory.

Comment 28

[Ed Morley [:emorley]]

(In reply to Rafael Ávila de Espíndola (:espindola) from comment #26)
> https://hg.mozilla.org/integration/mozilla-inbound/rev/3ca66b666f85
> 
> Yes, we will probably have to switch to 2.5. This is just a step in the
> right direction.

https://hg.mozilla.org/mozilla-central/rev/3ca66b666f85

Leaving open for remaining work.

Comment 29

[Rail Aliiev [:rail]]

Comment on attachment 591353 [details] [diff] [review]
Always extract source and build in the same directory

Review of attachment 591353 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with the following fix:

::: build/unix/build-toolchain/build-gcc.py
@@ +99,5 @@
>  ##############################################
>  
> +# The directories end up in the debug info, so the easy way of getting
> +# a reproducible build is to run it in a know absolute directory.
> +base_dir = "/tmp/moz-toolchain"

Hmmm, this may be a problem because /tmp is not a separate mount.
Can you use /builds/slave/moz-toolschain so it will be purged by automation?

Comment 30

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/a90aec7e5250

Comment 31

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: Build glibc with the new gcc

This patch has two small changes:
*) Don't build the c++ compiler on the first stage, saving some time
*) Build glibc with the compiler we just built (i.e. same stage). This is so that
  *) We only have to worry about glibc building with one gcc version
  *) The last glibc is built with at full compiler. In a future patch I will have to strip down the stage 1 gcc a bit more.

Comment 32

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/a90aec7e5250

Not sure if this is done yet, leaving open.

Comment 33

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/6f3b4c36b285

Comment 34

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: Use the built libc when linking

This patch adds a patch that was missing from hg (sorry about that) and changes the glibc build to be installed in $prefix/lib64. This is where gcc looks first, and so makes sure we use our glibc when linking, no just for headers.

Comment 35

[Marco Bonardo [:mak]]

https://hg.mozilla.org/mozilla-central/rev/6f3b4c36b285

Comment 36

[Joe Drew (not getting mail)]

https://hg.mozilla.org/mozilla-central/rev/7a0a7c36def8

Comment 37

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: make sure the libc.so script doesn't refer to build dir

libc.so is a linker script, and without this patch it points to the build directory. With this patch it uses a relative path so that the correct objects are used after the build directory is deleted.

Comment 38

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/e42f47918faf

Comment 39

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/e42f47918faf

Not sure if this needs to be left open; but doing so seeing how many times it's been resolved and then reverted. If there are any more landings on inbound, adding "[leave open]" to the whiteboard will prevent people merging from resolving - if that helps? :-)

Comment 40

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: Start dowgrading glibc

For glibc 2.12.2 all that was needed was manually running autoconf.

Comment 41

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/e19de454a479

Comment 42

[Marco Bonardo [:mak]]

https://hg.mozilla.org/mozilla-central/rev/e19de454a479

Comment 43

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: downgrade glibc to 2.11.1

The patch is a bit big because glibc 2.11.1 doesn't build with make 3.82, so we have to build make 3.81 first.

Comment 44

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/769f2cdb6ba6

Comment 45

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: Downgrade glibc to 2.10.1.

Downgrade clang to 2.10.1.

This time all that was needed was remove the broken version check that think that 2.21 < 2.13.

Comment 46

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/8610c2c3f8b9

Comment 47

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: Downgrade clang to 2.9

This required backporting patches that avoid the extremely brittle linker script patching that old glibc's used to do.

Comment 48

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=ae678c91e86b

Comment 49

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: dowgrade glibc to 2.7

Surprisingly all that was needed this time was rebasing the patch.

Comment 50

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/d56d6d85775c

Comment 51

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: dowgrade glibc to 2.6.1

getting close :-)

Comment 52

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/ae678c91e86b
https://hg.mozilla.org/mozilla-central/rev/d56d6d85775c

Comment 53

[Ed Morley [:emorley]]

and:
https://hg.mozilla.org/mozilla-central/rev/8610c2c3f8b9
https://hg.mozilla.org/mozilla-central/rev/769f2cdb6ba6
https://hg.mozilla.org/mozilla-central/rev/d56d6d85775c

Comment 54

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/747e04f2beb8

Comment 55

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: dowgrade glibc to 2.5.1, fix reproducibility problem

With this patch we are finally at glibc 2.5.1. The patch also includes a fix (adding -p to gzip) to make the build reproducible again.

Comment 56

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/8baf029a9616

Comment 57

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: fix some reproducibility problems

3 small fixes to make the results more reproducible:

*    Build our own linux headers.
*    Disable multilib as we don't build a 32 bit glibc.
*    Build c++ on stage1 as the binutils configures wants a c++ preprocessor.

Comment 58

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/747e04f2beb8
https://hg.mozilla.org/mozilla-central/rev/8baf029a9616

Comment 59

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/dafe158fdbe7

Comment 60

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/dafe158fdbe7

Comment 61

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: build unifdef

We need to build unifdef since the version in centos and the one in fedora 16 produce different results.

Comment 62

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Comment on attachment 600242 [details] [diff] [review]
build unifdef

https://hg.mozilla.org/integration/mozilla-inbound/rev/a5fb523f4916

Comment 63

[Marco Bonardo [:mak]]

https://hg.mozilla.org/mozilla-central/rev/a5fb523f4916

Comment 64

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: swap the glibc and gcc build order during stage2

This avoids libstdc++ being built with the system glibc headers.

Comment 65

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/c75d0a91f7ed

Comment 66

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: disable fixinc and point FLAGS_FOR_TARGET to our headers

This patch does two things
* disable gcc's fixinc. It was copying system headers into the toolchain.tar file
* point FLAGS_FOR_TARGET to the correct header install dir. Stage2 was still using the glibc headers from /usr/include

With these fixes the last header difference is auto-host.h.

Comment 67

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/01ef08fb102d

Comment 68

[Matt Brubeck (:mbrubeck)]

https://hg.mozilla.org/mozilla-central/rev/c75d0a91f7ed

Comment 69

[Matt Brubeck (:mbrubeck)]

https://hg.mozilla.org/mozilla-central/rev/01ef08fb102d

Comment 70

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: Disable lto

Without --enable-lto or --disable-lto configure will enable it if it finds libelf-dev installed, which makes the build harder to reproducible.

Comment 71

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/94d710277cd4

Comment 72

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: Set PATH when building stage2

gcc's configure runs ldd to find glibc's version. If we don't set the PATH then it find the host glibc version.

Comment 73

[Ed Morley [:emorley]]

(In reply to Rafael Ávila de Espíndola (:espindola) from comment #71)
> https://hg.mozilla.org/integration/mozilla-inbound/rev/94d710277cd4

https://hg.mozilla.org/mozilla-central/rev/94d710277cd4

Comment 74

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/186b88e3ade8

Comment 75

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/186b88e3ade8

Comment 76

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: disable zlib when building binutils

It is only used for compressed debug info which we don't use and is normally enabled iff /usr/include/zlib.h is found.

Comment 77

[Marco Bonardo [:mak]]

https://hg.mozilla.org/mozilla-central/rev/7ee9b08c2785

Comment 78

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: make lib a symlink to lib64

gcc (or its build system) has a bug in that xgcc will look at inst/lib but not inst/lib64, so libraries built by gcc during its build process use the system libc.

The easy way to fix this is to just create a link. With this libiberty.a built on centos5 becomes identical to the one built on fedora16.

Comment 79

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/d6871d8255cc

Comment 80

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/d6871d8255cc

Comment 81

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: backport libtool's 74c8993c178a1386ea5e2363a01d919738402f30

This patch backports part of libtool's 74c8993c178a1386ea5e2363a01d919738402f30 commit to the copy in gcc 4.5. With this we produce identical libstdc++.a in centos and fedora.

Comment 82

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/9c9dc2ee76ef

Leaving open per the whiteboard...

Comment 83

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: do a 3 step bootstrap

In the two step bootstrap we got all the .a files identical. Unfortunately, given the order we have to build gcc and glibc, stage1 .a files end up being linked in stage2 .so files.

By doing a 3 step bootstrap we get a lot more files with identical md5s. For example, simple utilities like sln are now bit by bit identical.

Comment 84

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Comment on attachment 623915 [details] [diff] [review]
do a 3 step bootstrap

https://hg.mozilla.org/integration/mozilla-inbound/raw-rev/de94a42610a8

Comment 85

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: build gawk

glibc's build uses gawk and the versions in centos 5 and fedora 16 produce different results. With this we now get an identical libc-2.5.1.so.

Comment 86

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/de94a42610a8

Comment 87

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Comment on attachment 624268 [details] [diff] [review]
build gawk

https://hg.mozilla.org/integration/mozilla-inbound/rev/71f88d947796

Comment 88

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/71f88d947796

Comment 89

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Attachment: build zlib

Binutils' configure tries to add -lz to the link line if it can. That fails on fedora 16 because the system one depends on a newer libc than the one we are using.

By building zlib -lz gets used in both the centos and fedora builds.

Comment 90

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Comment on attachment 624740 [details] [diff] [review]
build zlib

https://hg.mozilla.org/integration/mozilla-inbound/rev/bc8c8f0afdb7

Comment 91

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/bc8c8f0afdb7

Comment 92

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Lets call this fixed by tooltool. We still have to move the gcc packages to use it, but it is less confusing to have a bug for that.