Closed Bug 1179818 Opened 9 years ago Closed 9 years ago

Get glibc from tooltool

Categories

(Firefox Build System :: General, defect)

Product:
Firefox Build System
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: dustin, Assigned: glandium)

References

Details

(Whiteboard: [build])

Attachments

(2 files, 3 obsolete files)

Add glibc to the GCC tooltool package
4.09 KB, patch
dustin
: feedback-
Details | Diff | Splinter Review
Remove '|| exit 1' from build-gcc.sh now that there is 'set -e'
2.86 KB, patch
tbsaunde
: review+
Details | Diff | Splinter Review 
We currently build against the versions of these libraries installed in the build environment.  Especially for glibc, this is problematic since updates to the build environment may require updates to these libraries, yet we want to keep the library versions stable to avoid any unintended compatibility changes in shipped versions of Firefox.

It seems the best solution is to install those libs from tooltool, so that (a) we can have stable versions and (b) updates can be handled in-tree.
glandium suggested we probably wanted to build our own glibc at a specific version, then build our gcc, pointing it at that glibc, then put them both in tooltool.
Yeah, I think we need to separate this out in two bugs:
- one to add glibc to the gcc tarball
- one for gtk+2/glib (gtk+3 is already in tooltool along libraries it needs)
Also note that the script used to create the gcc tarballs used to build the glibc, so it should "only" be a matter of digging in mozilla-central's history and forward-porting.
This currently has all desktop jobs in TC failing.  I'll see if I can take a whack at it, on the heels of my "success" with java in bug 1161075..
Assignee: nobody → dustin
Looks like that was
  http://hg.mozilla.org/mozilla-central/file/2d495522a53a/build/unix/build-toolchain
and the glibc build was removed in bug 913442 because it was easier that way.
So, old version:
  http://hg.mozilla.org/mozilla-central/file/2d495522a53a/build/unix/build-toolchain/build-gcc.py
which downloaded all of the dependencies directly, combined them, and then built gcc using the usual three-stage process.

New version:
  http://hg.mozilla.org/mozilla-central/file/f7e1f596d57d/build/unix/build-gcc/build-gcc.sh
which downloads and builds binutils, then downloads gcc, runs download_prerequisites, and builds that.

I'm curious if this marks a loss of the three-stage build process, and whether that was problematic, but I suppose the question is unrelated to this bug.
A few notes from Trevor:
 - gcc's build process does the three-stage thing itself
 - the old Python script wasn't entirely functional, so the rewrite was from first principles: "I need to build a compiler"
From poking around, it looks like the Python script tried to interleave building glibc with building gcc, presumably feeling it important to build a gcc that could run against an ancient glibc.  I don't think we need to do that, if we can manage to set up the mozconfigs so that we run the *build* linker against the glibc from tooltool, while the runtime linker looks to the system glibc.
Attached file build-glibc.sh (obsolete) — 
This is the script I'm using, building with the desktop-build image (Trusty).  However, it's failing fairly late into the 'make' process, with the following.  (the warnings seem common to every gcc invocation)

----

gcc ../sysdeps/unix/sysv/linux/syslog.c -c -std=gnu99 -fgnu89-inline -O2 -Wall -Winline -Wwrite-strings -fmerge-all-constants -g -Wstrict-prototypes   -fexceptions   -I../include -I/home/worker/workspace/glibc-objdir/misc -I/home/worker/workspace/glibc-objdir -I../sysdeps/x86_64/elf -I../nptl/sysdeps/unix/sysv/linux/x86_64 -I../sysdeps/unix/sysv/linux/x86_64 -I../sysdeps/unix/sysv/linux/wordsize-64 -I../nptl/sysdeps/unix/sysv/linux -I../nptl/sysdeps/pthread -I../sysdeps/pthread -I../sysdeps/unix/sysv/linux -I../sysdeps/gnu -I../sysdeps/unix/common -I../sysdeps/unix/mman -I../sysdeps/unix/inet -I../nptl/sysdeps/unix/sysv -I../sysdeps/unix/sysv -I../sysdeps/unix/x86_64 -I../nptl/sysdeps/unix -I../sysdeps/unix -I../sysdeps/posix -I../sysdeps/x86_64/fpu -I../sysdeps/x86_64/multiarch -I../nptl/sysdeps/x86_64 -I../sysdeps/x86_64 -I../sysdeps/wordsize-64 -I../sysdeps/ieee754/ldbl-96 -I../sysdeps/ieee754/dbl-64/wordsize-64 -I../sysdeps/ieee754/dbl-64 -I../sysdeps/ieee754/flt-32 -I../sysdeps/ieee754 -I../sysdeps/generic/elf -I../sysdeps/generic -I../nptl  -I.. -I../libio -I.  -D_LIBC_REENTRANT -include ../include/libc-symbols.h       -o /home/worker/workspace/glibc-objdir/misc/syslog.o -MD -MP -MF /home/worker/workspace/glibc-objdir/misc/syslog.o.dt -MT /home/worker/workspace/glibc-objdir/misc/syslog.o 
In file included from ../posix/sys/types.h:26:0,
                 from ../include/sys/types.h:1,
                 from ../misc/syslog.c:34,
                 from ../sysdeps/unix/sysv/linux/syslog.c:10:
../include/features.h:322:0: warning: "__STDC_IEC_559__" redefined [enabled by default]
 #define __STDC_IEC_559__  1
 ^
In file included from <command-line>:0:0:
/usr/include/stdc-predef.h:41:0: note: this is the location of the previous definition
 # define __STDC_IEC_559__  1
 ^
In file included from ../posix/sys/types.h:26:0,
                 from ../include/sys/types.h:1,
                 from ../misc/syslog.c:34,
                 from ../sysdeps/unix/sysv/linux/syslog.c:10:
../include/features.h:323:0: warning: "__STDC_IEC_559_COMPLEX__" redefined [enabled by default]
 #define __STDC_IEC_559_COMPLEX__ 1
 ^
In file included from <command-line>:0:0:
/usr/include/stdc-predef.h:49:0: note: this is the location of the previous definition
 # define __STDC_IEC_559_COMPLEX__ 1
 ^
In file included from ../posix/sys/types.h:26:0,
                 from ../include/sys/types.h:1,
                 from ../misc/syslog.c:34,
                 from ../sysdeps/unix/sysv/linux/syslog.c:10:
../include/features.h:326:0: warning: "__STDC_ISO_10646__" redefined [enabled by default]
 #define __STDC_ISO_10646__  200009L
 ^
In file included from <command-line>:0:0:
/usr/include/stdc-predef.h:54:0: note: this is the location of the previous definition
 #define __STDC_ISO_10646__  201103L
 ^
In file included from ../sysdeps/unix/sysv/linux/syslog.c:10:0:
../misc/syslog.c: In function '__vsyslog_chk':
../misc/syslog.c:144:9: warning: variable 'prioff' set but not used [-Wunused-but-set-variable]
  size_t prioff, msgoff;
         ^
In file included from <command-line>:0:0:
../misc/syslog.c:123:30: error: inlining failed in call to always_inline 'syslog': function not inlinable
 ldbl_strong_alias (__syslog, syslog)
                              ^
./../include/libc-symbols.h:101:26: note: in definition of macro '_strong_alias'
   extern __typeof (name) aliasname __attribute__ ((alias (#name)));
                          ^
../sysdeps/generic/math_ldbl_opt.h:12:44: note: in expansion of macro 'strong_alias'
 #define ldbl_strong_alias(name, aliasname) strong_alias (name, aliasname)
                                            ^
../misc/syslog.c:123:1: note: in expansion of macro 'ldbl_strong_alias'
 ldbl_strong_alias (__syslog, syslog)
 ^
In file included from ../sysdeps/unix/sysv/linux/syslog.c:10:0:
../misc/syslog.c:155:9: error: called from here
   syslog(INTERNALLOG,
         ^
make[2]: *** [/home/worker/workspace/glibc-objdir/misc/syslog.o] Error 1
make[2]: Leaving directory `/home/worker/workspace/glibc-2.12.2/misc'
make[1]: *** [misc/subdir_lib] Error 2
make[1]: Leaving directory `/home/worker/workspace/glibc-2.12.2'
make: *** [all] Error 2
Glandium suggests that new gcc + old glibc = bad.  So I'll re-try this with the gcc used to build Firefox.
Or, potentially easier than scrapping together a toolchain that can build this ancient warhorse, we could just pluck the library files from the CentOS 6.2 RPM.

Glandium points out that there are licensing issues here, where redistributing GPL'd binaries requires redistributing the source, too.  That would include having some kind of human-discernible link from the object files (the tarball in tooltool) to the source (something else in tooltool, most likely).
Don't we ostensibly already have this problem distributing gcc binaries in tooltool?
Yes -- and should probably be fixed too. My thinking was to just include in the object tarball a "source.tt" pointing to the source tarball, and then upload both together.
OK, after a lot of false starts, this is what I've got put together:
 https://bitbucket.org/djmitche/mozilla-central/commits/bb9f148d6d8f
basically, extract everything from the glibc RPMs and dump it into a tarball.

The task is
  https://tools.taskcluster.net/task-inspector/#0WxoEL0YRC-NLl9M_2QlGw/0

Of the list of files in the log, can I limit things to just the libs and the headers?
Flags: needinfo?(mh+mozilla)
This needs to be mixed with the gcc package otherwise the setup is going to be error-prone. We also need both a 32 bits and a 64 bits glibc. And to make sure the gcc binaries can run without having to fiddle with LD_LIBRARY_PATHs.

> Of the list of files in the log, can I limit things to just the libs and the headers?

Depends what you call "just the libs". There are non lib files that are in the lib directory and that are essential.

It seems to me it would all be simpler if you just built everything (glibc+gcc) in a centos 6 docker image. I get that you're trying to have docker containers with fresh distros to build firefox (although I'd actually question that too), but I don't see a reason all builds of all things /need/ to be on fresh distros.
Flags: needinfo?(mh+mozilla)
This script repackages both bitlength glibc's.

I'm missing the link between gcc and glibc.  From what I understand talking to Trevor, building both together is difficult and doesn't actually work very well, so in the latest iteration of building gcc, the side-build of glibc has been dropped.  It seems like the benefit to building both together is that gcc itself runs against glibc -- but I don't see why that's important (and as you note, doing so requires fiddling with LD_LIBRARY_PATH, since gcc won't be at a pre-defined path).

If building in a CentOS 6 docker image is easier, then there is some dependency in the result on CentOS 6, and it is not portable to Ubuntu.  There will come a time, very soon, when it becomes very difficult to run CentOS 6 at all, and I'd like to migrate away from it before that time arrives.
(In reply to Dustin J. Mitchell [:dustin] from comment #17)
> This script repackages both bitlength glibc's.
> 
> I'm missing the link between gcc and glibc.  From what I understand talking
> to Trevor, building both together is difficult and doesn't actually work
> very well, so in the latest iteration of building gcc, the side-build of
> glibc has been dropped.  It seems like the benefit to building both together
> is that gcc itself runs against glibc -- but I don't see why that's
> important (and as you note, doing so requires fiddling with LD_LIBRARY_PATH,
> since gcc won't be at a pre-defined path).

With the disclaimer I'm no expert on building cross tool chains, I believe gcc may actually "build against" two different libc's the one its binaries are linked against, but it also wants to know about the one for the target and which the libraries gcc ships are linked against.  You are right gcc's binaries shouldn't need to use a old libc, but it does seem safest to have gcc build its libraries against the old glibc.

> If building in a CentOS 6 docker image is easier, then there is some
> dependency in the result on CentOS 6, and it is not portable to Ubuntu. 
> There will come a time, very soon, when it becomes very difficult to run
> CentOS 6 at all, and I'd like to migrate away from it before that time
> arrives.

With the admission that I've been doing e10s to the exclusion of all else for the last year and so haven't been paying attention and so don't know the reasons for wanting to move builds off centos6 it seems to me by far the easiest thing to do is keep building firefox on centos6 until we decide to stop supporting running firefox there.  And I wouldn't expect it will become hard to run centos6 containers before we stop supporting it, but that's certainly said from a position of ignorance.
(In reply to Dustin J. Mitchell [:dustin] from comment #17)
> This script repackages both bitlength glibc's.
> 
> I'm missing the link between gcc and glibc.  From what I understand talking
> to Trevor, building both together is difficult and doesn't actually work
> very well, so in the latest iteration of building gcc, the side-build of
> glibc has been dropped.  It seems like the benefit to building both together
> is that gcc itself runs against glibc -- but I don't see why that's
> important (and as you note, doing so requires fiddling with LD_LIBRARY_PATH,
> since gcc won't be at a pre-defined path).

There is the glibc gcc needs to run itself, and there is the glibc gcc links against by default. If you get the latter separately from gcc, then you need to pass flags to make gcc use it, and you risk those flags being skipped by some weird part of the build system.

As for the old script, what was difficult with it is that it tried to do *reproducible* builds of the whole toolchain package, where the same source would generate the exact same bits. /That/ was the difficult part.

> If building in a CentOS 6 docker image is easier, then there is some
> dependency in the result on CentOS 6, and it is not portable to Ubuntu. 

I can run the gcc from tooltool, which was built in a CentOS 6 chroot, just fine on my Debian system.
Something I just thought about: it's desirable to keep being able to build with the current environment we're using (Centos 6) to allow try builds of older changesets with the environment they were originally built with.
(In reply to Mike Hommey [:glandium] from comment #20)
> Something I just thought about: it's desirable to keep being able to build
> with the current environment we're using (Centos 6) to allow try builds of
> older changesets with the environment they were originally built with.

Although, we currently suck at that.
glibc maintains backwards-ABI compatibility, right? So if we build our toolchain against an older glibc then it should have no problem running on a newer one?
We're not keeping Buildbot around, so the capability to build in the "old" (today's) environment is going to be lost no matter what.  Looking forward, we'll do a substantially better job of preserving build environments (Linux only) as long as the docker images hang around and are still compatible with existing hardware and kernels.

> It seems to me it would all be simpler if you just built everything
> (glibc+gcc) in a centos 6 docker image.

In fact, that's exactly what this latest patch does, except that instead of building glibc, it just takes an already-built glibc from that environment (in the form of upstream RPMs).

The result is rather large, and most of that bulk is in l10n files and other stuff that I *suspect* can be omitted.  I suppose the thing to do is demonstrate the concept first -- link successfully against the big glibc tarball -- then start whittling files away until things stop working.
Repacked in https://tools.taskcluster.net/task-inspector/#8SW8Tj79QG2dAGZaVLRWdA/0 without /usr/share/info, but with everything else left in.

[
{
"size": 9478200,
"visibility": "public",
"digest": "4e14898cad0fbb9e8c023b1cfca0bf3bddd99e56532810063420bc56dea7cd05bf8fe115945c061c48a79945c91925afc947bbaf2fbf51e062cf7f26456720b7",
"algorithm": "sha512",
"unpack": "true",
"filename": "glibc-2.12-1.47.el6-i686.tar.xz"
},
{
"size": 15951482,
"visibility": "public",
"digest": "5d7813fc5ecc656e8aef849a1b5602c4662b657aaaf7a7470695e285b1af6ee261739cd5c65a5be85115cf791346ea884c2623fc7e1c8984fe47d0d0ee066972",
"algorithm": "sha512",
"filename": "glibc-2.12-1.47.el6.src.rpm"
},
{
"size": 9567204,
"visibility": "public",
"digest": "8f780d9ffc3cf58b0f22420cc1ecaec03bee82eb14e030b4fa414ecaf54de2414bbb6ba2e5f72c0a7374ae9eb1ac5380fb1a6ead41609a7357b19dbb81d4a087",
"algorithm": "sha512",
"unpack": "true",
"filename": "glibc-2.12-1.47.el6-x86_64.tar.xz"
}
]
The point is, it needs to be part of the gcc package. Otherwise it's cumbersome to setup and a source of future problems.
Toolchains seem to be a black art -- the folks who understand them don't seem to like to write documentation.  All I can find online are wiki pages of "this is how I built my toolchain" with lots of "edit this file and comment this out or a later step will blow up" and not a lot of explanation.

There's also no documentation of the symbol versioning that glibc uses.  The best I can find is https://www.kernel.org/pub/software/libs/glibc/hjl/compat/ which isn't accurate (AFAICT it was about a draft proposal for how to handle versioning).

So I'm really stabbing in the dark.  Arguably I'm not the best person to work on this.

Things I'm thinking of exploring:
 * resuscitate the old Python script in-tree
 * use crosstools-ng to build a native toolset (not clear this is supported)
[INFO ]  Performing some trivial sanity checks
[INFO ]  Build started 20150716.154907
[INFO ]  Building environment variables
[WARN ]  Directory '/home/worker/src' does not exist.
[WARN ]  Will not save downloaded tarballs to local storage.
[EXTRA]  Preparing working directories
[EXTRA]  Installing user-supplied crosstool-NG configuration
[EXTRA]  =================================================================
[EXTRA]  Dumping internal crosstool-NG configuration
[EXTRA]    Building a toolchain for:
[EXTRA]      build  = x86_64-unknown-linux-gnu
[EXTRA]      host   = x86_64-unknown-linux-gnu
[EXTRA]      target = x86_64-unknown-linux-gnu
[EXTRA]  Dumping internal crosstool-NG configuration: done in 0.07s (at 00:02)
[INFO ]  =================================================================
[INFO ]  Retrieving needed toolchain components' tarballs
[EXTRA]    Retrieving 'linux-4.0.4'
[EXTRA]    Retrieving 'gmp-6.0.0a'
[EXTRA]    Retrieving 'mpfr-3.1.2'
[EXTRA]    Retrieving 'isl-0.14'
[EXTRA]    Retrieving 'mpc-1.0.2'
[EXTRA]    Retrieving 'binutils-2.25'
[EXTRA]    Retrieving 'gcc-5.1.0'
[EXTRA]    Retrieving 'glibc-2.21'
[EXTRA]    Retrieving 'gdb-7.9.1'
[INFO ]  Retrieving needed toolchain components' tarballs: done in 54.60s (at 00:57)
[INFO ]  =================================================================
[INFO ]  Extracting and patching toolchain components
[EXTRA]    Extracting 'linux-4.0.4'
[EXTRA]    Patching 'linux-4.0.4'
[EXTRA]    Extracting 'gmp-6.0.0a'
[EXTRA]    Patching 'gmp-6.0.0a'
[EXTRA]    Extracting 'mpfr-3.1.2'
[EXTRA]    Patching 'mpfr-3.1.2'
[EXTRA]    Extracting 'isl-0.14'
[EXTRA]    Patching 'isl-0.14'
[EXTRA]    Extracting 'mpc-1.0.2'
[EXTRA]    Patching 'mpc-1.0.2'
[EXTRA]    Extracting 'binutils-2.25'
[EXTRA]    Patching 'binutils-2.25'
[EXTRA]    Extracting 'gcc-5.1.0'
[EXTRA]    Patching 'gcc-5.1.0'
[EXTRA]    Extracting 'glibc-2.21'
[EXTRA]    Patching 'glibc-2.21'
[EXTRA]    Extracting 'gdb-7.9.1'
[EXTRA]    Patching 'gdb-7.9.1'
[INFO ]  Extracting and patching toolchain components: done in 94.81s (at 02:32)
[INFO ]  =================================================================
[INFO ]  Installing GMP for host
[EXTRA]    Configuring GMP
[EXTRA]    Building GMP
[EXTRA]    Installing GMP
[INFO ]  Installing GMP for host: done in 62.25s (at 03:34)
[INFO ]  =================================================================
[INFO ]  Installing MPFR for host
[EXTRA]    Configuring MPFR
[EXTRA]    Building MPFR
[EXTRA]    Installing MPFR
[INFO ]  Installing MPFR for host: done in 22.73s (at 03:57)
[INFO ]  =================================================================
[INFO ]  Installing ISL for host
[EXTRA]    Configuring ISL
[EXTRA]    Building ISL
[EXTRA]    Installing ISL
[INFO ]  Installing ISL for host: done in 35.04s (at 04:32)
[INFO ]  =================================================================
[INFO ]  Installing MPC for host
[EXTRA]    Configuring MPC
[EXTRA]    Building MPC
[EXTRA]    Installing MPC
[INFO ]  Installing MPC for host: done in 8.40s (at 04:40)
[INFO ]  =================================================================
[INFO ]  Installing binutils for host
[EXTRA]    Configuring binutils
[EXTRA]    Building binutils
[EXTRA]    Installing binutils
[EXTRA]    Installing ld wrapper
[INFO ]  Installing binutils for host: done in 261.87s (at 09:02)
[INFO ]  =================================================================
[INFO ]  Installing pass-1 core C compiler
[EXTRA]    Configuring gcc
[EXTRA]    Building gcc
[EXTRA]    Installing gcc
[INFO ]  Installing pass-1 core C compiler: done in 570.49s (at 18:32)
[INFO ]  =================================================================
[INFO ]  Installing kernel headers
[EXTRA]    Installing kernel headers
[EXTRA]    Checking installed headers
[INFO ]  Installing kernel headers: done in 19.35s (at 18:52)
[INFO ]  =================================================================
[INFO ]  Installing C library headers & start files
[EXTRA]    Configuring C library
[EXTRA]    Installing C library headers
[EXTRA]    Installing C library start files
[INFO ]  Installing C library headers & start files: done in 24.80s (at 19:17)
[INFO ]  =================================================================
[INFO ]  Installing pass-2 core C compiler
[EXTRA]    Configuring gcc
[EXTRA]    Building gcc
[EXTRA]    Installing gcc
[INFO ]  Installing pass-2 core C compiler: done in 614.32s (at 29:31)
[INFO ]  =================================================================
[INFO ]  Installing C library
[EXTRA]    Configuring C library
[EXTRA]    Building C library
[EXTRA]    Installing C library
[INFO ]  Installing C library: done in 420.73s (at 36:32)
[INFO ]  =================================================================
[INFO ]  Installing final compiler
[EXTRA]    Configuring gcc
[EXTRA]    Building gcc
[43:53] \ ^R
[EXTRA]    Installing gcc
[INFO ]  Installing final compiler: done in 752.10s (at 49:04)
[INFO ]  =================================================================
[INFO ]  Installing cross-gdb
[EXTRA]    Configuring cross-gdb
[EXTRA]    Building cross-gdb
[EXTRA]    Installing cross-gdb
[EXTRA]    Installing '.gdbinit' template
[INFO ]  Installing cross-gdb: done in 195.48s (at 52:19)
[INFO ]  =================================================================
[INFO ]  Installing gdbserver
[EXTRA]    Configuring gdbserver
[EXTRA]    Building gdbserver
[EXTRA]    Installing gdbserver
[INFO ]  Installing gdbserver: done in 62.11s (at 53:21)
[INFO ]  =================================================================
[INFO ]  Cleaning-up the toolchain's directory
[INFO ]    Stripping all toolchain executables
[EXTRA]    Installing the populate helper
[EXTRA]    Installing a cross-ldd helper
[EXTRA]    Creating toolchain aliases
[EXTRA]    Removing access to the build system tools
[EXTRA]    Removing installed documentation
[INFO ]  Cleaning-up the toolchain's directory: done in 6.56s (at 53:28)
[INFO ]  Build completed at 20150716.164234
[INFO ]  (elapsed: 53:27.62)
[INFO ]  Finishing installation (may take a few seconds)...
OK, I was able to build a hello-world script using that toolchain.  It appears that it is correctly using the toolchain glibc during link (based on an strace -eopen of the gcc run).  The resulting binary uses the system glibc by default, but if run with LD_LIBRARY_PATH set to point to the toolchain, runs against that glibc.  /home/worker/x-tools is about 200M.
Running against the system glibc should be fine, as long as it produces binaries linked against the toolchain glibc so that they work on e.g. CentOS 6.
I added

export CC=/home/worker/x-tools/x86_64-unknown-linux-gnu/bin/x86_64-unknown-linux-gnu-gcc
export CXX=/home/worker/x-tools/x86_64-unknown-linux-gnu/bin/x86_64-unknown-linux-gnu-g++

to build/mozconfig.common.override (temporarily; this probably belongs in build/unix/mozconfig.linux) to see what happens.  One important question that remains is, is the install relocatable -- that is, if I move x-tools within ~/workspace, will it still work?
19:51:57     INFO -  configure:21968: checking GSTREAMER_LIBS
19:51:57     INFO -  configure:22015: /usr/bin/ccache /home/worker/x-tools/x86_64-unknown-linux-gnu/bin/x86_64-unknown-linux-gnu-gcc -o conftest  -std=gnu99 -fgnu89-inline -fno-strict-aliasing -ffunction-sections -fdata-sections -fno-math-errno -pthread  -lpthread  -Wl,-z,noexecstack -Wl,-z,text -Wl,--build-id -pthread -lgstapp-0.10 -lgstbase-0.10 -lgstreamer-0.10 -lgobject-2.0 -lgmodule-2.0 -lgthread-2.0 -lxml2 -lglib-2.0   -lgstvideo-0.10 conftest.c -ldl  1>&5
19:51:57     INFO -  /home/worker/x-tools/x86_64-unknown-linux-gnu/lib/gcc/x86_64-unknown-linux-gnu/5.1.0/../../../../x86_64-unknown-linux-gnu/bin/ld.bfd: cannot find -lgstapp-0.10
19:51:57     INFO -  /home/worker/x-tools/x86_64-unknown-linux-gnu/lib/gcc/x86_64-unknown-linux-gnu/5.1.0/../../../../x86_64-unknown-linux-gnu/bin/ld.bfd: cannot find -lgstbase-0.10
19:51:57     INFO -  /home/worker/x-tools/x86_64-unknown-linux-gnu/lib/gcc/x86_64-unknown-linux-gnu/5.1.0/../../../../x86_64-unknown-linux-gnu/bin/ld.bfd: cannot find -lgstreamer-0.10
19:51:57     INFO -  /home/worker/x-tools/x86_64-unknown-linux-gnu/lib/gcc/x86_64-unknown-linux-gnu/5.1.0/../../../../x86_64-unknown-linux-gnu/bin/ld.bfd: cannot find -lgobject-2.0
19:51:57     INFO -  /home/worker/x-tools/x86_64-unknown-linux-gnu/lib/gcc/x86_64-unknown-linux-gnu/5.1.0/../../../../x86_64-unknown-linux-gnu/bin/ld.bfd: cannot find -lgmodule-2.0
19:51:57     INFO -  /home/worker/x-tools/x86_64-unknown-linux-gnu/lib/gcc/x86_64-unknown-linux-gnu/5.1.0/../../../../x86_64-unknown-linux-gnu/bin/ld.bfd: cannot find -lgthread-2.0
19:51:57     INFO -  /home/worker/x-tools/x86_64-unknown-linux-gnu/lib/gcc/x86_64-unknown-linux-gnu/5.1.0/../../../../x86_64-unknown-linux-gnu/bin/ld.bfd: cannot find -lxml2
19:51:57     INFO -  /home/worker/x-tools/x86_64-unknown-linux-gnu/lib/gcc/x86_64-unknown-linux-gnu/5.1.0/../../../../x86_64-unknown-linux-gnu/bin/ld.bfd: cannot find -lglib-2.0
19:51:57     INFO -  /home/worker/x-tools/x86_64-unknown-linux-gnu/lib/gcc/x86_64-unknown-linux-gnu/5.1.0/../../../../x86_64-unknown-linux-gnu/bin/ld.bfd: cannot find -lgstvideo-0.10
...
19:51:57     INFO -  configure: error: gstreamer-plugins-base found, but no libgstvideo. Something has gone terribly wrong. Try reinstalling gstreamer-plugins-base; failing that, disable the gstreamer backend with --disable-gstreamer.

yes, "terribly wrong" :)

worker@taskcluster-worker:~$ find /usr/ -name 'libgmodule*'
/usr/lib/x86_64-linux-gnu/libgmodule-2.0.so
/usr/lib/x86_64-linux-gnu/libgmodule-2.0.so.0.4002.0
/usr/lib/x86_64-linux-gnu/libgmodule-2.0.so.0
/usr/lib/x86_64-linux-gnu/libgmodule-2.0.a

So I'll try again with LDFLAGS="-L /usr/lib/x86_64-linux-gnu $LDFLAGS"
20:08:22     INFO -  /usr/lib/x86_64-linux-gnu/libgstvideo-0.10.so: undefined reference to `orc_program_add_destination'

I am officially totally lost here..
Assignee: dustin → nobody
gtk got handled in bug 1186003.
Summary: Get glibc, gtk+3, gtk+2, glib from tooltool → Get glibc, glib from tooltool
Whiteboard: [build]
Blocks: 1187047
No longer blocks: 1187047
Note that glib is part of the gtk3 tooltool package.
Summary: Get glibc, glib from tooltool → Get glibc from tooltool
Blocks: 1179805
Attached patch PoC patch (obsolete) — — Splinter Review 
With this patch, and some manual fiddling in gcc/lib/libc.so (which I need to translate in the patch), I get the expected result:

$ cat <<EOF >test.c
#include <string.h>
int foo(void *a, void *b, size_t len) {
  return memcpy(a, b, len) == a;
}
EOF
$ gcc -o test.so -shared test.c -fPIC
$ objdump -T test.so | grep memcpy
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.14  memcpy
$ ./gcc/bin/gcc -o test.so -shared test.c -fPIC
$ objdump -T test.so | grep memcpy
0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 memcpy

Now, I just need to figure how to make the same work with -m32.
This builds properly in the CentOS 6 environment we use for linux32 builds, with bison and flex installed, and the resulting package is in tooltool and builds fine on try. The manifest is:

{
"size": 101526252, 
"visibility": "public", 
"digest": "8f8c90a98e6f588c1f0383888728a3da335a06a99412471d9ba7d412deed922f1e5999ec68f98480bb055b80ddad84032b6bcbc08130e55047889a56e0dde01e", 
"algorithm": "sha512", 
"filename": "gcc.tar.xz"
}

Dustin, can you test this package on TC with the Ubuntu env?
Assignee: nobody → mh+mozilla
Attachment #8632331 - Attachment is obsolete: true
Attachment #8641440 - Attachment is obsolete: true
Attachment #8641545 - Flags: review?(tbsaunde+mozbugs)
Attachment #8641545 - Flags: feedback?(dustin)
Heh, forgot to git add the extra patch.
Attachment #8641545 - Attachment is obsolete: true
Attachment #8641545 - Flags: review?(tbsaunde+mozbugs)
Attachment #8641545 - Flags: feedback?(dustin)
Attachment #8641548 - Flags: review?(tbsaunde+mozbugs)
Attachment #8641548 - Flags: feedback?(dustin)
Additional cleanup
Attachment #8641550 - Flags: review?(tbsaunde+mozbugs)
That's as simple as patching the manifest (https://hg.mozilla.org/try/rev/afece77da3cc) and pushing to try:
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=7a0b3d6d7d41&exclusion_profile=false
Well, almost as simple - the job is hidden, hence exclusion_profile=false
*ahem*
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=5f988c69911f&exclusion_profile=false
Unfortunately, we've got multiple bustages of the taskcluster builds right now, so I can't tell if this works or not; it fails due to the gtk+3 patch.  I'll try again with a tree prior to bug 1186003.
Comment on attachment 8641548 [details] [diff] [review]
Add glibc to the GCC tooltool package

OK, this is a meaningful-looking error: the libgstreamer on that host is expecting a newer glibc (2.17):
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=a801fc12156d&exclusion_profile=false

 https://s3-us-west-2.amazonaws.com/taskcluster-public-artifacts/E2SU7rlGTXWgeNmRI9-54g/0/public/logs/live_backing.log
19:21:12     INFO -  configure:21732: checking for alsa
19:21:12     INFO -  configure:21739: checking MOZ_ALSA_CFLAGS
19:21:12     INFO -  configure:21744: checking MOZ_ALSA_LIBS
19:21:12     INFO -  configure:21854: checking for libpulse
19:21:12     INFO -  configure:21861: checking MOZ_PULSEAUDIO_CFLAGS
19:21:12     INFO -  configure:21866: checking MOZ_PULSEAUDIO_LIBS
19:21:12     INFO -  configure:21987: checking for gstreamer-0.10 >= 0.10.25
19:21:12     INFO -                        gstreamer-app-0.10
19:21:12     INFO -                        gstreamer-plugins-base-0.10
19:21:12     INFO -  configure:21998: checking GSTREAMER_CFLAGS
19:21:12     INFO -  configure:22005: checking GSTREAMER_LIBS
19:21:12     INFO -  configure:22052: /usr/bin/ccache /home/worker/workspace/build/src/gcc/bin/gcc -o conftest  -std=gnu99 -fgnu89-inline -fno-strict-aliasing -ffunction-sections -fdata-sections -fno-math-errno -pthread  -lpthread  -Wl,-z,noexecstack -Wl,-z,text -Wl,--build-id -pthread -lgstapp-0.10 -lgstbase-0.10 -lgstreamer-0.10 -lgobject-2.0 -lgmodule-2.0 -lgthread-2.0 -lxml2 -lglib-2.0   -lgstvideo-0.10 conftest.c -ldl  1>&5
19:21:12     INFO -  /usr/lib/x86_64-linux-gnu/libgstreamer-0.10.so: undefined reference to `__fdelt_chk@GLIBC_2.15'
19:21:12     INFO -  /usr/lib/x86_64-linux-gnu/libgstbase-0.10.so: undefined reference to `memcpy@GLIBC_2.14'
19:21:12     INFO -  /usr/lib/x86_64-linux-gnu/libgstreamer-0.10.so: undefined reference to `clock_gettime@GLIBC_2.17'
19:21:12     INFO -  /usr/lib/x86_64-linux-gnu/libgstreamer-0.10.so: undefined reference to `clock_getres@GLIBC_2.17'
19:21:12     INFO -  collect2: error: ld returned 1 exit status
19:21:12     INFO -  configure: failed program was:
19:21:12     INFO -  #line 22045 "configure"
19:21:12     INFO -  #include "confdefs.h"
19:21:12     INFO -  int main() {
19:21:12     INFO -  return 0;
19:21:12     INFO -  ; return 0; }
19:21:12     INFO -  configure: error: gstreamer-plugins-base found, but no libgstvideo. Something has gone terribly wrong. Try reinstalling gstreamer-plugins-base; failing that, disable the gstreamer backend with --disable-gstreamer.

It's possible that putting gstreamer into tooltool as well would fix that.  I still don't have a bug # for that project, so I'm not sure what its status is.  Alternately, switching to a CentOS 6 image may help, too (bug 1189892)
Attachment #8641548 - Flags: feedback?(dustin) → feedback-
Depends on: 1189892
Attachment #8641550 - Flags: review?(tbsaunde+mozbugs) → review+
(In reply to Dustin J. Mitchell [:dustin] from comment #42)
> Comment on attachment 8641548 [details] [diff] [review]
> Add glibc to the GCC tooltool package
> 
> OK, this is a meaningful-looking error: the libgstreamer on that host is
> expecting a newer glibc (2.17):

At least it means the gcc/glibc package works as expected, since it's barfing that those symbol don't exist in the glibc it has.

> It's possible that putting gstreamer into tooltool as well would fix that. 
> I still don't have a bug # for that project

I added it as a dependency to wherever you were asking before. It's bug 947287

> , so I'm not sure what its status
> is.  Alternately, switching to a CentOS 6 image may help, too (bug 1189892)

Switching to a CentOS 6 image would make this gcc/glibc package useless.
Anyways, considering bug 1189892, I'm declaring this wontfix.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → WONTFIX
Attachment #8641548 - Flags: review?(tbsaunde+mozbugs)
No longer blocks: 1164617
Component: Build Config → General
Product: Firefox → Firefox Build System
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: