1578535 - figure out how to provide a recent libstdc++ for aarch64 cross builds

Status: RESOLVED FIXED

(Firefox Build System :: General, task)

Description

[Nathan Froyd [:froydnj]]

I'm seeing issues (bug 1577275 comment 4) where I think our compilers on aarch64-linux are picking up libstdc++ headers that aren't at the version we expect them to be. Our linux64-clang packages, at least, only include bits for x86_64-unknown-linux-gnu, so what I expect is happening is that the headers included in the linux64-clang packages are getting rejected, and clang is simply using whatever appears on the system.

I see two options:

1. Generate all the necessary bits for aarch64-unknown-linux-gnu headers to be included with our GCC packages and then by extension our linux64-clang packages.
2. Install whatever packages we need in our aarch64-linux build image to get the right headers.

Option 2 is expedient, but I think we'd prefer to get away from the build image providing headers that we need. I don't particularly care to go implementing option 1, though maybe we could just do a libstdc++ cross-build on our aarch64-linux build image, and pull that package in somehow?

Opinions, glandium?

Comment 1

[Mike Hommey [:glandium]]

Do we actually care that much about the aarch64-linux builds? I mean, I only really set them up because the fuzzing team was interested in setting up aarch64 linux builds, and that was a first step to get there. And they don't actually use the same base image as the linux64 builds, so they're not actually stuck with an old system libstdc++... as in, we could just switch the docker image for those builds from Debian 9 to Debian 10, which would upgrade libstdc++ from 6.3 to 8.3... (that's true of all builds currently based on Debian 9 images, BTW, eventually, we should upgrade all of them to Debian 10, and we should be able to do it any time)

Comment 2

[Nathan Froyd [:froydnj]]

Of course, when we make the obvious switches, we wind up with configure not running:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=70a27bf9eff598f30bbed0a39fd3983a4aca31c9&selectedJob=264970867

INFO -  DEBUG: configure:4040: /builds/worker/fetches/sccache/sccache /builds/worker/fetches/clang/bin/clang -std=gnu99 --target=aarch64-linux-gnu -o conftest  -fcrash-diagnostics-dir=/builds/worker/artifacts -Qunused-arguments    conftest.c  1>&5
INFO -  DEBUG: /builds/worker/fetches/binutils/bin/ld: cannot find crtbegin.o: No such file or directory
INFO -  DEBUG: /builds/worker/fetches/binutils/bin/ld: cannot find -lgcc
INFO -  DEBUG: /builds/worker/fetches/binutils/bin/ld: cannot find -lgcc
INFO -  DEBUG: clang-8: error: linker command failed with exit code 1 (use -v to see invocation)
INFO -  DEBUG: configure: failed program was:
INFO -  DEBUG: #line 4033 "configure"
INFO -  DEBUG: #include "confdefs.h"
INFO -  DEBUG:
INFO -  DEBUG: int main() {
INFO -  DEBUG:
INFO -  DEBUG: ; return 0; }
INFO -  DEBUG: configure: error: compiler is incompatible with sanitize options
INFO -  ERROR: old-configure failed

Will have to add a bunch of debugging output because interactive tasks are broken. :(

Comment 3

[Mike Hommey [:glandium]]

$ find / -name crtbegin.o
find: '/proc/72/map_files': Permission denied
find: '/proc/74/map_files': Permission denied
/usr/lib/gcc/x86_64-linux-gnu/8/32/crtbegin.o
/usr/lib/gcc/x86_64-linux-gnu/8/crtbegin.o
/usr/lib/gcc/x86_64-linux-gnu/8/x32/crtbegin.o

Your docker image is missing the arm64 files.

Comment 4

[Mike Hommey [:glandium]]

For some reason, libgcc-8-dev:arm64 is not installed.

Comment 5

[Nathan Froyd [:froydnj]]

(In reply to Mike Hommey [:glandium] from comment #4)

For some reason, libgcc-8-dev:arm64 is not installed.

I assume it's not appropriate to special-case this in debian-build? What should be installing this package, anyway?

Comment 6

[Mike Hommey [:glandium]]

So, this is how we end up with libgcc-6-dev installed in the debian-9 image:
libgtk-3-dev depends on libpango1.0-dev which depends on libharbuzz-dev which depends on libicu-dev which depends on libstdc++-6-dev which depends on libgcc-6-dev.

In debian 10, the dependency of libicu-dev on libstdc++-n-dev was dropped, so libstdc++-n-dev and libgcc-n-dev don't end up installed.

Inconveniently, n varies by Debian release...

Comment 7

[Nathan Froyd [:froydnj]]

Attachment: Bug 1578535 - explicitly install libstdc++-$VERSION-dev packages; r=glandium

On older Debian versions, libstdc++-$VERSION-dev is implicitly brought
in by other development packages. On newer versions, this dependency
has been removed. Let's go ahead and explicitly declare which version
we want to install for each Debian version.

Comment 8

[Nathan Froyd [:froydnj]]

Attachment: Bug 1578535 - base arm64-related build images off debian10; r=glandium

Our build toolchains don't contain libstdc++ headers for aarch64, so our
aarch64 builds rely on whatever libstdc++ headers the system has
installed. To bring in newer headers on our aarch64 builds, then, we
need to update the system images for those builds, which this patch does.

Depends on D45861

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0621d05839f6
explicitly install libstdc++-$VERSION-dev packages; r=glandium
https://hg.mozilla.org/integration/autoland/rev/5d82bc2f1e9e
base arm64-related build images off debian10; r=glandium

Comment 10

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/0621d05839f6
https://hg.mozilla.org/mozilla-central/rev/5d82bc2f1e9e