Closed Bug 1540082 Opened 6 months ago Closed 5 months ago

Build and package sanitizer and coverage runtimes for AArch64 Linux in our Clang build

Categories

(Firefox Build System :: Toolchains, enhancement)

ARM64
Linux
enhancement
Not set

Tracking

(firefox68 fixed)

RESOLVED FIXED
mozilla68
Tracking Status
firefox68 --- fixed

People

(Reporter: decoder, Assigned: froydnj)

References

(Blocks 1 open bug)

Details

(Keywords: sec-want)

Attachments

(3 files)

Since bug 1532952 landed, we now have AArch64 Linux builds (thanks :glandium!).

However, in order to make ASan+Fuzzing builds happen in that configuration, we still need the sanitizer runtimes (and if possible the coverage runtime) to be built with our Clang. Currently it only includes i386 and x86_64 as well as various runtimes for Android and Mac.

Blocks: 1540084

Do our x86-64 Linux packages automatically come with the necessary sanitizer runtimes for x86-64 Linux? Or do those bits only get built specially for our linux64-clang-8-macosx-cross builds?

Or maybe a better way of asking this is: which clang build does this need to be added to?

Flags: needinfo?(choller)

I believe our Clang Linux64 build comes with all the necessary sanitizer and profiling runtimes (both for x86_64 and i386 btw), also for Mac. So this needs to be added to our regular Clang Linux64 build, probably by modifying build/build-clang/build-clang.py to build and package the runtimes for aarch64. :chmanchester suggested that this might work similarly to how we build the runtimes for Android in that script.

Flags: needinfo?(choller)

OK, I think I almost understand how to build the arm64 runtime libraries as a part of build-clang.py. The one wrinkle is that we need an arm64 sysroot of some kind. We obviously have one in our debian9-arm64-build packages, but I think we'd prefer to use debian7 as our image (that's what we use for our current toolchain builds) so as to ensure the packages work on the broadest array of Linux machines.

Setting up a debian7-arm64-build image, however, ran into repeated errors:

Fetched 33.3 kB in 10s (3141 B/s)
W: Size of file /var/lib/apt/lists/partial/queue.taskcluster.net_v1_task_DTAKoiiUQOymNdgsPvUQfQ_artifacts_public_build_debian_Packages is not what the server reported 3395 228
W: Size of file /var/lib/apt/lists/partial/queue.taskcluster.net_v1_task_Eo-Dx64DRo-721x1JLguOQ_artifacts_public_build_debian_Packages is not what the server reported 2411 228
W: Size of file /var/lib/apt/lists/partial/queue.taskcluster.net_v1_task_KLS%5f1-xlSx6owxH1bmDgyQ_artifacts_public_build_debian_Packages is not what the server reported 2511 228
W: Size of file /var/lib/apt/lists/partial/queue.taskcluster.net_v1_task_RIAQT7lhRsuFHGMLVW7p9w_artifacts_public_build_debian_Packages is not what the server reported 4370 228
W: Size of file /var/lib/apt/lists/partial/queue.taskcluster.net_v1_task_Ssx2Wq90TWKQrbpzykTj3w_artifacts_public_build_debian_Packages is not what the server reported 4893 228
W: Size of file /var/lib/apt/lists/partial/queue.taskcluster.net_v1_task_W9QbiAr6RuiBPg1NWvK8Ww_artifacts_public_build_debian_Packages is not what the server reported 1193 228
W: Size of file /var/lib/apt/lists/partial/queue.taskcluster.net_v1_task_Xr-NuKlHSrmg-dD%5frRdW4Q_artifacts_public_build_debian_Packages is not what the server reported 1081 228
W: Size of file /var/lib/apt/lists/partial/queue.taskcluster.net_v1_task_ZakI-b9UQ%5fG9CzwAZF5C2A_artifacts_public_build_debian_Packages is not what the server reported 4382 228
W: Size of file /var/lib/apt/lists/partial/queue.taskcluster.net_v1_task_bOfUFTLFRXmi21JE%5fBy9aQ_artifacts_public_build_debian_Packages is not what the server reported 2165 228
W: Size of file /var/lib/apt/lists/partial/queue.taskcluster.net_v1_task_c6ol15-jS2yPT6Euw9xd3A_artifacts_public_build_debian_Packages is not what the server reported 6201 228
W: Size of file /var/lib/apt/lists/partial/queue.taskcluster.net_v1_task_fFCY3H4OSfevZHhkZfwr6A_artifacts_public_build_debian_Packages is not what the server reported 747 228
W: Failed to fetch http://snapshot.debian.org/archive/debian/20171210T214726Z/dists/wheezy/Release  Unable to find expected entry 'main/binary-arm64/Packages' in Release file (Wrong sources.list entry or malformed file)

W: Failed to fetch http://snapshot.debian.org/archive/debian/20171210T214726Z/dists/wheezy-updates/Release  Unable to find expected entry 'main/binary-arm64/Packages' in Release file (Wrong sources.list entry or malformed file)

W: Failed to fetch http://snapshot.debian.org/archive/debian/20171210T214726Z/dists/wheezy-backports/Release  Unable to find expected entry 'main/binary-arm64/Packages' in Release file (Wrong sources.list entry or malformed file)

W: Failed to fetch http://snapshot.debian.org/archive/debian-security/20171210T214726Z/dists/wheezy/updates/Release  Unable to find expected entry 'main/binary-arm64/Packages' in Release file (Wrong sources.list entry or malformed file)

E: Some index files failed to download. They have been ignored, or old ones used instead.

and so we couldn't even build the necessary images to start seeing whether the sysroot would work properly. (An earlier build that didn't have any arm64 development files installed at least started building the arm64 runtime libraries before running into header file mismatches.) It looks like debian 8 was the first version to support arm64, boo.

At this point we have a couple of options:

  1. Build arm64 libraries as part of the normal linux64-clang build, but build on debian 8. This option changes the system requirements (glibc etc.) for the toolchains that we distribute via mach bootstrap.
  2. Build arm64 libraries as part of the normal linux64-clang build, but build on debian 9. Same issues as option 1.
  3. Have an entirely separate toolchain that includes arm64 libraries for arm64 cross builds. We can build this on debian 9, since we don't care about external clients using this toolchain. (Unless the fuzzing team is intending to download these toolchains...?)
  4. Build an entirely separate toolchain that includes arm64 libraries (built on debian 9), but repackage just the arm64 libraries as part of our normal linux64-clang build. This option is similar to what we do for our mac toolchain builds (see https://searchfox.org/mozilla-central/source/taskcluster/scripts/misc/build-clang-8-linux-macosx-cross.sh )

I don't think we want to do option 1 or 2. I'm guessing that option 3 is out because the fuzzing team will actually want to use these toolchains...? Or is the only thing that matters the builds themselves, and we don't care about the toolchains?

I don't really like option 4, but if the fuzzing team needs the toolchains in addition to the actual Firefox builds, that may be the direction we have to go. decoder, am I correct in my assumptions? glandium, do you have opinions, or other options that I missed?

Flags: needinfo?(mh+mozilla)
Flags: needinfo?(choller)

(In reply to Nathan Froyd [:froydnj] from comment #3)

I don't really like option 4, but if the fuzzing team needs the toolchains in addition to the actual Firefox builds, that may be the direction we have to go. decoder, am I correct in my assumptions?

I don't think we necessarily need the toolchains. What we need is the builds, we fetch them directly from TC. If we wanted to make our own custom builds for some reason (e.g. local testing), then we also have native build environments available for that. Thanks for helping out on this!

Flags: needinfo?(choller)

Option 5: build compiler-rt separately. We should actually do that for macos-cross and android-cross too.

Flags: needinfo?(mh+mozilla)

(In reply to Mike Hommey [:glandium] from comment #5)

Option 5: build compiler-rt separately. We should actually do that for macos-cross and android-cross too.

Like, build compiler-rt for each (or all) separate architectures, and then combine them into the appropriate toolchains?

Flags: needinfo?(mh+mozilla)

Yes.

Flags: needinfo?(mh+mozilla)

(or not combine them at all, but I suspect it wouldn't work great)

Depends on: 1544568

I'm currently trying option 3, since that seems the most straightforward to integrate and doesn't require writing a bunch of code for building compiler-rt separately. Unfortunately, we don't have a docker image set up with arm64 sysroots and prepared to build toolchains, so I'm trying to fight with docker to make one.

A successful aarch64 runtime build with a bunch of other patch nonsense mixed in:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=6ae6a773dcb9ec3b92b928e3c244dcac3a2742d8

I'll try to get that cleaned up and patches posted.

We need this image for building clang on machines with arm64
sysroots. (Note that this image is a linux x86-64 image, just with
some arm64 cross-compilation packages available.)

This change enables us to build compiler-rt and related
libraries (e.g. address sanitizer, etc.) for whatever targets we like,
assuming that we have an accessible sysroot for the target on the build
machine.

Depends on D28404

Analogously to the existing linux64-clang-8-android-cross build, this
build is a linux x86-64 build with runtime library support for aarch64.

Depends on D28405

Pushed by nfroyd@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/27527aa56004
add a `toolchain-arm64-build` docker image; r=nalexander
https://hg.mozilla.org/integration/autoland/rev/9f95b1b2f9e1
add support for extra runtime targets in build-clang.py; r=firefox-build-system-reviewers,chmanchester
https://hg.mozilla.org/integration/autoland/rev/6cbad9f5b259
add an aarch64-cross clang build; r=nalexander
Status: NEW → RESOLVED
Closed: 5 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla68
Assignee: nobody → nfroyd
Regressions: 1547289
You need to log in before you can comment on or make changes to this bug.