1674180 - syncLibsAndUpdateGenerationIDFromDistDir fails for asan build

Status: RESOLVED FIXED

(Firefox Build System :: Android Studio and Gradle Integration, defect, P5)

Description

[[ex-Mozilla] Agi Sferro | :agi]

11:32.39 Error reading file: invalid section reference.
11:32.49 FAILURE: Build failed with an exception.
11:32.49 * Where:
11:32.49 Script '/home/agi/workspace/mozilla-central/mobile/android/gradle/with_gecko_binaries.gradle' line: 100
11:32.49 * What went wrong:
11:32.49 Execution failed for task ':geckoview:syncLibsAndUpdateGenerationIDFromDistDirForWithGeckoBinariesDebug'.
11:32.49 > Process 'command '/home/agi/.mozbuild/android-ndk-r20/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-readobj'' finished with non-zero exit value 1
11:32.49 * Try:
11:32.49 Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
11:32.49 * Get more help at https://help.gradle.org
11:32.49 Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
11:32.49 Use '--warning-mode all' to show the individual deprecation warnings.
11:32.49 See https://docs.gradle.org/5.1.1/userguide/command_line_interface.html#sec:command_line_warnings
11:32.49 BUILD FAILED in 11s
11:32.49 77 actionable tasks: 62 executed, 15 up-to-date
11:32.93 Makefile:150: recipe for target 'recurse_android-archive-geckoview' failed
11:32.93 make[3]: *** [recurse_android-archive-geckoview] Error 1
11:32.93 /home/agi/workspace/mozilla-central/config/recurse.mk:32: recipe for target 'android-archive-geckoview' failed
11:32.93 make[2]: *** [android-archive-geckoview] Error 2
11:32.94 /home/agi/workspace/mozilla-central/config/rules.mk:355: recipe for target 'default' failed
11:32.94 make[1]: *** [default] Error 2
11:32.94 client.mk:89: recipe for target 'build' failed
11:32.94 make: *** [build] Error 2
11:32.94 258 compiler warnings present.
11:33.03 /usr/bin/notify-send --app-name=Mozilla Build System Mozilla Build System Build failed

my mozconfig (on Linux):

ac_add_options --enable-application=mobile/android
ac_add_options --target=x86_64

# Artifacts
# ac_add_options --enable-artifact-builds
# mk_add_options MOZ_OBJDIR=./objdir-artifact

# ac_add_options --enable-warnings-as-errors

# Debug
# ac_add_options --enable-debug
# ac_add_options --disable-optimize

# ASAN
export LLVM_SYMBOLIZER="/home/agi/.mozbuild/clang/bin/llvm-symbolizer"
export MOZ_CLANG_RT_ASAN_LIB_PATH="/home/agi/.mozbuild/clang/lib64/clang/9.0.1/lib/linux/libclang_rt.asan-x86_64-android.so"
ac_add_options --enable-address-sanitizer

# Mandatory options required for ASan builds (both on Linux and Mac)
export MOZ_DEBUG_SYMBOLS=1
ac_add_options --enable-debug-symbols
ac_add_options --disable-install-strip
ac_add_options --disable-jemalloc
ac_add_options --disable-crashreporter
ac_add_options --disable-profiling

# Fuzzing
ac_add_options --enable-fuzzing
ac_add_options --disable-debug
ac_add_options --enable-optimize="-O2 -gline-tables-only"

mk_add_options MOZ_OBJDIR=./objdir-fuzzing

note this might be because I'm using Clang 9.

Comment 1

[Nick Alexander :nalexander [he/him]]

The immediate workaround is to ape https://bugzilla.mozilla.org/show_bug.cgi?id=1667948 and don't do the generation stuff if we're building ASAN.

If you could hardcode the path to the llvm-readobj corresponding to your actual toolchain, I'd be interested if that worked.

P5/S4: we'll take a patch but this isn't particularly relevant to anybody 'cuz very few folks build ASAN on Android locally.

Comment 2

[[ex-Mozilla] Agi Sferro | :agi]

I tripped on this enough time to fix this. The issue here is that we're using the wrong toolchain, Gecko is built using (normally) ~/.mozbuild/clang/bin/clang, so we should use the llvm-* tools from that toolchain, instead of using the NDK which is a different toolchain (much older as the time of writing). I'll post a patch.

Comment 3

[Nick Alexander :nalexander [he/him]]

(In reply to Agi Sferro | :agi | ni? for questions | ⏰ PST | he/him from comment #2)

I tripped on this enough time to fix this. The issue here is that we're using the wrong toolchain, Gecko is built using (normally) ~/.mozbuild/clang/bin/clang, so we should use the llvm-* tools from that toolchain, instead of using the NDK which is a different toolchain (much older as the time of writing). I'll post a patch.

Patches welcome. I think at the time that I wrote this code we didn't have llvm-readobj in the toolchain clang. Or maybe I wasn't confident it would work across architectures?

Comment 4

[[ex-Mozilla] Agi Sferro | :agi]

(In reply to Nick Alexander :nalexander [he/him] from comment #3)

(In reply to Agi Sferro | :agi | ni? for questions | ⏰ PST | he/him from comment #2)

I tripped on this enough time to fix this. The issue here is that we're using the wrong toolchain, Gecko is built using (normally) ~/.mozbuild/clang/bin/clang, so we should use the llvm-* tools from that toolchain, instead of using the NDK which is a different toolchain (much older as the time of writing). I'll post a patch.

Patches welcome. I think at the time that I wrote this code we didn't have llvm-readobj in the toolchain clang. Or maybe I wasn't confident it would work across architectures?

Ah man you're right. llvm-readobj is not in the normal toolchain, that's annoying. I wonder what controls that, I have llvm-readobj from the linux64-clang-11-android-cross toolchain: https://searchfox.org/mozilla-central/source/taskcluster/ci/toolchain/clang.yml#144

Comment 5

[[ex-Mozilla] Agi Sferro | :agi]

Oh I think I downloaded the full toolchain (clang.tar.zst) while the one coming from mach bootstrap is stripped down somehow, somewhere

Comment 6

[[ex-Mozilla] Agi Sferro | :agi]

maybe here: https://searchfox.org/mozilla-central/source/taskcluster/scripts/misc/build-dist-toolchains.sh#12

Comment 7

[Nick Alexander :nalexander [he/him]]

(In reply to Agi Sferro | :agi | ni? for questions | ⏰ PST | he/him from comment #5)

Oh I think I downloaded the full toolchain (clang.tar.zst) while the one coming from mach bootstrap is stripped down somehow, somewhere

If my memory is correct, this is (part of) the difference between a -cross toolchain and a regular toolchain. The regular ones can target different architectures but don't have full support for everything we use in automation. I don't know more of the details, nor do I know the size difference. If there's not much in it, perhaps we could have bootstrap for mobile/android install the -cross toolchain?

glandium: do you have information and/or an opinion here?

Comment 8

[[ex-Mozilla] Agi Sferro | :agi]

Actually nevermind, I just blew up my .mozbuild folder and ran a mach bootstrap and I do have clang/bin/llvm-readobj on Linux. Maybe this is a problem on Mac? Let's see.

Comment 9

[Mike Hommey [:glandium]]

What are you doing with llvm-readobj that you can't do with llvm-objdump, for which configure will give you the right path to with the variable LLVM_OBJDUMP?

Comment 10

[Mike Hommey [:glandium]]

And to answer your question, the -cross toolchains are only really needed when you want to do sanitizer builds.

Comment 11

[Nick Alexander :nalexander [he/him]]

(In reply to Mike Hommey [:glandium] from comment #9)

What are you doing with llvm-readobj that you can't do with llvm-objdump, for which configure will give you the right path to with the variable LLVM_OBJDUMP?

Currently, we:

1. read the binary build ID with --hex-dump=.note.gnu.build-id
2. read another section with --hex-dump
3. write that other section with --add-section

It's the last one that I don't think can be done with llvm-objdump. I would be thrilled to be proven wrong, or to have an alternate approach suggested.

Comment 12

[Mike Hommey [:glandium]]

$OBJCOPY does support adding sections.

But what is this hackery for?

Comment 13

[[ex-Mozilla] Agi Sferro | :agi]

But what is this hackery for?

Gradle has a bug where if a native library is not modified between builds it doesn't include it in the artifact :/ so we modify the so files enough that gradle thinks they changed: https://searchfox.org/mozilla-central/rev/f982032c7c7618c626165bb557968f478a1952dc/mobile/android/gradle/with_gecko_binaries.gradle#43-56

Comment 14

[[ex-Mozilla] Agi Sferro | :agi]

Attachment: Bug 1674180 - Use the llvm-readobj from the build instead of the NDK.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/66e707832d80
Use the llvm-readobj from the build instead of the NDK. r=nalexander

Comment 16

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/66e707832d80