Closed Bug 1482167 Opened 6 years ago Closed 5 years ago

Create --enable-fuzzing ASAN build job for Android x86-64 firefox.

Categories

(Firefox Build System :: Task Configuration, task)

x86_64
Android
task
Not set
normal

Tracking

(firefox73 fixed)

RESOLVED FIXED
mozilla73
Tracking Status
firefox73 --- fixed

People

(Reporter: truber, Assigned: truber)

References

(Blocks 2 open bugs)

Details

Attachments

(1 file, 2 obsolete files)

To support fuzzing on Android, we need an ASAN fuzzing build. x86_64 is supported by the NDK and emulator, and makes the most sense as a platform for fuzzing automation.
Nathan, do you know what I'm missing here? [task 2018-08-09T15:01:36.546Z] 15:01:36 INFO - DEBUG: configure:1499: /builds/worker/workspace/build/src/sccache2/sccache /builds/worker/workspace/build/src/clang/bin/clang -fgnu89-inline -std=gnu99 -o conftest -fno-short-enums -fno-exceptions -isystem /builds/worker/workspace/build/src/android-ndk/platforms/android-21/arch-x86_64/usr/include -isystem /builds/worker/workspace/build/src/android-ndk/platforms/android-21/arch-x86_64/usr/include -gcc-toolchain /builds/worker/workspace/build/src/android-ndk/toolchains/x86_64-4.9/prebuilt/linux-x86_64 -D__ANDROID_API__=21 -L/builds/worker/workspace/build/src/android-ndk/platforms/android-21/arch-x86_64/usr/lib -Wl,-rpath-link=/builds/worker/workspace/build/src/android-ndk/platforms/android-21/arch-x86_64/usr/lib --sysroot=/builds/worker/workspace/build/src/android-ndk/platforms/android-21/arch-x86_64 -Wl,--allow-shlib-undefined -gcc-toolchain /builds/worker/workspace/build/src/android-ndk/toolchains/x86_64-4.9/prebuilt/linux-x86_64 conftest.c 1>&5 [task 2018-08-09T15:01:36.546Z] 15:01:36 INFO - DEBUG: /builds/worker/workspace/build/src/android-ndk/toolchains/x86_64-4.9/prebuilt/linux-x86_64/lib/gcc/x86_64-linux-android/4.9.x/../../../../x86_64-linux-android/bin/ld: error: cannot open crt1.o: No such file or directory [task 2018-08-09T15:01:36.547Z] 15:01:36 INFO - DEBUG: /builds/worker/workspace/build/src/android-ndk/toolchains/x86_64-4.9/prebuilt/linux-x86_64/lib/gcc/x86_64-linux-android/4.9.x/../../../../x86_64-linux-android/bin/ld: error: cannot open crti.o: No such file or directory [task 2018-08-09T15:01:36.547Z] 15:01:36 INFO - DEBUG: /builds/worker/workspace/build/src/android-ndk/toolchains/x86_64-4.9/prebuilt/linux-x86_64/lib/gcc/x86_64-linux-android/4.9.x/../../../../x86_64-linux-android/bin/ld: error: cannot open crtn.o: No such file or directory [task 2018-08-09T15:01:36.547Z] 15:01:36 INFO - DEBUG: /builds/worker/workspace/build/src/android-ndk/toolchains/x86_64-4.9/prebuilt/linux-x86_64/lib/gcc/x86_64-linux-android/4.9.x/../../../../x86_64-linux-android/bin/ld: error: cannot find -lgcc_s [task 2018-08-09T15:01:36.547Z] 15:01:36 INFO - DEBUG: /builds/worker/workspace/build/src/android-ndk/toolchains/x86_64-4.9/prebuilt/linux-x86_64/lib/gcc/x86_64-linux-android/4.9.x/../../../../x86_64-linux-android/bin/ld: error: cannot find -lgcc_s [task 2018-08-09T15:01:36.547Z] 15:01:36 INFO - DEBUG: clang-6.0: error: linker command failed with exit code 1 (use -v to see invocation) https://treeherder.mozilla.org/#/jobs?repo=try&revision=48ce4506668b238d57bfe87f9d28b6998245028e
Flags: needinfo?(nfroyd)
It's possible that this job needs to use clang from the NDK, rather than our custom-built clang, because our clang doesn't currently know how to find crt*.o for x86-64 android? I'd try to figure out what the linker thinks its search paths are supposed to be, and see if that provides insight into why it's not choosing whatever directory crt*.o lives in.
Flags: needinfo?(nfroyd)
Use -Wl,--verbose for that.
There's no --target in the clang command, which I guess causes it to look for things that don't make sense for Android (like crt[ni0].o and gcc_s, which aren't in the NDK), even locally when using the NDK clang. I think I have it added properly and updated try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=aed31d96aaed505e5eba256047eaa64096d8bb7c [task 2018-08-13T20:21:25.743Z] 20:21:25 INFO - TARGET = Some("x86_64-linux-android") [task 2018-08-13T20:21:25.743Z] 20:21:25 INFO - HOST = Some("x86_64-unknown-linux-gnu") [task 2018-08-13T20:21:25.744Z] 20:21:25 INFO - CC_x86_64_linux_android = None [task 2018-08-13T20:21:25.744Z] 20:21:25 INFO - TARGET_CC = None [task 2018-08-13T20:21:25.745Z] 20:21:25 INFO - CC = None [task 2018-08-13T20:21:25.745Z] 20:21:25 INFO - CFLAGS_x86_64-linux-android = None [task 2018-08-13T20:21:25.746Z] 20:21:25 INFO - TARGET_CFLAGS = None [task 2018-08-13T20:21:25.746Z] 20:21:25 INFO - CFLAGS = None [task 2018-08-13T20:21:25.746Z] 20:21:25 INFO - DEBUG = Some("false") [task 2018-08-13T20:21:25.747Z] 20:21:25 INFO - running: "x86_64-linux-android-gcc" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-Wall" "-Wextra" "-o" "/builds/worker/workspace/build/src/obj-firefox/toolkit/library/x86_64-linux-android/release/build/lmdb-sys-51e568d8916ceae1/out/mdb.o" "-c" "/builds/worker/workspace/build/src/third_party/rust/lmdb-sys/lmdb/libraries/liblmdb/mdb.c" [task 2018-08-13T20:21:25.747Z] 20:21:25 INFO - thread 'main' panicked at ' [task 2018-08-13T20:21:25.747Z] 20:21:25 INFO - Internal error occurred: Failed to find tool. Is `x86_64-linux-android-gcc` installed? I'm not sure why the build environment isn't used there: CC/CFLAGS aren't inherited at all.
Does your tree include my patches from bug 1409276? I just got done fixing that whole mess recently.
Yes, but this is an ASan build, so it wouldn't get the vars you added for the cc crate, right?
I have a build almost passing (I think) here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=c1a8eb5d254223540c4b3d1307ec3b06687de404 Currently it's broken linking libxul which I don't understand: [task 2018-11-02T15:45:23.660Z] 15:45:23 INFO - /builds/worker/workspace/build/src/sccache2/sccache /builds/worker/workspace/build/src/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/clang++ -std=gnu++14 --target=x86_64-linux-android -Qunused-arguments -isystem /builds/worker/workspace/build/src/android-ndk/sysroot/usr/include/x86_64-linux-android -isystem /builds/worker/workspace/build/src/android-ndk/sysroot/usr/include -gcc-toolchain /builds/worker/workspace/build/src/android-ndk/toolchains/x86_64-4.9/prebuilt/linux-x86_64 -D__ANDROID_API__=21 -U_FORTIFY_SOURCE -fno-common -Qunused-arguments -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wshadow-field-in-constructor-modified -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wfloat-overflow-conversion -Wfloat-zero-conversion -Wloop-analysis -Wc++1z-compat -Wc++2a-compat -Wcomma -Wimplicit-fallthrough -Werror=non-literal-null-conversion -Wstring-conversion -Wtautological-overlap-compare -Wtautological-unsigned-enum-zero-compare -Wtautological-unsigned-zero-compare -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-unknown-warning-option -Wno-return-type-c-linkage -fno-sized-deallocation -fsanitize=address -fno-short-enums -fno-exceptions -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++/include -I/builds/worker/workspace/build/src/android-ndk/sources/android/support/include -I/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++abi/include -U_FORTIFY_SOURCE -fno-common -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pipe -g -O2 -gline-tables-only -fno-omit-frame-pointer -funwind-tables -Werror -fPIC -shared -Wl,--gc-sections -Wl,-h,libxul.so -o libxul.so /builds/worker/workspace/build/src/obj-firefox/toolkit/library/libxul_so.list -fsanitize=address -rdynamic -L/builds/worker/workspace/build/src/android-ndk/platforms/android-21/arch-x86_64/usr/lib -Wl,-rpath-link=/builds/worker/workspace/build/src/android-ndk/platforms/android-21/arch-x86_64/usr/lib --sysroot=/builds/worker/workspace/build/src/android-ndk/platforms/android-21/arch-x86_64 -Wl,--allow-shlib-undefined -gcc-toolchain /builds/worker/workspace/build/src/android-ndk/toolchains/x86_64-4.9/prebuilt/linux-x86_64 -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,nocopyreloc -Wl,-Bsymbolic-functions -Wl,--build-id=sha1 -Wl,--icf=safe -Wl,-Bsymbolic -Wl,--hash-style=sysv -Wl,-rpath-link,/builds/worker/workspace/build/src/obj-firefox/dist/bin -Wl,-rpath-link,/usr/local/lib ../../security/nss/lib/crmf/crmf_crmf/libcrmf.a ../../js/src/build/libjs_static.a /builds/worker/workspace/build/src/obj-firefox/x86_64-linux-android/release/libgkrust.a ../../mozglue/build/libmozglue.so ../../security/libnss3.so ../../config/external/lgpllibs/liblgpllibs.so -landroid -llog -lm -ldl -L/builds/worker/workspace/build/src/android-ndk/sources/cxx-stl/llvm-libc++/libs/x86_64 -lc++_static -lc++abi [task 2018-11-02T15:45:23.660Z] 15:45:23 INFO - /builds/worker/workspace/build/src/android-ndk/toolchains/x86_64-4.9/prebuilt/linux-x86_64/lib/gcc/x86_64-linux-android/4.9.x/../../../../x86_64-linux-android/bin/ld: error: /builds/worker/workspace/build/src/obj-firefox/toolkit/library/../../accessible/android/AccessibleWrap.o: requires dynamic R_X86_64_PC32 reloc against '_ZN7mozilla4a11y19HyperTextAccessible11ReplaceTextERK12nsTSubstringIDsE' which may overflow at runtime; recompile with -fPIC [task 2018-11-02T15:45:23.660Z] 15:45:23 INFO - /builds/worker/workspace/build/src/android-ndk/toolchains/x86_64-4.9/prebuilt/linux-x86_64/lib/gcc/x86_64-linux-android/4.9.x/../../../../x86_64-linux-android/bin/ld: error: /builds/worker/workspace/build/src/obj-firefox/toolkit/library/../../accessible/android/ProxyAccessibleWrap.o: requires dynamic R_X86_64_PC32 reloc against '_ZNK7mozilla4a11y10Accessible15HasNumericValueEv' which may overflow at runtime; recompile with -fPIC [task 2018-11-02T15:45:23.661Z] 15:45:23 INFO - /builds/worker/workspace/build/src/android-ndk/toolchains/x86_64-4.9/prebuilt/linux-x86_64/lib/gcc/x86_64-linux-android/4.9.x/../../../../x86_64-linux-android/bin/ld: error: /builds/worker/workspace/build/src/obj-firefox/toolkit/library/../../accessible/android/SessionAccessibility.o: requires dynamic R_X86_64_PC32 reloc against '_ZNK7mozilla4a11y10Accessible14HasGenericTypeENS0_14AccGenericTypeE' which may overflow at runtime; recompile with -fPIC [task 2018-11-02T15:45:23.661Z] 15:45:23 INFO - /builds/worker/workspace/build/src/android-ndk/toolchains/x86_64-4.9/prebuilt/linux-x86_64/lib/gcc/x86_64-linux-android/4.9.x/../../../../x86_64-linux-android/bin/ld: error: /builds/worker/workspace/build/src/obj-firefox/toolkit/library/../../accessible/android/SessionAccessibility.o: requires dynamic R_X86_64_PC32 reloc against '_ZNK7mozilla4a11y10Accessible14HasGenericTypeENS0_14AccGenericTypeE' which may overflow at runtime; recompile with -fPIC [task 2018-11-02T15:45:23.661Z] 15:45:23 INFO - /builds/worker/workspace/build/src/android-ndk/toolchains/x86_64-4.9/prebuilt/linux-x86_64/lib/gcc/x86_64-linux-android/4.9.x/../../../../x86_64-linux-android/bin/ld: error: read-only segment has dynamic relocations [task 2018-11-02T15:45:23.661Z] 15:45:23 INFO - /builds/worker/workspace/build/src/accessible/android/AccessibleWrap.cpp:187: error: undefined reference to 'mozilla::a11y::HyperTextAccessible::ReplaceText(nsTSubstring<char16_t> const&)' [task 2018-11-02T15:45:23.661Z] 15:45:23 INFO - /builds/worker/workspace/build/src/accessible/android/ProxyAccessibleWrap.cpp:155: error: undefined reference to 'mozilla::a11y::Accessible::HasNumericValue() const' [task 2018-11-02T15:45:23.662Z] 15:45:23 INFO - /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/a11y/Accessible.h:593: error: undefined reference to 'mozilla::a11y::Accessible::HasGenericType(mozilla::a11y::AccGenericType) const' [task 2018-11-02T15:45:23.662Z] 15:45:23 INFO - /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/a11y/Accessible.h:593: error: undefined reference to 'mozilla::a11y::Accessible::HasGenericType(mozilla::a11y::AccGenericType) const' [task 2018-11-02T15:45:23.662Z] 15:45:23 INFO - clang++: error: linker command failed with exit code 1 (use -v to see invocation) [task 2018-11-02T15:45:23.662Z] 15:45:23 INFO - /builds/worker/workspace/build/src/config/rules.mk:710: recipe for target 'libxul.so' failed [task 2018-11-02T15:45:23.662Z] 15:45:23 INFO - make[4]: *** [libxul.so] Error 1 As far as I can see, the compilation of those files (AccessibleWrap.o, etc.) is already -fPIC, and the methods it complains on are inlined, so I don't know what that error message means.
It looks like nothing ever includes HyperTextAccessible-inl.h, where those inline methods are defined, so the linker is complaining there are no definitions. Throwing an #include "HyperTextAccessible-inl.h" in appropriate places will probably help.
(In reply to Nathan Froyd [:froydnj] from comment #9) > Throwing an #include "HyperTextAccessible-inl.h" in appropriate places will probably help. Thanks! That did it. I've got a build working here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=a1041c681c1127f53280a53fc14ae4b42bbb04c3 It requires running android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/asan_device_setup on a rooted device, since I haven't figured out how to package the asan .so yet.
Blocks: 1293677
Depends on: 1526533
No longer depends on: 1526533

Latest "success": https://treeherder.mozilla.org/#/jobs?repo=try&revision=1aef3f74d503fdaa9ffc95144675a12256c64687

This is failing in a packaging step:

20:06:03     INFO -  > Task :geckoview_example:packageWithGeckoBinariesDebug FAILED
20:06:12     INFO -  > Task :geckoview:packageWithGeckoBinariesDebugAndroidTest
20:06:12     INFO -  FAILURE: Build failed with an exception.
20:06:12     INFO -  * What went wrong:
20:06:12     INFO -  Execution failed for task ':geckoview_example:packageWithGeckoBinariesDebug'.
20:06:12     INFO -  > Execution of compression failed.

This is after target.apk is already built, which works after asan_device_setup is run.

Depends on: 1527896
Depends on: 1533560

This could also use https://developer.android.com/ndk/guides/wrap-script which was added in 8.0 (26), which would make rooting unnecessary. Not sure if such an APK would still work with rooting on pre-8.0.

Depends on: 1537751

(In reply to Nathan Froyd [:froydnj] from bug 1537751 comment #12)

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7beac64a0d816c1f93a69cb1ef3314d334bd12da

The asan build error is the C compiler getting invoked from rust not receiving the correct build flags, which seems very solvable. Obviously the C compiler gets the correct build flags in all other Android configurations (?), so it's just a matter of tracing down where those build flags don't get transferred down to Rust correctly. Happy to answer questions about that.

Copying this here.

Flags: needinfo?(jschwartzentruber)

I got past the target.apk here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=24f1ff9a375329c7d911bed8d292e4fe4cd4589c

I fixed the build failure by removing some of the special cases in config/makefiles/rust.mk, but I'll do a full test try once the whole build passes to make sure that didn't break anything elsewhere.

Nick, the heap increase in bug 1527896 doesn't seem to be enough. libxul is now 877Mb, which is approaching the 1.2Gb of desktop ASan libxul. Any idea what I should increase it to, or if there is another way? I don't know why geckoview_example:packageWithGeckoBinariesDebug is failing where fennec is not.

Flags: needinfo?(jschwartzentruber) → needinfo?(nalexander)

(In reply to Jesse Schwartzentruber (:truber) from comment #15)

I got past the target.apk here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=24f1ff9a375329c7d911bed8d292e4fe4cd4589c

I fixed the build failure by removing some of the special cases in config/makefiles/rust.mk, but I'll do a full test try once the whole build passes to make sure that didn't break anything elsewhere.

Nick, the heap increase in bug 1527896 doesn't seem to be enough. libxul is now 877Mb, which is approaching the 1.2Gb of desktop ASan libxul. Any idea what I should increase it to, or if there is another way? I don't know why geckoview_example:packageWithGeckoBinariesDebug is failing where fennec is not.

Try applying

diff --git a/mobile/android/geckoview_example/build.gradle b/mobile/android/geckoview_example/build.gradle
--- a/mobile/android/geckoview_example/build.gradle
+++ b/mobile/android/geckoview_example/build.gradle
@@ -22,6 +22,10 @@ android {
         targetCompatibility JavaVersion.VERSION_1_8
     }
 
+    dexOptions {
+        javaMaxHeapSize "4g"
+    }
+
     buildTypes {
         release {
             minifyEnabled false

to see if the same fix will work in this situation. After that, we bump everything to 6g to see if that helps. If not, the next thing to see would be to dig into Gradle itself and figure out if anything has changed in newer Gradle versions (I think 5.1 or 5.2 is latest, and we build against 4.10.2 IIRC).

Flags: needinfo?(nalexander) → needinfo?(jschwartzentruber)

(In reply to Nick Alexander :nalexander [he/him] from comment #16)

After that, we bump everything to 6g to see if that helps. If not, the next thing to see would be to dig into Gradle itself and figure out if anything has changed in newer Gradle versions (I think 5.1 or 5.2 is latest, and we build against 4.10.2 IIRC).

Your patch got me further, but I did have to bump everything to 6g to make it all pass. I'll open bugs to get all the changes landed.
Try is here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=88c109a25fdf9353a855c33845b729ea8c1fa050

Flags: needinfo?(jschwartzentruber)

(In reply to Jesse Schwartzentruber (:truber) from comment #17)

(In reply to Nick Alexander :nalexander [he/him] from comment #16)

After that, we bump everything to 6g to see if that helps. If not, the next thing to see would be to dig into Gradle itself and figure out if anything has changed in newer Gradle versions (I think 5.1 or 5.2 is latest, and we build against 4.10.2 IIRC).

Your patch got me further, but I did have to bump everything to 6g to make it all pass. I'll open bugs to get all the changes landed.
Try is here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=88c109a25fdf9353a855c33845b729ea8c1fa050

Meh, no need for additional tickets -- you can fold the bumps into this one. I'm not concerned about bumping to "6g" globally; most consumers won't actually consume that, and there's little difference between OOM due to JVM limit and OOM due to OS limit.

There's a try here, but failing on windows asan & ccov.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=1f3046f784bd56fec41f5fe97193b2ace2577935

I'm not sure if the ccov failure is my fault, but the asan failures are a missing dll (asan?).

https://treeherder.mozilla.org/#/jobs?repo=try&revision=bacd51b400fc73566d0501574756f9dfdd5f0c78

This enables ASan for C code used in Rust crates on all platforms except Windows. This was needed because otherwise Rust CC wasn't getting the Android NDK headers. I could strip out the sanitizer CFLAGS/LDFLAGS, but it seems like a good idea to have this code instrumented by ASan.
Windows could not find the ASan DLLs when linking Rust with C, and I don't know why.

I'm surprised this builds at all. There's a reason we don't pass compiler flags down when building with asan, and I don't see why you're not hitting it.

DO NOT LAND because the conditional isn't correct: this is convenient
for local testing and not the final form.

The given script could hard-code the NDK ABI.

See https://developer.android.com/ndk/guides/wrap-script.

truber: this ended up being better with a different expression than I anticipated. Add my patches onto (or under) your stack, and then, in your try job's mozconfig, set MOZ_CLANG_RT_ASAN_LIB_PATH, following https://searchfox.org/mozilla-central/source/build/win64/mozconfig.asan#11 or a similar approach. You should get a wrap.sh and the appropriate library in both the Fennec APK and the GeckoView AAR.

With the "wrong" conditional that I have in a few places, I get that with an artifact build and

export MOZ_CLANG_RT_ASAN_LIB_PATH=/Users/nalexander/.mozbuild/linux64-clang-8-android-cross/lib64/clang/8.0.0/lib/linux/libclang_rt.asan-i686-android.so

in my mozconfig.

I can't speak to the other compilation issues you and glandium are discussing, but that should at least get the packaging bits moving.

Flags: needinfo?(jschwartzentruber)

I've updated the patch to only disable the rust.mk bits for non-cross-compile + asan builds, as discussed with glandium. Nick, your patches work perfectly as far as I can see, except for the problem that the resulting .apk crashes on launch. I'm going to open another bug for that, since this is enough to start fuzzing using asan_device_setup.

Flags: needinfo?(jschwartzentruber)
Blocks: 1576213
Depends on: 1577814
Depends on: 1580234
Depends on: 1596950
Depends on: 1596953
Pushed by rmaries@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/bee049f7e1c0 Create --enable-fuzzing ASAN build job for Android x86-64 firefox. r=froydnj
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla73
Attachment #9073426 - Attachment is obsolete: true
Attachment #9073427 - Attachment is obsolete: true
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: