Create --enable-fuzzing ASAN build job for Android x86-64 firefox.
Categories
(Firefox Build System :: Task Configuration, task)
Tracking
(firefox73 fixed)
Tracking | Status | |
---|---|---|
firefox73 | --- | fixed |
People
(Reporter: truber, Assigned: truber)
References
(Blocks 2 open bugs)
Details
Attachments
(1 file, 2 obsolete files)
Assignee | ||
Comment 1•6 years ago
|
||
Assignee | ||
Comment 2•6 years ago
|
||
Comment 3•6 years ago
|
||
Comment 4•6 years ago
|
||
Assignee | ||
Comment 5•6 years ago
|
||
Comment 6•6 years ago
|
||
Assignee | ||
Comment 7•6 years ago
|
||
Assignee | ||
Comment 8•6 years ago
|
||
Comment 9•6 years ago
|
||
Assignee | ||
Comment 10•6 years ago
|
||
Assignee | ||
Comment 11•6 years ago
|
||
Latest "success": https://treeherder.mozilla.org/#/jobs?repo=try&revision=1aef3f74d503fdaa9ffc95144675a12256c64687
This is failing in a packaging step:
20:06:03 INFO - > Task :geckoview_example:packageWithGeckoBinariesDebug FAILED
20:06:12 INFO - > Task :geckoview:packageWithGeckoBinariesDebugAndroidTest
20:06:12 INFO - FAILURE: Build failed with an exception.
20:06:12 INFO - * What went wrong:
20:06:12 INFO - Execution failed for task ':geckoview_example:packageWithGeckoBinariesDebug'.
20:06:12 INFO - > Execution of compression failed.
This is after target.apk is already built, which works after asan_device_setup
is run.
Assignee | ||
Comment 12•6 years ago
|
||
Assignee | ||
Comment 13•6 years ago
|
||
This could also use https://developer.android.com/ndk/guides/wrap-script which was added in 8.0 (26), which would make rooting unnecessary. Not sure if such an APK would still work with rooting on pre-8.0.
Assignee | ||
Comment 14•6 years ago
|
||
(In reply to Nathan Froyd [:froydnj] from bug 1537751 comment #12)
https://treeherder.mozilla.org/#/jobs?repo=try&revision=7beac64a0d816c1f93a69cb1ef3314d334bd12da
The asan build error is the C compiler getting invoked from rust not receiving the correct build flags, which seems very solvable. Obviously the C compiler gets the correct build flags in all other Android configurations (?), so it's just a matter of tracing down where those build flags don't get transferred down to Rust correctly. Happy to answer questions about that.
Copying this here.
Assignee | ||
Comment 15•5 years ago
|
||
I got past the target.apk here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=24f1ff9a375329c7d911bed8d292e4fe4cd4589c
I fixed the build failure by removing some of the special cases in config/makefiles/rust.mk
, but I'll do a full test try once the whole build passes to make sure that didn't break anything elsewhere.
Nick, the heap increase in bug 1527896 doesn't seem to be enough. libxul is now 877Mb, which is approaching the 1.2Gb of desktop ASan libxul. Any idea what I should increase it to, or if there is another way? I don't know why geckoview_example:packageWithGeckoBinariesDebug
is failing where fennec is not.
Comment 16•5 years ago
|
||
(In reply to Jesse Schwartzentruber (:truber) from comment #15)
I got past the target.apk here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=24f1ff9a375329c7d911bed8d292e4fe4cd4589c
I fixed the build failure by removing some of the special cases in
config/makefiles/rust.mk
, but I'll do a full test try once the whole build passes to make sure that didn't break anything elsewhere.Nick, the heap increase in bug 1527896 doesn't seem to be enough. libxul is now 877Mb, which is approaching the 1.2Gb of desktop ASan libxul. Any idea what I should increase it to, or if there is another way? I don't know why
geckoview_example:packageWithGeckoBinariesDebug
is failing where fennec is not.
Try applying
diff --git a/mobile/android/geckoview_example/build.gradle b/mobile/android/geckoview_example/build.gradle
--- a/mobile/android/geckoview_example/build.gradle
+++ b/mobile/android/geckoview_example/build.gradle
@@ -22,6 +22,10 @@ android {
targetCompatibility JavaVersion.VERSION_1_8
}
+ dexOptions {
+ javaMaxHeapSize "4g"
+ }
+
buildTypes {
release {
minifyEnabled false
to see if the same fix will work in this situation. After that, we bump everything to 6g to see if that helps. If not, the next thing to see would be to dig into Gradle itself and figure out if anything has changed in newer Gradle versions (I think 5.1 or 5.2 is latest, and we build against 4.10.2 IIRC).
Assignee | ||
Comment 17•5 years ago
|
||
(In reply to Nick Alexander :nalexander [he/him] from comment #16)
After that, we bump everything to 6g to see if that helps. If not, the next thing to see would be to dig into Gradle itself and figure out if anything has changed in newer Gradle versions (I think 5.1 or 5.2 is latest, and we build against 4.10.2 IIRC).
Your patch got me further, but I did have to bump everything to 6g to make it all pass. I'll open bugs to get all the changes landed.
Try is here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=88c109a25fdf9353a855c33845b729ea8c1fa050
Comment 18•5 years ago
|
||
(In reply to Jesse Schwartzentruber (:truber) from comment #17)
(In reply to Nick Alexander :nalexander [he/him] from comment #16)
After that, we bump everything to 6g to see if that helps. If not, the next thing to see would be to dig into Gradle itself and figure out if anything has changed in newer Gradle versions (I think 5.1 or 5.2 is latest, and we build against 4.10.2 IIRC).
Your patch got me further, but I did have to bump everything to 6g to make it all pass. I'll open bugs to get all the changes landed.
Try is here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=88c109a25fdf9353a855c33845b729ea8c1fa050
Meh, no need for additional tickets -- you can fold the bumps into this one. I'm not concerned about bumping to "6g" globally; most consumers won't actually consume that, and there's little difference between OOM due to JVM limit and OOM due to OS limit.
Assignee | ||
Comment 19•5 years ago
|
||
There's a try here, but failing on windows asan & ccov.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=1f3046f784bd56fec41f5fe97193b2ace2577935
I'm not sure if the ccov failure is my fault, but the asan failures are a missing dll (asan?).
Assignee | ||
Comment 20•5 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=bacd51b400fc73566d0501574756f9dfdd5f0c78
This enables ASan for C code used in Rust crates on all platforms except Windows. This was needed because otherwise Rust CC wasn't getting the Android NDK headers. I could strip out the sanitizer CFLAGS/LDFLAGS, but it seems like a good idea to have this code instrumented by ASan.
Windows could not find the ASan DLLs when linking Rust with C, and I don't know why.
Comment 21•5 years ago
|
||
I'm surprised this builds at all. There's a reason we don't pass compiler flags down when building with asan, and I don't see why you're not hitting it.
Comment 22•5 years ago
|
||
DO NOT LAND because the conditional isn't correct: this is convenient
for local testing and not the final form.
The given script could hard-code the NDK ABI.
Comment 23•5 years ago
|
||
Depends on D35618
Comment 24•5 years ago
|
||
truber: this ended up being better with a different expression than I anticipated. Add my patches onto (or under) your stack, and then, in your try job's mozconfig
, set MOZ_CLANG_RT_ASAN_LIB_PATH
, following https://searchfox.org/mozilla-central/source/build/win64/mozconfig.asan#11 or a similar approach. You should get a wrap.sh
and the appropriate library in both the Fennec APK and the GeckoView AAR.
With the "wrong" conditional that I have in a few places, I get that with an artifact build and
export MOZ_CLANG_RT_ASAN_LIB_PATH=/Users/nalexander/.mozbuild/linux64-clang-8-android-cross/lib64/clang/8.0.0/lib/linux/libclang_rt.asan-i686-android.so
in my mozconfig
.
I can't speak to the other compilation issues you and glandium are discussing, but that should at least get the packaging bits moving.
Assignee | ||
Comment 25•5 years ago
|
||
I've updated the patch to only disable the rust.mk
bits for non-cross-compile + asan builds, as discussed with glandium. Nick, your patches work perfectly as far as I can see, except for the problem that the resulting .apk crashes on launch. I'm going to open another bug for that, since this is enough to start fuzzing using asan_device_setup
.
Assignee | ||
Comment 26•5 years ago
|
||
Here's an updated try matching the latest patch. https://treeherder.mozilla.org/#/jobs?repo=try&revision=422fd8513f6bab46debc3cc137f905c320ac622a
Comment 27•5 years ago
|
||
Comment 28•5 years ago
|
||
bugherder |
Updated•4 years ago
|
Updated•4 years ago
|
Description
•