1659191 - Android sanitizer builds should include llvm-symbolizer

Status: ASSIGNED

(Firefox Build System :: Toolchains, enhancement, P3)

Description

[Jesse Schwartzentruber (:truber)]

On all other platforms, we package llvm-symbolizer to ensure symbolization of sanitizer backtraces works properly, so that things like suppression lists work on-device.

Comment 1

[Nathan Froyd [:froydnj]]

That would require llvm-symbolizer to be compiled as an Android binary, yes? Does LLVM even support compiling its standalone programs that way?

Comment 2

[Jesse Schwartzentruber (:truber)]

Yes, I have a wip patch here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=b9b7acf51ad2ec99c9022c26a0e05afdf19a08d5

From previous try's, the packaging doesn't seem to work and llvm-symbolizer still isn't included. Once that works I still need to check if lib/<arch> is in PATH or if I need to explicitly add external_symbolizer_path to ASAN_OPTIONS in wrap.sh.

Comment 3

[Ricky Stewart]

I've been trying to make this work, but TaskCluster doesn't want to run the build-android-x86_64-asan-fuzzing/opt job on try even if I request it. :(

Comment 4

[Ricky Stewart]

@aki --- do you have any thoughts about why I can't perform build build-android-x86_64-asan-fuzzing/opt on try? Still not working for me today. (I'm not necessarily ruling out user error but I haven't been able to find anything...)

Comment 5

[Aki Sasaki (not active)]

Here's what I did:

• First I added a new task, thinking that maybe you didn't actually schedule the task in the first place. (Then I noticed I was wrong and tried to cancel it, but the cancel failed because the add task already finished.)
• Click on the decision task. In the bottom left, click on the U-PvQ-8wQRe7e1CxUuGuRg to go to the task at https://firefox-ci-tc.services.mozilla.com/tasks/U-PvQ-8wQRe7e1CxUuGuRg
• Click on Task group at the top left. Notice there are 2 build-android-x86_64-asan-fuzzing/opt tasks scheduled: one for your try push, and one for my add task action. Both are unscheduled. (Unscheduled tasks don't appear on treeherder.) Click on one, e.g. https://firefox-ci-tc.services.mozilla.com/tasks/du-Ke9S0S6KRzVu-3ongYw to see what's going on.
• Click on See more at the bottom left to figure out what's going on.
• Notice all the dependencies are green, except one red, toolchain-android-x86_64-llvm-symbolizer. Click on that and go to https://firefox-ci-tc.services.mozilla.com/tasks/ZDAfzEQjSECxO17cjnEndg
• (In the bottom left of the symbolizer task, notice all the exception dependent tasks: these are all the tasks that failed because this one is broken.)
• In the top right of the symbolizer task, click on the live log.
• Scroll down. See PermissionError: [Errno 13] Permission denied: 'workspace/build/src/taskcluster/scripts/misc/build-android-llvm-symbolizer.sh'

Your task isn't running on Try because the llvm symbolizer task on Try is busted from a permissions issue.

Comment 6

[Ricky Stewart]

Hmph. Thank you. There's apparently a dark underbelly to Taskcluster that I'm not aware of because it's not surfaced in the main UI.

Now to figure out why this script (which is definitely executable) isn't executable on the worker...

Comment 7

[Ricky Stewart]

Waiting for Connor's feedback on an apparent VCS issue here.

Comment 8

[Mitchell Hentges [:mhentges] 🦀]

Update on this:

• The VCS issue is that a script is marked as executable in hg, but it _isn't _ executable on the worker. Weird.
• After following up with Connor, I'm not sure that it's specifically an hg issue.
• Though this is a valuable ticket, it's not at the top of the priority stack, so I'm going to backlog it for now.

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Comment 10

[Jesse Schwartzentruber (:truber)]

Attachment: WIP: Bug 1659191 - Package llvm-symbolizer with Android asan build.

Comment 11

[Jesse Schwartzentruber (:truber)]

Updated this to use build-llvm-common. Pointing to this executable using ASAN_OPTIONS=external_symbolizer_path=... works with geckoview_example in an x86_64 emulator.

llvm-symbolizer is packaged in the jsshell zip, but not in the apk. I believe this is getting filtered by apk builder which only allows lib*.so and gdbserver.

Since wrap.sh gets included in build.gradle, I'm wondering if the same approach could be used for llvm-symbolizer. Copy llvm-symbolizer into $topobjdir/extras/lib/x86_64/$ANDROID_CPU_ARCH/ and then srcDir $topobjdir/extras in build.gradle. I don't know how to perform that copy to test this approach.

Comment 12

[Mike Hommey [:glandium]]

You probably only need something like https://searchfox.org/mozilla-central/rev/3269d4c928ef0d8310c2f57634e9b6057aa636e9/browser/installer/package-manifest.in#434-436 in the equivalent manifest for android.

Comment 13

[Jesse Schwartzentruber (:truber)]

(In reply to Mike Hommey [:glandium] from comment #12)

You probably only need something like https://searchfox.org/mozilla-central/rev/3269d4c928ef0d8310c2f57634e9b6057aa636e9/browser/installer/package-manifest.in#434-436 in the equivalent manifest for android.

This is already in the patch, but only includes llvm-symbolizer in jsshell.zip not in the APK.
https://phabricator.services.mozilla.com/D142379#change-ZxaU6bnHGZCc

Comment 14

[Mike Hommey [:glandium]]

Do you have a link to a try?

Comment 15

[Jesse Schwartzentruber (:truber)]

(In reply to Mike Hommey [:glandium] from comment #14)

Do you have a link to a try?

https://treeherder.mozilla.org/jobs?repo=try&revision=f25630e02e695420826f7a88a1dfa49628155646