Startup crash on ASan builds
Categories
(Firefox Build System :: Android Studio and Gradle Integration, defect, P2)
Tracking
(Not tracked)
People
(Reporter: tsmith, Unassigned)
References
(Depends on 1 open bug, Blocks 1 open bug)
Details
(Whiteboard: [fuzzblocker][geckoview])
Attachments
(9 files)
|
45.92 KB,
text/plain
|
Details | |
|
106.53 KB,
text/plain
|
Details | |
|
16.78 KB,
text/plain
|
Details | |
|
45.62 KB,
text/plain
|
Details | |
|
10.06 KB,
text/plain
|
Details | |
|
10.06 KB,
text/plain
|
Details | |
|
79.64 KB,
application/octet-stream
|
Details | |
|
134.90 KB,
application/octet-stream
|
Details | |
|
769.48 KB,
application/octet-stream
|
Details |
This happens frequently when launching GVE. We are setting ASAN_OPTIONS=log_path=/sdcard/asan.log. The logs contain an error message but no stack. We are using the build from https://firefox-ci-tc.services.mozilla.com/tasks/index/gecko.v2.mozilla-central.latest.mobile/android-x86_64-fuzzing-asan
Any ideas or tips for debugging would be helpful.
Contents of asan.log
at /sdcard/sanitizer_logs/report.log.8100 <
=================================================================
==8100==ERROR: AddressSanitizer: SEGV on unknown address 0x634db0180018 (pc 0x7a40cf152155 bp 0x000082f34aae sp 0x7ffe4c039580 T0)
==8100==The signal is caused by a READ memory access.
at /sdcard/sanitizer_logs/report.log.8100 <
=================================================================
==8100==ERROR: AddressSanitizer: SEGV on unknown address 0x634db0180018 (pc 0x7a40cf152155 bp 0x000082f34aae sp 0x7ffe4c039580 T0)
==8100==The signal is caused by a READ memory access.
|
||
Comment 1•4 years ago
|
||
I don't see anything helpful in the logs. You could try to inspect libxul.so to see what symbol 0x634db0180018 corresponds to. Is there a guide on how to run ASAN on android? Maybe I can look at it.
|
|
||
Comment 2•4 years ago
|
||
Moving to General as I think it's unlikely that this is specific to GVE and more of a generic GV issue.
|
Reporter | |
Comment 3•4 years ago
•
|
||
There is nothing extra that needs to be done to run with ASan. Use an ASan build on Android 9 or later (We have been using 9). I am also able to reproduce this by opening and closing GVE about 6 or 7 times manually. The empty white screen (no url bar) is shown when the crash happens.
Agi, can you try the .apk from the link in description?
|
|
||
Comment 4•4 years ago
|
||
Sorry I meant how to build this locally, I would need symbols / a debugger to see what's going on here.
|
|
Reporter | |
Comment 5•4 years ago
|
||
Jesse knows more about the build process than I do.
|
|
||
Comment 6•4 years ago
|
||
I just ran the APK on my emulator and I see this:
08-12 15:37:09.211 3640 3640 F DEBUG : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
08-12 15:37:09.211 3640 3640 F DEBUG : Build fingerprint: 'Android/sdk_phone_x86_64/generic_x86_64:8.1.0/OSM1.180201.023/4931629:userdebug/test-keys'
08-12 15:37:09.211 3640 3640 F DEBUG : Revision: '0'
08-12 15:37:09.211 3640 3640 F DEBUG : ABI: 'x86_64'
08-12 15:37:09.211 3640 3640 F DEBUG : pid: 3629, tid: 3629, name: app_process64 >>> /system/bin/app_process64 <<<
08-12 15:37:09.211 3640 3640 F DEBUG : signal 31 (SIGSYS), code 1 (SYS_SECCOMP), fault addr --------
08-12 15:37:09.211 3640 3640 F DEBUG : Cause: seccomp prevented call to disallowed x86_64 system call 0
08-12 15:37:09.211 3640 3640 F DEBUG : rax 0000000000000059 rbx 00007a4e3b0aa0b0 rcx ffffffffffffffff rdx 0000000000001000
08-12 15:37:09.211 3640 3640 F DEBUG : rsi 00007a4e3b0aa0b0 rdi 00007a4e3af67b90
08-12 15:37:09.211 3640 3640 F DEBUG : r8 00007a4e389b19c0 r9 0000000000000000 r10 0000000080000000 r11 0000000000000246
08-12 15:37:09.211 3640 3640 F DEBUG : r12 00007a4e3cc56394 r13 00007a4e38971e90 r14 0000000000001000 r15 00007a4e3cc59134
08-12 15:37:09.211 3640 3640 F DEBUG : cs 0000000000000033 ss 000000000000002b
08-12 15:37:09.211 3640 3640 F DEBUG : rip 00007a4e3af9bc1e rbp 0000000000000001 rsp 00007fffe8f1e1c0 eflags 0000000000000246
08-12 15:37:09.213 3640 3640 F DEBUG :
08-12 15:37:09.213 3640 3640 F DEBUG : backtrace:
08-12 15:37:09.213 3640 3640 F DEBUG : #00 pc 0000000000055c1e /data/app/org.mozilla.geckoview_example-8FhkumkJ0mc0kcK3GN5BXw==/lib/x86_64/libclang_rt.asan-x86_64-android.so (offset 0x4b000) (__sanitizer::ReadBinaryName(char*, unsigned long)+30)
08-12 15:37:09.213 3640 3640 F DEBUG : #01 pc 000000000004d8be /data/app/org.mozilla.geckoview_example-8FhkumkJ0mc0kcK3GN5BXw==/lib/x86_64/libclang_rt.asan-x86_64-android.so (offset 0x4b000) (__sanitizer::CacheBinaryName()+30)
08-12 15:37:09.213 3640 3640 F DEBUG : #02 pc 00000000000c47d8 /data/app/org.mozilla.geckoview_example-8FhkumkJ0mc0kcK3GN5BXw==/lib/x86_64/libclang_rt.asan-x86_64-android.so (offset 0x4b000) (__asan::AsanInitInternal()+72)
08-12 15:37:09.213 3640 3640 F DEBUG : #03 pc 0000000000096bba /data/app/org.mozilla.geckoview_example-8FhkumkJ0mc0kcK3GN5BXw==/lib/x86_64/libclang_rt.asan-x86_64-android.so (offset 0x4b000) (pthread_mutex_lock+42)
08-12 15:37:09.213 3640 3640 F DEBUG : #04 pc 00000000000aaeeb /system/lib64/libc.so (jemalloc_constructor+91)
08-12 15:37:09.213 3640 3640 F DEBUG : #05 pc 0000000000027a9f /system/bin/linker64 (__dl__ZL10call_arrayIPFviPPcS1_EEvPKcPT_mbS5_+255)
08-12 15:37:09.213 3640 3640 F DEBUG : #06 pc 0000000000027ce9 /system/bin/linker64 (__dl__ZN6soinfo17call_constructorsEv+441)
08-12 15:37:09.213 3640 3640 F DEBUG : #07 pc 0000000000027bc8 /system/bin/linker64 (__dl__ZN6soinfo17call_constructorsEv+152)
08-12 15:37:09.213 3640 3640 F DEBUG : #08 pc 0000000000027bc8 /system/bin/linker64 (__dl__ZN6soinfo17call_constructorsEv+152)
08-12 15:37:09.213 3640 3640 F DEBUG : #09 pc 00000000000237e0 /system/bin/linker64 (__dl___linker_init+3712)
08-12 15:37:09.213 3640 3640 F DEBUG : #10 pc 000000000002a5e7 /system/bin/linker64 (_start+7)
08-12 15:37:09.213 3640 3640 F DEBUG : #11 pc 0000000000000007 <unknown>
|
|
||
Comment 7•4 years ago
|
||
On a API 29 device I get a slightly different stack
08-12 15:43:26.882 4223 4223 F DEBUG : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
08-12 15:43:26.882 4223 4223 F DEBUG : Build fingerprint: 'Android/sdk_phone_x86_64/generic_x86_64:10/QPP6.190730.005.B1/5775370:userdebug/test-keys'
08-12 15:43:26.882 4223 4223 F DEBUG : Revision: '0'
08-12 15:43:26.882 4223 4223 F DEBUG : ABI: 'x86_64'
08-12 15:43:26.883 4223 4223 F DEBUG : Timestamp: 2020-08-12 15:43:26-0700
08-12 15:43:26.883 4223 4223 F DEBUG : pid: 4209, tid: 4209, name: app_process64 >>> /system/bin/app_process64 <<<
08-12 15:43:26.883 4223 4223 F DEBUG : uid: 10103
08-12 15:43:26.883 4223 4223 F DEBUG : signal 31 (SIGSYS), code 1 (SYS_SECCOMP), fault addr --------
08-12 15:43:26.883 4223 4223 F DEBUG : Cause: seccomp prevented call to disallowed x86_64 system call 4
08-12 15:43:26.883 4223 4223 F DEBUG : rax 0000000000000004 rbx 00007d77b26ed5a9 rcx 00007d77b2714d81 rdx 0000000000000000
08-12 15:43:26.883 4223 4223 F DEBUG : r8 0000000000000004 r9 00007d77b26e7d60 r10 0000000000001000 r11 0000000000000246
08-12 15:43:26.883 4223 4223 F DEBUG : r12 000000000000000a r13 00007d77b2825108 r14 00007d77b28a57c8 r15 0000000000000000
08-12 15:43:26.883 4223 4223 F DEBUG : rdi 00007d77b26ed5a9 rsi 00007fffbc356130
08-12 15:43:26.883 4223 4223 F DEBUG : rbp 00007fffbc356b40 rsp 00007fffbc356130 rip 00007d77b2714d81
08-12 15:43:26.888 4223 4223 F DEBUG :
08-12 15:43:26.888 4223 4223 F DEBUG : backtrace:
08-12 15:43:26.888 4223 4223 F DEBUG : #00 pc 0000000000054d81 /data/app/org.mozilla.geckoview_example-Y8ALcxp_S8euOzIZGJ7Upw==/lib/x86_64/libclang_rt.asan-x86_64-android.so (__sanitizer::FileExists(char const*)+33)
08-12 15:43:26.888 4223 4223 F DEBUG : #01 pc 00000000000501c2 /data/app/org.mozilla.geckoview_example-Y8ALcxp_S8euOzIZGJ7Upw==/lib/x86_64/libclang_rt.asan-x86_64-android.so (__sanitizer::FindPathToBinary(char const*)+18)
08-12 15:43:26.888 4223 4223 F DEBUG : #02 pc 00000000000629e6 /data/app/org.mozilla.geckoview_example-Y8ALcxp_S8euOzIZGJ7Upw==/lib/x86_64/libclang_rt.asan-x86_64-android.so (__sanitizer::Symbolizer::PlatformInit()+406)
08-12 15:43:26.888 4223 4223 F DEBUG : #03 pc 0000000000060a33 /data/app/org.mozilla.geckoview_example-Y8ALcxp_S8euOzIZGJ7Upw==/lib/x86_64/libclang_rt.asan-x86_64-android.so (__sanitizer::Symbolizer::GetOrInit()+51)
08-12 15:43:26.888 4223 4223 F DEBUG : #04 pc 0000000000062c05 /data/app/org.mozilla.geckoview_example-Y8ALcxp_S8euOzIZGJ7Upw==/lib/x86_64/libclang_rt.asan-x86_64-android.so (__sanitizer::Symbolizer::LateInitialize()+5)
08-12 15:43:26.888 4223 4223 F DEBUG : #05 pc 00000000000c4980 /data/app/org.mozilla.geckoview_example-Y8ALcxp_S8euOzIZGJ7Upw==/lib/x86_64/libclang_rt.asan-x86_64-android.so (__asan::AsanInitInternal()+496)
08-12 15:43:26.888 4223 4223 F DEBUG : #06 pc 000000000007dc63 /data/app/org.mozilla.geckoview_example-Y8ALcxp_S8euOzIZGJ7Upw==/lib/x86_64/libclang_rt.asan-x86_64-android.so (strcmp+915)
08-12 15:43:26.888 4223 4223 F DEBUG : #07 pc 000000000008af10 /apex/com.android.runtime/lib64/bionic/libc.so (__libc_init_vdso(libc_globals*)+528) (BuildId: a08a19770d6696739c847e29c3f5f650)
08-12 15:43:26.888 4223 4223 F DEBUG : #08 pc 000000000009fd65 /apex/com.android.runtime/lib64/bionic/libc.so (__libc_init_globals()+85) (BuildId: a08a19770d6696739c847e29c3f5f650)
08-12 15:43:26.888 4223 4223 F DEBUG : #09 pc 000000000008a886 /apex/com.android.runtime/lib64/bionic/libc.so (__libc_preinit_impl()+38) (BuildId: a08a19770d6696739c847e29c3f5f650)
08-12 15:43:26.888 4223 4223 F DEBUG : #10 pc 0000000000065caf /apex/com.android.runtime/bin/linker64 (__dl__ZL10call_arrayIPFviPPcS1_EEvPKcPT_mbS5_+255) (BuildId: 8c58e8673bbdf607f2614ae6235399f5)
08-12 15:43:26.888 4223 4223 F DEBUG : #11 pc 0000000000065ef1 /apex/com.android.runtime/bin/linker64 (__dl__ZN6soinfo17call_constructorsEv+433) (BuildId: 8c58e8673bbdf607f2614ae6235399f5)
08-12 15:43:26.888 4223 4223 F DEBUG : #12 pc 0000000000065dd8 /apex/com.android.runtime/bin/linker64 (__dl__ZN6soinfo17call_constructorsEv+152) (BuildId: 8c58e8673bbdf607f2614ae6235399f5)
08-12 15:43:26.888 4223 4223 F DEBUG : #13 pc 0000000000065dd8 /apex/com.android.runtime/bin/linker64 (__dl__ZN6soinfo17call_constructorsEv+152) (BuildId: 8c58e8673bbdf607f2614ae6235399f5)
08-12 15:43:26.888 4223 4223 F DEBUG : #14 pc 000000000006185c /apex/com.android.runtime/bin/linker64 (__dl__ZL29__linker_init_post_relocationR19KernelArgumentBlockR6soinfo+4348) (BuildId: 8c58e8673bbdf607f2614ae6235399f5)
08-12 15:43:26.888 4223 4223 F DEBUG : #15 pc 0000000000060712 /apex/com.android.runtime/bin/linker64 (__dl___linker_init+434) (BuildId: 8c58e8673bbdf607f2614ae6235399f5)
08-12 15:43:26.888 4223 4223 F DEBUG : #16 pc 0000000000068ab7 /apex/com.android.runtime/bin/linker64 (__dl__start+7) (BuildId: 8c58e8673bbdf607f2614ae6235399f5)
|
|
||
Comment 8•4 years ago
|
||
It looks like the asan code is trying to find the binary name but fails to do so?
Also do we have arm64 builds for this? I'm wondering if this is just an emulator quirk.
|
|
Reporter | |
Comment 9•4 years ago
|
||
We are also using the simulator. Hmm we run with SELinux set to Permissive maybe that is the issue you are seeing. You can try adb shell setenforce 0 as root.
|
||
Comment 10•4 years ago
|
||
(In reply to Tyson Smith [:tsmith] from comment #9)
You can try
adb shell setenforce 0as root.
Disabling seccomp is the answer. This is an upstream bug in AddressSanitizer (https://github.com/google/sanitizers/issues/1101). I think it only affects x86_64. We don't have arm64 builds yet.
|
|
||
Comment 11•4 years ago
|
||
(In reply to Agi Sferro | :agi | ⏰ PST | he/him from comment #4)
Sorry I meant how to build this locally, I would need symbols / a debugger to see what's going on here.
It's been a while, but the only thing you should need is the clang toolchain with android runtimes included (linux64-clang-android-cross), and a mozconfig that uses it (like android-x86_64-nightly-fuzzing-asan).
|
||
Updated•4 years ago
|
|
|
Reporter | |
Updated•4 years ago
|
|
|
||
Updated•4 years ago
|
|
|
||
Updated•4 years ago
|
|
|
||
Comment 12•4 years ago
|
||
Copying discussion from matrix
truber: the clang needs to have the runtime libs for android built. you should be able to do this using build-clang.py and clang-11-android.json on mac if you have the android ndk installed, but I've only done it on Linux and it's been a while
as for why bootstrap doesn't include it, we don't build it in taskcluster because it's only been done on Linux afaik. and also ... bootstrap doesn't pull it on Linux either and I don't know why
I download the clang-android-cross package from the firefox-ci cache and extract it in ~/.mozbuild myselftruber: You might also be able to extract the linux64-clang-11-android-cross over the mac one, with tar -k so only the android runtimes are extracted.
|
|
||
Comment 13•4 years ago
|
||
This sounds very much like a clang bug. I'm trying clang-10 to see if it's fixed there.
|
|
||
Comment 14•4 years ago
|
||
Different startup crash in clang-10:
10-29 12:40:46.210 4608 4608 F DEBUG : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
10-29 12:40:46.210 4608 4608 F DEBUG : Build fingerprint: 'Android/sdk_phone_x86_64/generic_x86_64:10/QPP6.190730.005.B1/5775370:userdebug/test-keys'
10-29 12:40:46.210 4608 4608 F DEBUG : Revision: '0'
10-29 12:40:46.210 4608 4608 F DEBUG : ABI: 'x86_64'
10-29 12:40:46.214 4608 4608 F DEBUG : Timestamp: 2020-10-29 12:40:46-0700
10-29 12:40:46.214 4608 4608 F DEBUG : pid: 4596, tid: 4596, name: app_process64 >>> /system/bin/app_process64 <<<
10-29 12:40:46.214 4608 4608 F DEBUG : uid: 10105
10-29 12:40:46.214 4608 4608 F DEBUG : signal 31 (SIGSYS), code 1 (SYS_SECCOMP), fault addr --------
10-29 12:40:46.214 4608 4608 F DEBUG : Cause: seccomp prevented call to disallowed x86_64 system call 4
10-29 12:40:46.214 4608 4608 F DEBUG : rax 0000000000000004 rbx 00007c6007cae76a rcx 00007c6007cd765c rdx 0000000000000000
10-29 12:40:46.214 4608 4608 F DEBUG : r8 0000000000000004 r9 00007c6007ca7515 r10 0000000000008000 r11 0000000000000246
10-29 12:40:46.214 4608 4608 F DEBUG : r12 000000000000000a r13 00007c6007de8448 r14 00007c6007e68b18 r15 0000000000000000
10-29 12:40:46.214 4608 4608 F DEBUG : rdi 00007c6007cae76a rsi 00007ffd11a8e440
10-29 12:40:46.214 4608 4608 F DEBUG : rbp 00007c6007cae76a rsp 00007ffd11a8e440 rip 00007c6007cd765c
10-29 12:40:46.215 4608 4608 F DEBUG :
10-29 12:40:46.215 4608 4608 F DEBUG : backtrace:
10-29 12:40:46.215 4608 4608 F DEBUG : #00 pc 000000000005565c /data/app/org.mozilla.geckoview_example-2aj5MfS-jZfzx6Ib8lr7qw==/lib/x86_64/libclang_rt.asan-x86_64-android.so (__sanitizer::internal_strlcpy(char*, char const*, unsigned long)+956)
Just noticed there's 11 available, so trying that.
|
|
||
Comment 15•4 years ago
|
||
Ah ok so that's seccomp ^, disabling it I finally get GVE running!
|
|
||
Comment 16•4 years ago
|
||
When built with clang-11 from taskcluster I cannot reproduce crashes anymore. :tsmith is there anything specific that you're doing to trigger a crash? does it still crash for you?
|
|
||
Comment 17•4 years ago
|
||
What emulator version are you using? When this was filed, we were using android-28, but now the latest is android-30. I still observe the crash with a recent taskcluster build on android-28, but on android-30, it doesn't look like ASAN is loaded at all. When I go to about:crashcontent (or about:crashparent) I don't see an ASAN backtrace as expected.
|
|
||
Comment 18•4 years ago
|
||
I'm using android-29. Navigating to about:crashparent gets me this (which seems to indicate that ASAN is running?)
10-30 12:45:26.655 28668 28668 F DEBUG : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
10-30 12:45:26.655 28668 28668 F DEBUG : Build fingerprint: 'Android/sdk_phone_x86_64/generic_x86_64:10/QPP6.190730.005.B1/5775370:userdebug/test-keys'
10-30 12:45:26.655 28668 28668 F DEBUG : Revision: '0'
10-30 12:45:26.655 28668 28668 F DEBUG : ABI: 'x86_64'
10-30 12:45:26.656 28668 28668 F DEBUG : Timestamp: 2020-10-30 12:45:26-0700
10-30 12:45:26.656 28668 28668 F DEBUG : pid: 7101, tid: 7146, name: Web Content >>> org.mozilla.geckoview_example:tab0 <<<
10-30 12:45:26.656 28668 28668 F DEBUG : uid: 10106
10-30 12:45:26.656 28668 28668 F DEBUG : signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
10-30 12:45:26.656 28668 28668 F DEBUG : Abort message: '=================================================================
10-30 12:45:26.656 28668 28668 F DEBUG : ==7101==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x743240561284 bp 0x743249501c90 sp 0x743249501b80 T15)
10-30 12:45:26.656 28668 28668 F DEBUG : ==7101==The signal is caused by a WRITE memory access.
10-30 12:45:26.656 28668 28668 F DEBUG : ==7101==Hint: address points to the zero page.
10-30 12:45:26.656 28668 28668 F DEBUG : #0 0x743240561284 (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0x15820284)
10-30 12:45:26.656 28668 28668 F DEBUG : #1 0x7432336010cd (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0x88c00cd)
10-30 12:45:26.656 28668 28668 F DEBUG : #2 0x7432335e8696 (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0x88a7696)
10-30 12:45:26.656 28668 28668 F DEBUG : #3 0x743235ac24e8 (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0xad814e8)
10-30 12:45:26.656 28668 28668 F DEBUG : #4 0x743240522d34 (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0x157e1d34)
10-30 12:45:26.656 28668 28668 F DEBUG : #5 0x74324051b254 (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0x157da254)
10-30 12:45:26.656 28668 28668 F DEBUG : #6 0x74324046585f (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0x1572485f)
10-30 12:45:26.656 28668 28668 F DEBUG : #7 0x7432404b83b5 (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0x157773b5)
10-30 12:45:26.656 28668 28668 F DEBUG : #8 0x743240460a8b (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0x1571fa8b)
10-30 12:45:26.656 28668 28668 F DEBUG : #9 0x74323bed0781 (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0x1118f781)
10-30 12:45:26.656 28668 28668 F DEBUG : #10 0x743234c1f608 (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0x9ede608)
10-30 12:45:26.656 28668 28668 F DEBUG : #11 0x7432349749d8 (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0x9c339d8)
10-30 12:45:26.656 28668 28668 F DEBUG : #12 0x743234970972 (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0x9c2f972)
10-30 12:45:26.656 28668 28668 F DEBUG : #13 0x7432349729f1 (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0x9c319f1)
10-30 12:45:26.656 28668 28668 F DEBUG : #14 0x74323497335d (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0x9c3235d)
10-30 12:45:26.656 28668 28668 F DEBUG : #15 0x743233348c67 (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0x8607c67)
10-30 12:45:26.656 28668 28668 F DEBUG : #16 0x74323333e3dd (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0x85fd3dd)
10-30 12:45:26.656 28668 28668 F DEBUG : #17 0x74323333b585 (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0x85fa585)
10-30 12:45:26.656 28668 28668 F DEBUG : #18 0x74323333bb1c (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0x85fab1c)
10-30 12:45:26.656 28668 28668 F DEBUG : #19 0x74323333fc74 (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0x85fec74)
10-30 12:45:26.656 28668 28668 F DEBUG : #20 0x74323336ff72 (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0x862ef72)
10-30 12:45:26.656 28668 28668 F DEBUG : #21 0x74323337a2a1 (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0x86392a1)
10-30 12:45:26.656 28668 28668 F DEBUG : #22 0x74323497cedc (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0x9c3bedc)
10-30 12:45:26.656 28668 28668 F DEBUG : #23 0x7432347fd2c2 (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0x9abc2c2)
10-30 12:45:26.656 28668 28668 F DEBUG : #24 0x74323c9f283a (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0x11cb183a)
10-30 12:45:26.656 28668 28668 F DEBUG : #25 0x7432411f164f (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0x164b064f)
10-30 12:45:26.656 28668 28668 F DEBUG : #26 0x7432347fd2c2 (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0x9abc2c2)
10-30 12:45:26.656 28668 28668 F DEBUG : #27 0x7432411f0503 (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libxul.so+0x164af503)
10-30 12:45:26.656 28668 28668 F DEBUG : #28 0x743248922ca1 (/data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libmozglue.so+0x122ca1)
10-30 12:45:26.656 28668 28668 F DEBUG : #29 0x74329e8be641 (/apex/com.android.runtime/lib64/libart.so+0x174641)
10-30 12:45:26.656 28668 28668 F DEBUG : #30 0x12eda9d7 ([anon:dalvik-main space (region space)]+0x1a9d7)
10-30 12:45:26.656 28668 28668 F DEBUG :
10-30 12:45:26.656 28668 28668 F DEBUG : AddressSanitizer can not provide additional info.
10-30 12:45:26.656 28668 28668 F DEBUG : SUMMA
10-30 12:45:26.656 28668 28668 F DEBUG : rax 0000000000000000 rbx 0000000000001bbd rcx 00007433242973f8 rdx 0000000000000006
10-30 12:45:26.656 28668 28668 F DEBUG : r8 0000000000000000 r9 0000000000000000 r10 0000743296d59dc0 r11 0000000000000246
10-30 12:45:26.656 28668 28668 F DEBUG : r12 00007433261eafc8 r13 0000000000000000 r14 0000743296d59e48 r15 0000000000001bea
10-30 12:45:26.656 28668 28668 F DEBUG : rdi 0000000000001bbd rsi 0000000000001bea
10-30 12:45:26.656 28668 28668 F DEBUG : rbp 0000743296d5abb0 rsp 0000743296d59db8 rip 00007433242973f8
10-30 12:45:26.869 28668 28668 F DEBUG :
10-30 12:45:26.869 28668 28668 F DEBUG : backtrace:
10-30 12:45:26.869 28668 28668 F DEBUG : #00 pc 00000000000943f8 /apex/com.android.runtime/lib64/bionic/libc.so (syscall+24) (BuildId: a08a19770d6696739c847e29c3f5f650)
10-30 12:45:26.869 28668 28668 F DEBUG : #01 pc 0000000000097146 /apex/com.android.runtime/lib64/bionic/libc.so (abort+182) (BuildId: a08a19770d6696739c847e29c3f5f650)
10-30 12:45:26.869 28668 28668 F DEBUG : #02 pc 000000000005d4f1 /data/app/org.mozilla.geckoview_example-NarqLKpVRLzBU1FaGKHkhA==/lib/x86_64/libclang_rt.asan-x86_64-android.so (__sanitizer::BackgroundThread(void*)+945)
10-30 12:45:26.869 28668 28668 F DEBUG : #03 pc 0000000000007a07 [anon:SetAlternateSignalStack]
|
|
||
Comment 19•4 years ago
|
||
Sorry, I missed that you said API 29 already in comment 7.
Your stack does look good. I'll try to reproduce in API 29, but it could be that this is API 28 specific, and we should open a new issue for lack of ASAN in API 30. If it works in API 29 that at least gives us a good platform for fuzzing.
|
|
||
Comment 20•4 years ago
|
||
:truber let me know if this is good enough for you in 29.
|
|
||
Comment 21•4 years ago
|
||
I'm seeing the same thing originally reported in 29. Below I've tried launching several times to see if I could get any consistency, but it's very unpredictable.
Launch #1:
- read SEGV on unknown address 0x619db0b04658 in
idmap2on launch - gve launches normally and shows
about:blank about:crashcontentcauses read SEGV at 0x0 with ASAN traceback in logcat
Launch #2:
- read SEGV on unknown address 0x60c058c86b60 in app_process64
- emulator shows whitescreen (launcher unresponsive for a minute)
Launch #3:
- read SEGV on unknown address 0x617c19fc5b60 in app_process64
- gve launches but tab has no address (
Enter URL or search keywords) about:crashcontentdoes nothing
Launch #4: same as #1
Launch #5: same as #2
Launch #6: same as #2
-- restarted emulator
Launch #7: same as #2
Launch #8: same as #2
Launch #9: same as #3
Launch #10: same as #1
Launch #11: same as #1
Launch #12: same as #1
Launch #13: same as #1
-- restarted emulator
Launch #14: same as #2
Launch #15: same as #1
Launch #16: same as #1
|
|
||
Comment 22•4 years ago
|
||
The above is using system image aosp 29.1.7, emulator 30.1.5.
|
|
||
Comment 23•4 years ago
|
||
Thanks! Jesse, could you attach some logs for the crash? We now dump the raw stacktrace in the logs. Also it would be nice if you could tell me what exact steps you do to reproduce, because I cannot reproduce locally (starting from a clean emulator, I'm assuming you're using the emulator from Android Studio?) Also which version of GeckoViewExample are you testing? (link to the APK would be great).
I'd wager that a SEGV on startup is likely this test we do to find out if signal handling works: https://searchfox.org/mozilla-central/rev/02cb78667e87ccc42fea5edc6f3f2dd2edd6ecd5/mozglue/linker/ElfLoader.cpp#1310
We don't really need this anymore, AFAIK.
|
|
||
Updated•4 years ago
|
| Comment hidden (spam) |
| Comment hidden (spam) |
|
|
||
Comment 27•3 years ago
|
||
(In reply to James Willcox (:snorp) (jwillcox@mozilla.com) (he/him) from comment #24)
I'd wager that a SEGV on startup is likely this test we do to find out if signal handling works: https://searchfox.org/mozilla-central/rev/02cb78667e87ccc42fea5edc6f3f2dd2edd6ecd5/mozglue/linker/ElfLoader.cpp#1310
We don't really need this anymore, AFAIK.
mmh. If it was that I would see it though, I think?, I definitely hit that code path when debugging for example.
|
||
Comment 28•3 years ago
|
||
(In reply to James Willcox (:snorp) (jwillcox@mozilla.com) (he/him) from comment #24)
I'd wager that a SEGV on startup is likely this test we do to find out if signal handling works: https://searchfox.org/mozilla-central/rev/02cb78667e87ccc42fea5edc6f3f2dd2edd6ecd5/mozglue/linker/ElfLoader.cpp#1310
We don't really need this anymore, AFAIK.
Actually, we do, wasm uses the result from that test: https://searchfox.org/mozilla-central/source/js/src/wasm/WasmSignalHandlers.cpp#1019
|
|
||
Comment 29•3 years ago
|
||
(In reply to Agi Sferro | :agi | ⏰ PST | he/him from comment #23)
Thanks! Jesse, could you attach some logs for the crash? We now dump the raw stacktrace in the logs. Also it would be nice if you could tell me what exact steps you do to reproduce, because I cannot reproduce locally (starting from a clean emulator, I'm assuming you're using the emulator from Android Studio?) Also which version of GeckoViewExample are you testing? (link to the APK would be great).
Sure. Normally we use scripts to automate AVD creation and launch, but I've reproduced all three cases in Studio too to make sure it isn't our scripts.
STR:
- clean emulator in Android Studio
- I'm cloning the Pixel device, and changing memory to 6Gb and internal storage to 5Gb
- the system image I used is the Intel Atom_64 for Pie (28) .. not the Google APIs image.
- launch emulator and disable seccomp:
adb root
adb shell setenforce 0
adb shell stop
adb shell start
adb unroot - load the APK with:
File>Profile or Debug APK - I had to manually specify an SDK to get it to launch, that's in
File>Project Structure...underProject Settings>Modules>geckoview_example.apk>Dependencies, I set it to the latest SDK studio installed by default, which is 30. - Then click the Run/Stop buttons while watching logcat
The APK I got from:
https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.mobile.android-x86_64-fuzzing-asan/artifacts/public/build/geckoview_example.apk
.. which pointed to:
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/QovgR-oQQAOlvNd8xKkHCg/runs/0/artifacts/public%2Fbuild%2Fgeckoview_example.apk
.. at the time I downloaded it.
|
|
||
Comment 30•3 years ago
|
||
|
|
||
Comment 31•3 years ago
|
||
|
|
||
Comment 32•3 years ago
|
||
|
|
||
Comment 33•3 years ago
|
||
I can finally reproduce this! I'll take a look on monday.
|
|
||
Updated•3 years ago
|
|
|
||
Comment 34•3 years ago
|
||
Adding a sleep and attaching a debugger at startup makes this problem go away, maybe it's a race condition?
|
|
||
Comment 35•3 years ago
|
||
(In reply to James Willcox (:snorp) (jwillcox@mozilla.com) (he/him) from comment #24)
I'd wager that a SEGV on startup is likely this test we do to find out if signal handling works: https://searchfox.org/mozilla-central/rev/02cb78667e87ccc42fea5edc6f3f2dd2edd6ecd5/mozglue/linker/ElfLoader.cpp#1310
We don't really need this anymore, AFAIK.
I tried to remove this and the problem still persists, the SEGV at startup doesn't seem responsible for this.
|
|
||
Comment 36•3 years ago
|
||
Apparently there need to be special code in wrap.sh to enable debugging, which explains why I was having a hard time debugging startup.
Opened Bug 1686514 for that.
|
|
||
Comment 37•3 years ago
|
||
This is in the logs when the problem happens:
I wrap.sh : AddressSanitizer: nested bug in the same thread, aborting.
Looking at the ASAN code, it seems like this is a race condition: https://chromium.googlesource.com/chromiumos/third_party/compiler-rt/+/59a9c97922c02a4cd76893a8d55614d5a3814d29/lib/asan/asan_report.cc#651
I'm gonna try compiling my own clang to add some more info there.
|
|
||
Comment 38•3 years ago
|
||
Not sure if this is helpful yet but I was able to get more verbose logging from ASAN
01-15 12:24:43.894 9062 9062 I wrap.sh : ASAN_OPTIONS: abort_on_error=1,debug=1,print_stats=1,log_path=stderr,verbosity=1,allow_user_segv_handler=1,alloc_dealloc_mismatch=0,detect_leaks=0,fast_unwind_on_check=1,fast_unwind_on_fatal=1,max_free_fill_size=268435456,max_malloc_fill_size=268435456,malloc_fill_byte=228,free_fill_byte=229,handle_sigill=1,allocator_may_return_null=1,log_to_syslog=false
01-15 12:24:43.897 9062 9062 I wrap.sh : LD_PRELOAD: /data/app/org.mozilla.geckoview_example-l0l17csdvhvtSbGvwc-nlQ==/lib/x86_64/libclang_rt.asan-x86_64-android.so
01-15 12:24:43.914 9062 9062 I wrap.sh : ==9070==AddressSanitizer: failed to intercept '__strndup'
01-15 12:24:43.914 9062 9062 I wrap.sh : '==9070==AddressSanitizer: failed to intercept '__strxfrm_l'
01-15 12:24:43.914 9062 9062 I wrap.sh : '==9070==AddressSanitizer: failed to intercept 'bcmp'
01-15 12:24:43.914 9062 9062 I wrap.sh : '==9070==AddressSanitizer: failed to intercept 'wait3'
01-15 12:24:43.914 9062 9062 I wrap.sh : '==9070==AddressSanitizer: failed to intercept '__wait4'
01-15 12:24:43.914 9062 9062 I wrap.sh : '==9070==AddressSanitizer: failed to intercept 'ftime'
01-15 12:24:43.914 9062 9062 I wrap.sh : '==9070==AddressSanitizer: failed to intercept 'pthread_setcancelstate'
01-15 12:24:43.914 9062 9062 I wrap.sh : '==9070==AddressSanitizer: failed to intercept 'pthread_setcanceltype'
01-15 12:24:43.914 9062 9062 I wrap.sh : '==9070==AddressSanitizer: failed to intercept 'getutid'
01-15 12:24:43.914 9062 9062 I wrap.sh : '==9070==AddressSanitizer: failed to intercept 'getutline'
01-15 12:24:43.914 9062 9062 I wrap.sh : '==9070==AddressSanitizer: failed to intercept '__wcsxfrm_l'
01-15 12:24:43.914 9062 9062 I wrap.sh : '==9070==AddressSanitizer: failed to intercept 'bsd_signal'
01-15 12:24:43.914 9062 9062 I wrap.sh : '==9070==AddressSanitizer: failed to intercept 'index'
01-15 12:24:43.914 9062 9062 I wrap.sh : '==9070==AddressSanitizer: libc interceptors initialized
01-15 12:24:43.915 9062 9062 I wrap.sh : || `[0x10007fff8000, 0x7fffffffffff]` || HighMem ||
01-15 12:24:43.915 9062 9062 I wrap.sh : || `[0x02008fff7000, 0x10007fff7fff]` || HighShadow ||
01-15 12:24:43.915 9062 9062 I wrap.sh : || `[0x00008fff7000, 0x02008fff6fff]` || ShadowGap ||
01-15 12:24:43.915 9062 9062 I wrap.sh : || `[0x00007fff8000, 0x00008fff6fff]` || LowShadow ||
01-15 12:24:43.915 9062 9062 I wrap.sh : || `[0x000000000000, 0x00007fff7fff]` || LowMem ||
01-15 12:24:43.915 9062 9062 I wrap.sh : MemToShadow(shadow): 0x00008fff7000 0x000091ff6dff 0x004091ff6e00 0x02008fff6fff
01-15 12:24:43.915 9062 9062 I wrap.sh : redzone=16
01-15 12:24:43.915 9062 9062 I wrap.sh : max_redzone=2048
01-15 12:24:43.915 9062 9062 I wrap.sh : quarantine_size_mb=16M
01-15 12:24:43.915 9062 9062 I wrap.sh : thread_local_quarantine_size_kb=64K
01-15 12:24:43.915 9062 9062 I wrap.sh : malloc_context_size=30
01-15 12:24:43.915 9062 9062 I wrap.sh : SHADOW_SCALE: 3
01-15 12:24:43.915 9062 9062 I wrap.sh : SHADOW_GRANULARITY: 8
01-15 12:24:43.915 9062 9062 I wrap.sh : SHADOW_OFFSET: 0x7fff8000
01-15 12:24:43.915 9062 9062 I wrap.sh : ==9070==Installed the sigaction for signal 11
01-15 12:24:43.915 9062 9062 I wrap.sh : ==9070==Installed the sigaction for signal 7
01-15 12:24:43.915 9062 9062 I wrap.sh : ==9070==Installed the sigaction for signal 8
01-15 12:24:43.915 9062 9062 I wrap.sh : ==9070==Installed the sigaction for signal 4
01-15 12:24:43.915 9062 9062 I wrap.sh : ==9070==T0: stack [0x7ffceecdf000,0x7ffcef4df000) size 0x800000; local=0x7ffcef4dad24
01-15 12:24:43.916 9062 9062 I wrap.sh : AddressSanitizer:DEADLYSIGNAL
01-15 12:24:43.916 9062 9062 I wrap.sh : =================================================================
01-15 12:24:43.916 9062 9062 I wrap.sh : ==9070==ERROR: AddressSanitizer: SEGV on unknown address 0x630d220684b0 (pc 0x78ee4837d155 bp 0x000082f34aae sp 0x7ffcef4dabd0 T0)
01-15 12:24:43.916 9062 9062 I wrap.sh : ==9070==The signal is caused by a READ memory access.
01-15 12:24:43.918 9062 9062 I wrap.sh : AddressSanitizer:DEADLYSIGNAL
01-15 12:24:43.918 9062 9062 I wrap.sh : AddressSanitizer: nested bug in the same thread, aborting.
01-15 12:24:43.918 9062 9062 I wrap.sh : Launching: /system/bin/app_process64 -XjdwpProvider:adbconnection -XjdwpOptions:suspend=n,server=y -Xcompiler-option --generate-mini-debug-info /system/bin --application --nice-name=org.mozilla.geckoview_example:tab0 com.android.internal.os.WrapperInit 4 29 android.app.ActivityThread seq=81
01-15 12:24:44.022 9062 9062 I wrap.sh : ==9063==AddressSanitizer: failed to intercept '__strndup'
01-15 12:24:44.022 9062 9062 I wrap.sh : '==9063==AddressSanitizer: failed to intercept '__strxfrm_l'
01-15 12:24:44.022 9062 9062 I wrap.sh : '==9063==AddressSanitizer: failed to intercept 'bcmp'
01-15 12:24:44.022 9062 9062 I wrap.sh : '==9063==AddressSanitizer: failed to intercept 'wait3'
01-15 12:24:44.022 9062 9062 I wrap.sh : '==9063==AddressSanitizer: failed to intercept '__wait4'
01-15 12:24:44.022 9062 9062 I wrap.sh : '==9063==AddressSanitizer: failed to intercept 'sigprocmask'
01-15 12:24:44.023 9062 9062 I wrap.sh : '==9063==AddressSanitizer: failed to intercept 'ftime'
01-15 12:24:44.023 9062 9062 I wrap.sh : '==9063==AddressSanitizer: failed to intercept 'pthread_setcancelstate'
01-15 12:24:44.023 9062 9062 I wrap.sh : '==9063==AddressSanitizer: failed to intercept 'pthread_setcanceltype'
01-15 12:24:44.023 9062 9062 I wrap.sh : '==9063==AddressSanitizer: failed to intercept 'getutid'
01-15 12:24:44.023 9062 9062 I wrap.sh : '==9063==AddressSanitizer: failed to intercept 'getutline'
01-15 12:24:44.023 9062 9062 I wrap.sh : '==9063==AddressSanitizer: failed to intercept '__wcsxfrm_l'
01-15 12:24:44.023 9062 9062 I wrap.sh : '==9063==AddressSanitizer: failed to intercept 'bsd_signal'
01-15 12:24:44.023 9062 9062 I wrap.sh : '==9063==AddressSanitizer: failed to intercept 'signal'
01-15 12:24:44.023 9062 9062 I wrap.sh : '==9063==AddressSanitizer: failed to intercept 'sigaction'
01-15 12:24:44.023 9062 9062 I wrap.sh : '==9063==AddressSanitizer: failed to intercept 'index'
01-15 12:24:44.023 9062 9062 I wrap.sh : '==9063==AddressSanitizer: libc interceptors initialized
01-15 12:24:44.026 9062 9062 I wrap.sh : || `[0x10007fff8000, 0x7fffffffffff]` || HighMem ||
01-15 12:24:44.026 9062 9062 I wrap.sh : || `[0x02008fff7000, 0x10007fff7fff]` || HighShadow ||
01-15 12:24:44.026 9062 9062 I wrap.sh : || `[0x00008fff7000, 0x02008fff6fff]` || ShadowGap ||
01-15 12:24:44.026 9062 9062 I wrap.sh : || `[0x00007fff8000, 0x00008fff6fff]` || LowShadow ||
01-15 12:24:44.026 9062 9062 I wrap.sh : || `[0x000000000000, 0x00007fff7fff]` || LowMem ||
01-15 12:24:44.026 9062 9062 I wrap.sh : MemToShadow(shadow): 0x00008fff7000 0x000091ff6dff 0x004091ff6e00 0x02008fff6fff
01-15 12:24:44.026 9062 9062 I wrap.sh : redzone=16
01-15 12:24:44.026 9062 9062 I wrap.sh : max_redzone=2048
01-15 12:24:44.026 9062 9062 I wrap.sh : quarantine_size_mb=16M
01-15 12:24:44.026 9062 9062 I wrap.sh : thread_local_quarantine_size_kb=64K
01-15 12:24:44.026 9062 9062 I wrap.sh : malloc_context_size=30
01-15 12:24:44.026 9062 9062 I wrap.sh : SHADOW_SCALE: 3
01-15 12:24:44.026 9062 9062 I wrap.sh : SHADOW_GRANULARITY: 8
01-15 12:24:44.026 9062 9062 I wrap.sh : SHADOW_OFFSET: 0x7fff8000
01-15 12:24:44.026 9062 9062 I wrap.sh : ==9063==Installed the sigaction for signal 11
01-15 12:24:44.026 9062 9062 I wrap.sh : ==9063==Installed the sigaction for signal 7
01-15 12:24:44.026 9062 9062 I wrap.sh : ==9063==Installed the sigaction for signal 8
01-15 12:24:44.026 9062 9062 I wrap.sh : ==9063==Installed the sigaction for signal 4
01-15 12:24:44.029 9062 9062 I wrap.sh : ==9063==T0: stack [0x7ffe0ac46000,0x7ffe0b446000) size 0x800000; local=0x7ffe0b442b64
01-15 12:24:44.029 9062 9062 I wrap.sh : ==9063==AddressSanitizer Init done
01-15 12:24:44.079 9062 9062 I wrap.sh : ==9063==T1: stack [0x74a8abcde000,0x74a8abdde4f0) size 0x1004f0; local=0x74a8abdde474
01-15 12:24:44.080 9062 9062 I wrap.sh : ==9063==T2: stack [0x74a8abbe0000,0x74a8abcdd4f0) size 0xfd4f0; local=0x74a8abcdd474
01-15 12:24:44.080 9062 9062 I wrap.sh : ==9063==T3: stack [0x74a8abae2000,0x74a8abbdf4f0) size 0xfd4f0; local=0x74a8abbdf474
01-15 12:24:44.081 9062 9062 I wrap.sh : ==9063==T4: stack [0x74a8ab9dc000,0x74a8abae14f0) size 0x1054f0; local=0x74a8abae1474
01-15 12:24:44.081 9062 9062 I wrap.sh : ==9063==T5: stack [0x74a8ab8d6000,0x74a8ab9db4f0) size 0x1054f0; local=0x74a8ab9db474
01-15 12:24:44.082 9062 9062 I wrap.sh : ==9063==T6: stack [0x74a8ab7d0000,0x74a8ab8d54f0) size 0x1054f0; local=0x74a8ab8d5474
01-15 12:24:44.082 9062 9062 I wrap.sh : ==9063==T7: stack [0x74a8ab6ca000,0x74a8ab7cf4f0) size 0x1054f0; local=0x74a8ab7cf474
01-15 12:24:44.113 9062 9062 I wrap.sh : ==9063==T8: stack [0x74a8ab4ce000,0x74a8ab5cb4f0) size 0xfd4f0; local=0x74a8ab5cb474
01-15 12:24:44.116 9062 9062 I wrap.sh : ==9063==T9: stack [0x74a8ab3d0000,0x74a8ab4cd4f0) size 0xfd4f0; local=0x74a8ab4cd474
01-15 12:24:44.345 9062 9062 I wrap.sh : ==9063==T10: stack [0x74a8a6270000,0x74a8a636d4f0) size 0xfd4f0; local=0x74a8a636d474
01-15 12:24:44.582 9062 9062 I wrap.sh : ==9063==T11: stack [0x74a89a537000,0x74a89a6344f0) size 0xfd4f0; local=0x74a89a634474
01-15 12:24:44.594 9062 9062 I wrap.sh : ==9063==T12: stack [0x74a899bac000,0x74a89a4b14f0) size 0x9054f0; local=0x74a89a4b1474
01-15 12:24:44.715 9062 9062 I wrap.sh : ==9063==T13: stack [0x74a899705000,0x74a8998024f0) size 0xfd4f0; local=0x74a899802474
01-15 12:24:44.715 9062 9062 I wrap.sh : ==9063==T13 TSDDtor
01-15 12:24:44.715 9062 9062 I wrap.sh : ==9063==T13 exited
01-15 12:24:44.781 9062 9062 I wrap.sh : ==9063==T14: stack [0x74a898eb1000,0x74a898fae4f0) size 0xfd4f0; local=0x74a898fae474
01-15 12:24:44.788 9062 9062 I wrap.sh : ==9063==T15: stack [0x74a898db3000,0x74a898eb04f0) size 0xfd4f0; local=0x74a898eb0474
01-15 12:24:44.794 9062 9062 I wrap.sh : ==9063==T16: stack [0x74a8ae079000,0x74a8ae0824f0) size 0x94f0; local=0x74a8ae082474
01-15 12:24:44.794 9062 9062 I wrap.sh : ==9063==T17: stack [0x74a898bb3000,0x74a898db24f0) size 0x1ff4f0; local=0x74a898db2474
01-15 12:24:44.794 9062 9062 I wrap.sh : ==9063==T18: stack [0x74a8989b3000,0x74a898bb24f0) size 0x1ff4f0; local=0x74a898bb2474
01-15 12:24:44.796 9062 9062 I wrap.sh : ==9063==T19: stack [0x74a8987b3000,0x74a8989b24f0) size 0x1ff4f0; local=0x74a8989b2474
01-15 12:24:44.796 9062 9062 I wrap.sh : ==9063==T20: stack [0x74a8985b3000,0x74a8987b24f0) size 0x1ff4f0; local=0x74a8987b2474
01-15 12:24:44.893 9062 9062 I wrap.sh : ==9063==T21: stack [0x74a85faa7000,0x74a85fba44f0) size 0xfd4f0; local=0x74a85fba4474
01-15 12:24:44.894 9062 9062 I wrap.sh : ==9063==T22: stack [0x74a85f9a9000,0x74a85faa64f0) size 0xfd4f0; local=0x74a85faa6474
01-15 12:24:44.895 9062 9062 I wrap.sh : ==9063==T23: stack [0x74a85f8ab000,0x74a85f9a84f0) size 0xfd4f0; local=0x74a85f9a8474
01-15 12:24:44.897 9062 9062 I wrap.sh : ==9063==T24: stack [0x74a85f7ad000,0x74a85f8aa4f0) size 0xfd4f0; local=0x74a85f8aa474
01-15 12:24:44.899 9062 9062 I wrap.sh : ==9063==T25: stack [0x74a85f6af000,0x74a85f7ac4f0) size 0xfd4f0; local=0x74a85f7ac474
01-15 12:24:44.912 9062 9062 I wrap.sh : ==9063==T26: stack [0x74a85f542000,0x74a85f63f4f0) size 0xfd4f0; local=0x74a85f63f474
01-15 12:24:44.914 9062 9062 I wrap.sh : ==9063==T27: stack [0x74a85f444000,0x74a85f5414f0) size 0xfd4f0; local=0x74a85f541474
01-15 12:24:44.915 9062 9062 I wrap.sh : ==9063==T28: stack [0x74a85f346000,0x74a85f4434f0) size 0xfd4f0; local=0x74a85f443474
|
|
||
Comment 39•3 years ago
•
|
||
Interestingly this is a corresponding "good" run (just the Installed the sigaction... part):
01-15 12:24:44.026 9062 9062 I wrap.sh : ==9063==Installed the sigaction for signal 11
01-15 12:24:44.026 9062 9062 I wrap.sh : ==9063==Installed the sigaction for signal 7
01-15 12:24:44.026 9062 9062 I wrap.sh : ==9063==Installed the sigaction for signal 8
01-15 12:24:44.026 9062 9062 I wrap.sh : ==9063==Installed the sigaction for signal 4
01-15 12:24:44.029 9062 9062 I wrap.sh : ==9063==T0: stack [0x7ffe0ac46000,0x7ffe0b446000) size 0x800000; local=0x7ffe0b442b64
01-15 12:24:44.029 9062 9062 I wrap.sh : ==9063==AddressSanitizer Init done
|
|
||
Comment 40•3 years ago
|
||
OMG I finally got something.
From reading some code (also hinted in Comment 37) this is a race condition (or maybe a re-entrancy problem) with handling the a SIGSEGV signal. so I set handle_segv=0 in wrap.sh and boom:
01-15 13:45:02.881 12387 12387 F DEBUG : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
01-15 13:45:02.881 12387 12387 F DEBUG : Build fingerprint: 'Android/sdk_phone_x86_64/generic_x86_64:9/PSR1.180720.012/4923214:userdebug/test-keys'
01-15 13:45:02.881 12387 12387 F DEBUG : Revision: '0'
01-15 13:45:02.881 12387 12387 F DEBUG : ABI: 'x86_64'
01-15 13:45:02.881 12387 12387 F DEBUG : pid: 12362, tid: 12362, name: app_process64 >>> /system/bin/app_process64 <<<
01-15 13:45:02.881 12387 12387 F DEBUG : signal 11 (SIGSEGV), code 2 (SEGV_ACCERR), fault addr 0x62d08ea1e018
01-15 13:45:02.881 12387 12387 F DEBUG : rax 0000000000000000 rbx 000077a97ef5e240 rcx 000062d08ea1e018 rdx 0000000082f34aae
01-15 13:45:02.881 12387 12387 F DEBUG : r8 000077a97ef5e470 r9 0000000000000000 r10 000077a97efac810 r11 0000000000000000
01-15 13:45:02.881 12387 12387 F DEBUG : r12 0000000000000000 r13 00000000020bcd2a r14 000077a97ef5e240 r15 00007ffd93105064
01-15 13:45:02.881 12387 12387 F DEBUG : rdi 000077a97ef5e240 rsi 00007ffd931050a0
01-15 13:45:02.881 12387 12387 F DEBUG : rbp 0000000082f34aae rsp 00007ffd93105000 rip 000077a97efe8155
01-15 13:45:02.882 12387 12387 F DEBUG :
01-15 13:45:02.882 12387 12387 F DEBUG : backtrace:
01-15 13:45:02.882 12387 12387 F DEBUG : #00 pc 000000000002d155 /system/bin/linker64 (__dl__ZNK6soinfo10gnu_lookupER10SymbolNamePK12version_infoPj+133)
01-15 13:45:02.882 12387 12387 F DEBUG : #01 pc 000000000002d0a1 /system/bin/linker64 (__dl__ZNK6soinfo19find_symbol_by_nameER10SymbolNamePK12version_infoPPK9elf64_sym+49)
01-15 13:45:02.882 12387 12387 F DEBUG : #02 pc 0000000000018074 /system/bin/linker64 (__dl__ZL19dlsym_linear_lookupP19android_namespace_tPKcPK12version_infoPP6soinfoS7_Pv+196)
01-15 13:45:02.882 12387 12387 F DEBUG : #03 pc 0000000000017c6a /system/bin/linker64 (__dl__Z8do_dlsymPvPKcS1_PKvPS_+362)
01-15 13:45:02.882 12387 12387 F DEBUG : #04 pc 000000000001307f /system/bin/linker64 (__dl__Z10dlsym_implPvPKcS1_PKv+63)
01-15 13:45:02.882 12387 12387 F DEBUG : #05 pc 0000000000063f2b /data/app/org.mozilla.geckoview_example-Amx9-FuKOhU91CONTfsuTg==/lib/x86_64/libclang_rt.asan-x86_64-android.so (offset 0x4c000) (__sanitizer::LLVMSymbolizer::SymbolizeFrame(unsigned long, __sanitizer::FrameInfo*)+715)
Is this anything you can use :truber? looks like the linker... is not... happy? I can dig further if this doesn't mean anything to you.
| Comment hidden (duplicate) |
|
|
||
Comment 42•3 years ago
|
||
The full logcat might be interesting too (this is with debug=1,verbosity=2)
|
|
||
Comment 43•3 years ago
|
||
|
|
||
Comment 44•3 years ago
•
|
||
It looks to me like a crash trying to symbolize a frame, but if we're symbolizing a frame, we've already crashed, so what's the original crash?
|
|
||
Comment 45•3 years ago
|
||
Looks like an OOM:
art_sigsegv_fault 0x000077bc8e2487b0
art::FaultManager::HandleFault(int, siginfo*, void*) 0x000077bc8e248c95
___lldb_unnamed_symbol22$$app_process64 0x00005a6ba7e9fbb6
___lldb_unnamed_symbol1$$libc.so 0x000077bd12fa79e0
NS_ABORT_OOM(unsigned long) nsDebugImpl.cpp:618
XPCJSContext::Initialize() XPCJSContext.cpp:1378
XPCJSContext::NewXPCJSContext() XPCJSContext.cpp:1411
nsXPConnect::InitJSContext() nsXPConnect.cpp:83
XREMain::XRE_mainRun() nsAppRunner.cpp:4968
XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) nsAppRunner.cpp:5440
XRE_main(int, char**, mozilla::BootstrapConfig const&) nsAppRunner.cpp:5503
::GeckoStart(JNIEnv *, char **, int, const mozilla::StaticXREAppData &) nsAndroidStartup.cpp:38
::Java_org_mozilla_gecko_mozglue_GeckoLoader_nativeRun(JNIEnv *, jclass, jobjectArray, int, int, int, int, int) APKOpen.cpp:375
art_quick_generic_jni_trampoline 0x000077bc8e669062
art_quick_invoke_static_stub 0x000077bc8e65ee17
art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*) 0x000077bc8e16a604
art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*) 0x000077bc8e33cb92
bool art::interpreter::DoCall<true, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*) 0x000077bc8e337248
bool art::interpreter::DoInvoke<(art::InvokeType)0, true, false>(art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*) 0x000077bc8e3752f3
void art::interpreter::ExecuteSwitchImplCpp<false, false>(art::interpreter::SwitchImplContext*) 0x000077bc8e35f936
ExecuteSwitchImplAsm 0x000077bc8e66af26
art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool) (.llvm.2620325170) 0x000077bc8e30ce8e
artQuickToInterpreterBridge 0x000077bc8e619548
art_quick_to_interpreter_bridge 0x000077bc8e6691ed
art_quick_invoke_stub 0x000077bc8e65eab5
art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*) 0x000077bc8e16a5f3
art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*) 0x000077bc8e55256a
art::InvokeVirtualOrInterfaceWithJValues(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, jvalue*) 0x000077bc8e55383b
art::Thread::CreateCallback(void*) 0x000077bc8e582af9
__pthread_start(void*) 0x000077bd13013bac
__start_thread 0x000077bd12fabf2e
I tried with 8GB, let's see if a 16GB memory emulator can handle this.
|
|
||
Comment 46•3 years ago
|
||
Compiling Debug to see if we trip in some MOZ_ASSERT before we get here.
|
|
||
Comment 47•3 years ago
|
||
In debug we get this (I think this is the same failure as release too):
art_sigsegv_fault 0x0000740bb34467b0
art::FaultManager::HandleFault(int, siginfo*, void*) 0x0000740bb3446c95
___lldb_unnamed_symbol22$$app_process64 0x000056796c692bb6
___lldb_unnamed_symbol1$$libc.so 0x0000740c38aa79e0
AutoAssertReportedException::~AutoAssertReportedException() BytecodeCompiler.cpp:68
bool CompileGlobalScriptToStencilImpl<mozilla::Utf8Unit>(JSContext*, js::frontend::CompilationStencil&, JS::SourceText<mozilla::Utf8Unit>&, js::ScopeKind) BytecodeCompiler.cpp:255
js::frontend::CompileGlobalScriptToStencil(JSContext*, js::frontend::CompilationStencil&, JS::SourceText<mozilla::Utf8Unit>&, js::ScopeKind) BytecodeCompiler.cpp:268
JSRuntime::initSelfHosting(JSContext*) SelfHosting.cpp:2887
JS::InitSelfHostedCode(JSContext*) jsapi.cpp:506
XPCJSContext::Initialize() XPCJSContext.cpp:1375
XPCJSContext::NewXPCJSContext() XPCJSContext.cpp:1411
nsXPConnect::InitJSContext() nsXPConnect.cpp:83
XREMain::XRE_mainRun() nsAppRunner.cpp:4968
XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) nsAppRunner.cpp:5440
XRE_main(int, char**, mozilla::BootstrapConfig const&) nsAppRunner.cpp:5503
::GeckoStart(JNIEnv *, char **, int, const mozilla::StaticXREAppData &) nsAndroidStartup.cpp:38
::Java_org_mozilla_gecko_mozglue_GeckoLoader_nativeRun(JNIEnv *, jclass, jobjectArray, int, int, int, int, int) APKOpen.cpp:375
art_quick_generic_jni_trampoline 0x0000740bb3867062
art_quick_invoke_static_stub 0x0000740bb385ce17
art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*) 0x0000740bb3368604
art::interpreter::ArtInterpreterToCompiledCodeBridge(art::Thread*, art::ArtMethod*, art::ShadowFrame*, unsigned short, art::JValue*) 0x0000740bb353ab92
bool art::interpreter::DoCall<true, false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*) 0x0000740bb3535248
bool art::interpreter::DoInvoke<(art::InvokeType)0, true, false>(art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, art::JValue*) 0x0000740bb35732f3
void art::interpreter::ExecuteSwitchImplCpp<false, false>(art::interpreter::SwitchImplContext*) 0x0000740bb355d936
ExecuteSwitchImplAsm 0x0000740bb3868f26
art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool) (.llvm.2620325170) 0x0000740bb350ae8e
artQuickToInterpreterBridge 0x0000740bb3817548
art_quick_to_interpreter_bridge 0x0000740bb38671ed
art_quick_invoke_stub 0x0000740bb385cab5
art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*) 0x0000740bb33685f3
art::(anonymous namespace)::InvokeWithArgArray(art::ScopedObjectAccessAlreadyRunnable const&, art::ArtMethod*, art::(anonymous namespace)::ArgArray*, art::JValue*, char const*) 0x0000740bb375056a
art::InvokeVirtualOrInterfaceWithJValues(art::ScopedObjectAccessAlreadyRunnable const&, _jobject*, _jmethodID*, jvalue*) 0x0000740bb375183b
art::Thread::CreateCallback(void*) 0x0000740bb3780af9
__pthread_start(void*) 0x0000740c38b13bac
|
|
||
Comment 48•3 years ago
|
||
Interestingly with this wrap.sh I don't get a crash on a debug build (taken from the android source):
cmd=$1
shift
HERE="$(cd "$(dirname "$0")" && pwd)"
export ASAN_OPTIONS=abort_on_error=1,debug=1,print_stats=1,log_path=stderr,verbosity=2,print_stats=1,handle_segv=0
export LD_PRELOAD=$HERE/libclang_rt.asan-x86_64-android.so
os_version=$(getprop ro.build.version.sdk)
if [ "$os_version" -eq "27" ]; then
cmd="$cmd -Xrunjdwp:transport=dt_android_adb,suspend=n,server=y -Xcompiler-option --debuggable $@"
elif [ "$os_version" -eq "28" ]; then
cmd="$cmd -XjdwpProvider:adbconnection -XjdwpOptions:suspend=n,server=y -Xcompiler-option --debuggable $@"
else
cmd="$cmd -XjdwpProvider:adbconnection $@"
fi
exec $cmd
without handle_segv I do get the crash in Comment 47. I'm wondering if the sevg handlers peck at each other causing a crash.
|
||
Updated•3 years ago
|
|
|
||
Updated•3 years ago
|
|
|
||
Updated•3 years ago
|
|
||
Comment 49•2 years ago
|
||
I got a nag email about this ticket, so I'll update it. There's no way that this can be P1: it's been pushed back a bunch of times and it just can't impact very many people. Bump the priority if it's impacting the internal team or automation significantly. I'm going to move it all the way to P5, since we'll take a patch (gladly!) but are unlikely to work on it without a compelling reason to do so.
|
|
Reporter | |
Comment 50•2 years ago
|
||
As far as I can tell this is the main (only?) blocker for fuzzing ASan builds on Android.
Thanks to the work done in bug 1686514 and bug 1762278 we now have debug-able ASan builds.
|
|
Reporter | |
Comment 51•2 years ago
|
||
Hey Agi, when you have a chance can you please have a look at this again now that it is unblocked?
|
|
||
Comment 52•2 years ago
|
||
I don't personally have time to look at it, but will discuss with the team if we can prioritize this.
|
||
Comment 53•2 years ago
|
||
Mike, is there a chance that you could help with this one ? :)
thanks
|
|
Reporter | |
Comment 54•2 years ago
|
||
A try build that reproduces the issue is available here: https://treeherder.mozilla.org/jobs?repo=try&revision=fe01c5cd091c6c8696e8b0fa4c1718701706e7ec
See the logcat output for the crash: https://firefoxci.taskcluster-artifacts.net/NSl_YNsTTF-wP6ygAVdk_A/0/public/test_info/logcat-emulator-5554.log
|
||
Comment 55•2 years ago
|
||
This bug prevents fuzzing from making progress; however, it has low severity. It is important for fuzz blocker bugs to be addressed in a timely manner (see here why?).
:nalexander, could you increase the severity?
For more information, please visit auto_nag documentation.
|
|
||
Comment 56•2 years ago
|
||
tsmith: I'll let you make a call about the real severity of this ticket.
|
|
Reporter | |
Updated•2 years ago
|
|
|
Reporter | |
Comment 57•2 years ago
|
||
(In reply to Nick Alexander :nalexander [he/him] [Back August 22, 2022] from comment #56)
tsmith: I'll let you make a call about the real severity of this ticket.
It is blocking us from fuzzing ASan builds on Android which is pretty significant.
|
|
Reporter | |
Comment 58•2 years ago
|
||
With bug 1784588 resolved I am now only seeing the idmap2 crash.
08-18 18:07:33.956 3327 3327 I idmap2 : =================================================================
08-18 18:07:33.956 3327 3327 I idmap2 : ==3327==ERROR: AddressSanitizer: SEGV on unknown address 0x62c830a89468 (pc 0x7b18d0dfc597 bp 0x0000a19746da sp 0x7ffc3ae9ff90 T0)
08-18 18:07:33.956 3327 3327 I idmap2 : ==3327==The signal is caused by a READ memory access.
08-18 18:07:33.959 3289 3289 E org.mozilla.geckoview_example:gpu: idmap2: AddressSanitizer:DEADLYSIGNAL
08-18 18:07:33.959 3289 3289 E org.mozilla.geckoview_example:gpu: AddressSanitizer:DEADLYSIGNAL
08-18 18:07:33.959 3289 3289 E org.mozilla.geckoview_example:gpu: AddressSanitizer: nested bug in the same thread, aborting.
08-18 18:07:33.959 3289 3289 W OverlayConfig: 'idmap2 create-multiple' failed: no mutable="false" overlays targeting "android" will be loaded
The crash seems to be gpu specific, :bhood is there someone on the gfx team that might have an idea what's crashing?
|
||
Comment 59•2 years ago
|
||
Jamie, another one for your entertainment. ;)
|
||
Comment 60•2 years ago
|
||
That really isn't anything to go on, other than that the crash occurs in the GPU process. Is it possible to get a backtrace?
|
|
Reporter | |
Comment 61•2 years ago
|
||
ASan is not providing a stack trace for these crashes. I do get a full ASan stack when opening about:crashcontent when I get lucky and the browser launch without crashing on start up (so I know it works).
Using ASAN_OPTIONS=handle_segv=0 I was able to get tombstones from the crash but it looks similar to comment 43.
I don't know how to symbolize these, the build I used can be found here: https://firefox-ci-tc.services.mozilla.com/tasks/index/gecko.v2.mozilla-central.pushdate.2022.08.18.20220818232425.mobile/android-x86_64-fuzzing-asan
|
|
Reporter | |
Comment 62•2 years ago
|
||
|
|
Reporter | |
Comment 63•2 years ago
|
||
Setting the priority back to P2. I seems it was set lower because it was ignored. Fuzzing without ASan can miss serious security issues.
|
|
||
Comment 64•2 years ago
|
||
I don't know how to symbolicate tombstones either. If you disable the GPU process (by setting layers.gpu-process.enabled to false) does it crash in the parent process instead? And does that give symbols?
|
|
Reporter | |
Comment 65•2 years ago
|
||
(In reply to Jamie Nicol [:jnicol] from comment #64)
I don't know how to symbolicate tombstones either. If you disable the GPU process (by setting
layers.gpu-process.enabledto false) does it crash in the parent process instead? And does that give symbols?
No luck.
owlish offered to bring this up in the next triage meeting. Hopefully someone from the GV team can help move this forward.
|
||
Comment 66•2 years ago
•
|
||
(In reply to Jamie Nicol [:jnicol] from comment #64)
I don't know how to symbolicate tombstones either. If you disable the GPU process (by setting
layers.gpu-process.enabledto false) does it crash in the parent process instead? And does that give symbols?
Gabriele, do you have any suggestions for symbolicating Android crash tombstones? IIUC, this bug is an ASan startup crash in our Android GPU process.
Note that this bug was filed two years ago. The ASan startup crash we're seeing now is surely different from the original crash two years ago, so we should probably ignore the bug comments and attachments from two years ago. :)
|
|
||
Comment 67•2 years ago
•
|
||
I figure I should at least add a comment about how far I got about this. The crashes I've looked at from derivatives of the try in comment 54 don't involve Gecko code at all. They all come from LD_PRELOAD'ing libclang_rt, and whatever it's doing ends up breaking things in a weird way, sometimes with a stack trace that comes from the dynamic loader. I haven't figured what's going wrong yet because I've been busy with other things, but I've also figured that the clang runtime for android is outdated and probably nobody uses ASan on recent android. For instance, the malloc hooks the runtime uses have been removed from Android several releases ago and nobody from Google has bothered to fix that situation. That's not supposed to cause problems, at least not the ones we're seeing, but it's not going to help even if we figure out what's going on and fix it.
|
||
Comment 68•2 years ago
|
||
IIRC Android tombstones print symbols if those are present in the binaries and indeed the ones attached here do have them (even though some appear mangled). We don't have a tool specifically for symbolicating tombstones but I think it might be possible to re-use fix-stacks for that purpose. Its input is a text file where it will look for lines that need symbolication, given a set of symbol files if it find matches it will replace them. To use it on tombstones one would have to provide an alternative method of matchine lines (the current one is here).
|
|
||
Updated•2 years ago
|
|
|
||
Comment 69•2 years ago
|
||
Here's the demangled backtrace from tombstone_00:
backtrace:
#00 pc 000000000005e597 /apex/com.android.runtime/bin/linker64 (soinfo::gnu_lookup(SymbolName&, version_info const*) const+279) (BuildId: 6ca433bbb4f29be954d1fe2c5fc84a6d)
#01 pc 0000000000045cad /apex/com.android.runtime/bin/linker64 (dlsym_linear_lookup(android_namespace_t*, char const*, version_info const*, soinfo**, soinfo*, void*)+141) (BuildId: 6ca433bbb4f29be954d1fe2c5fc84a6d)
#02 pc 0000000000045763 /apex/com.android.runtime/bin/linker64 (do_dlsym(void*, char const*, char const*, void const*, void**)+675) (BuildId: 6ca433bbb4f29be954d1fe2c5fc84a6d)
#03 pc 0000000000040332 /apex/com.android.runtime/bin/linker64 (dlsym_impl(void*, char const*, char const*, void const*)+82) (BuildId: 6ca433bbb4f29be954d1fe2c5fc84a6d)
#04 pc 000000000006da43 /data/app/~~aVV_zot7iyBngCA1_kpc7Q==/org.mozilla.geckoview_example-1Jk-Y_aDCBSlYfpnPupEuw==/lib/x86_64/libclang_rt.asan-x86_64-android.so (__sanitizer::Symbolizer::LateInitialize()+19)
#05 pc 00000000000d65c6 /data/app/~~aVV_zot7iyBngCA1_kpc7Q==/org.mozilla.geckoview_example-1Jk-Y_aDCBSlYfpnPupEuw==/lib/x86_64/libclang_rt.asan-x86_64-android.so (__asan::AsanInitInternal()+422)
#06 pc 000000000008a1af /data/app/~~aVV_zot7iyBngCA1_kpc7Q==/org.mozilla.geckoview_example-1Jk-Y_aDCBSlYfpnPupEuw==/lib/x86_64/libclang_rt.asan-x86_64-android.so (strcmp+927)
#07 pc 0000000000050dc2 /apex/com.android.runtime/lib64/bionic/libc.so (__libc_init_vdso(libc_globals*)+578) (BuildId: 3707c39fc397eeaa328142d90b50a973)
#08 pc 00000000000673c5 /apex/com.android.runtime/lib64/bionic/libc.so (__libc_init_globals()+85) (BuildId: 3707c39fc397eeaa328142d90b50a973)
#09 pc 00000000000506d6 /apex/com.android.runtime/lib64/bionic/libc.so (__libc_preinit_impl()+38) (BuildId: 3707c39fc397eeaa328142d90b50a973)
#10 pc 000000000005eec7 /apex/com.android.runtime/bin/linker64 (void call_array<void (*)(int, char**, char**)>(char const*, void (**)(int, char**, char**), unsigned long, bool, char const*)+263) (BuildId: 6ca433bbb4f29be954d1fe2c5fc84a6d)
#11 pc 000000000005f0e1 /apex/com.android.runtime/bin/linker64 (soinfo::call_constructors()+417) (BuildId: 6ca433bbb4f29be954d1fe2c5fc84a6d)
#12 pc 000000000005efc8 /apex/com.android.runtime/bin/linker64 (soinfo::call_constructors()+136) (BuildId: 6ca433bbb4f29be954d1fe2c5fc84a6d)
#13 pc 000000000005efc8 /apex/com.android.runtime/bin/linker64 (soinfo::call_constructors()+136) (BuildId: 6ca433bbb4f29be954d1fe2c5fc84a6d)
#14 pc 00000000000a10e5 /apex/com.android.runtime/bin/linker64 (__linker_init_post_relocation(KernelArgumentBlock&, soinfo&)+4949) (BuildId: 6ca433bbb4f29be954d1fe2c5fc84a6d)
#15 pc 000000000009fd64 /apex/com.android.runtime/bin/linker64 (__linker_init+644) (BuildId: 6ca433bbb4f29be954d1fe2c5fc84a6d)
#16 pc 0000000000061fb7 /apex/com.android.runtime/bin/linker64 (_start+7) (BuildId: 6ca433bbb4f29be954d1fe2c5fc84a6d)
Tyson and I looked for debug symbols for Android system images, and couldn't find anything. It looks like it's expected to build a debug image for platform development.
|
|
||
Comment 70•2 years ago
|
||
(In reply to Mike Hommey [:glandium] from comment #67)
For instance, the malloc hooks the runtime uses have been removed from Android several releases ago and nobody from Google has bothered to fix that situation.
Should we log an llvm issue for this? Thanks for looking, this is the most insight we've gotten so far.
|
|
||
Comment 71•11 months ago
|
||
Should we log an llvm issue for this?
probably. can one of you two do it?
|
|
||
Comment 72•11 months ago
|
||
|
||
Updated•8 months ago
|
Description
•