xpcshell crashes on Android 7 (ARM device and ARM and x86 emulators)
Categories
(Core :: JavaScript Engine, defect, P3)
Tracking
()
People
(Reporter: gbrown, Assigned: snorp)
References
(Blocks 1 open bug)
Details
(Keywords: crash, Whiteboard: [geckoview:p1])
Attachments
(2 files)
:bc reported seeing xpcshell crashing on an Android 7 arm device, so I tried on the Android 7 arm emulator and got the same result. $ ./mach android-emulator --version 7.0 --force-update (This is horribly slow to start, and not as well used as the 4.3 version, but I can run mochitests, etc on it -- I think it is a sane test environment.) $ ./mach xpcshell-test netwerk/test/unit/test_simple.js --log-mach-level=debug 1:20.92 adb DEBUG shell_output: adb -s emulator-5554 wait-for-device shell mkdir /sdcard/tests/xpc/minidumps; echo rc=$?, timeout: None, root: False, timedout: None, exitcode: 0, output: 1:21.23 adb DEBUG command_output: adb -s emulator-5554 wait-for-device push /tmp/tmp2dYGap /data/local/xpcb/xpcw, timeout: None, timedout: None, exitcode: 0, output: [100%] /data/local/xpcb/xpcw /tmp/tmp2dYGap: 1 file pushed. 0.1 MB/s (697 bytes in 0.005s) 1:21.33 adb DEBUG command_output: adb -s emulator-5554 wait-for-device push /tmp/tmpTsU40b /data/local/xpcb/cleardir, timeout: None, timedout: None, exitcode: 0, output: [100%] /data/local/xpcb/cleardir /tmp/tmpTsU40b: 1 file pushed. 0.0 MB/s (39 bytes in 0.004s) 1:21.33 adb DEBUG chmod: path=/data/local/xpcb, recursive=True, mask=777, root=False 1:21.53 adb DEBUG shell_output: adb -s emulator-5554 wait-for-device shell chmod -R 777 /data/local/xpcb; echo rc=$?, timeout: None, root: False, timedout: None, exitcode: 0, output: 1:21.53 WARNING MOZ_NODE_PATH environment variable not set. Tests requiring http/2 will fail. 1:21.68 INFO Running tests sequentially. 1:21.68 SUITE_START: xpcshell - running 1 tests 1:22.08 adb DEBUG shell_output: adb -s emulator-5554 wait-for-device shell /data/local/xpcb/cleardir /sdcard/tests/xpc/p; echo rc=$?, timeout: None, root: False, timedout: None, exitcode: 0, output: rm: /sdcard/tests/xpc/p: No such file or directory 1:22.08 INFO profile dir is /sdcard/tests/xpc/p 1:22.48 adb DEBUG shell_output: adb -s emulator-5554 wait-for-device shell /data/local/xpcb/cleardir /sdcard/tests/xpc/tmp; echo rc=$?, timeout: None, root: False, timedout: None, exitcode: 0, output: 1:22.59 adb DEBUG command_output: adb -s emulator-5554 wait-for-device push /tmp/tmpK8dY_2 /sdcard/tests/xpc/p/mozinfo.json, timeout: None, timedout: None, exitcode: 0, output: [100%] /sdcard/tests/xpc/p/mozinfo.json /tmp/tmpK8dY_2: 1 file pushed. 0.1 MB/s (901 bytes in 0.010s) 1:22.59 TEST_START: netwerk/test/unit/test_simple.js 1:28.80 adb DEBUG get_process_list: [[1, '/init', 'root'], [2, 'kthreadd', 'root'], [3, 'ksoftirqd/0', 'root'], [5, 'kworker/0:0H', 'root'], [6, 'kworker/u2:0', 'root'], [7, 'migration/0', 'root'], [8, 'rcu_preempt', 'root'], [9, 'rcu_bh', 'root'], [10, 'rcu_sched', 'root'], [11, 'khelper', 'root'], [12, 'kworker/u2:1', 'root'], [203, 'writeback', 'root'], [205, 'bioset', 'root'], [207, 'kblockd', 'root'], [301, 'kworker/0:1', 'root'], [302, 'rpciod', 'root'], [312, 'kswapd0', 'root'], [367, 'fsnotify_mark', 'root'], [383, 'nfsiod', 'root'], [395, 'crypto', 'root'], [522, 'kpsmoused', 'root'], [527, 'kworker/0:2', 'root'], [541, 'binder', 'root'], [559, 'deferwq', 'root'], [560, '/sbin/ueventd', 'root'], [803, 'kworker/0:1H', 'root'], [804, 'jbd2/vda-8', 'root'], [806, 'ext4-rsv-conver', 'root'], [808, 'jbd2/vdb-8', 'root'], [809, 'ext4-rsv-conver', 'root'], [813, 'jbd2/vdc-8', 'root'], [814, 'ext4-rsv-conver', 'root'], [815, '/system/bin/logd', 'logd'], [816, '/system/bin/debuggerd', 'root'], [817, '/system/bin/debuggerd64', 'root'], [818, '/system/bin/vold', 'root'], [823, 'kauditd', 'root'], [824, 'debuggerd64:signaller', 'root'], [826, 'debuggerd:signaller', 'root'], [848, '/sbin/healthd', 'root'], [850, '/system/bin/lmkd', 'root'], [851, '/system/bin/servicemanager', 'system'], [852, '/system/bin/surfaceflinger', 'system'], [856, 'zygote64', 'root'], [857, 'zygote', 'root'], [858, '/system/bin/audioserver', 'audioserver'], [859, '/system/bin/cameraserver', 'cameraserver'], [860, '/system/bin/drmserver', 'drm'], [862, '/system/bin/installd', 'root'], [863, '/system/bin/keystore', 'keystore'], [864, 'media.codec', 'mediacodec'], [867, '/system/bin/mediadrmserver', 'media'], [868, 'media.extractor', 'mediaex'], [869, '/system/bin/mediaserver', 'media'], [870, '/system/bin/netd', 'root'], [872, '/system/bin/rild', 'radio'], [874, '/system/bin/fingerprintd', 'system'], [875, '/system/bin/gatekeeperd', 'system'], [876, '/system/xbin/perfprofd', 'root'], [1110, 'system_server', 'system'], [1238, 'com.android.inputmethod.latin', 'u0_a39'], [1284, '/system/bin/sdcard', 'media_rw'], [1349, '/system/bin/sdcard', 'media_rw'], [1417, 'com.android.phone', 'radio'], [1431, 'com.android.settings', 'system'], [1618, 'com.android.systemui', 'u0_a19'], [1692, 'android.ext.services', 'u0_a10'], [1744, 'android.process.media', 'u0_a8'], [1761, 'com.android.keychain', 'system'], [1776, 'com.android.launcher3', 'u0_a12'], [1790, 'android.process.acore', 'u0_a1'], [1818, 'com.android.printspooler', 'u0_a48'], [1856, 'com.android.quicksearchbox', 'u0_a50'], [1879, 'com.android.dialer', 'u0_a6'], [1899, 'com.android.managedprovisioning', 'u0_a13'], [1913, 'com.android.messaging', 'u0_a51'], [1931, 'com.android.providers.calendar', 'u0_a2'], [1989, '/sbin/adbd', 'root'], [2122, '/system/bin/sh', 'root'], [2124, 'ps', 'root']] 1:28.80 TEST_END: FAIL, expected PASS - xpcshell return code: 139 1:28.80 INFO >>>>>>> 1:28.80 netwerk/test/unit/test_simple.js Full command: ['/data/local/xpcb/xpcw', u'/sdcard/tests/xpc/netwerk/test/unit', '-r', '/sdcard/tests/xpc/c/httpd.manifest', '--greomni', u'/data/local/xpcb/fennec-61.0a1.en-US.android-arm.apk', '-m', '-s', '-e', 'const _HEAD_JS_PATH = "/sdcard/tests/xpc/head.js";', '-e', 'const _MOZINFO_JS_PATH = "/sdcard/tests/xpc/p/mozinfo.json";', '-e', 'const _TESTING_MODULES_DIR = "/sdcard/tests/xpc/m";', '-f', '/sdcard/tests/xpc/head.js', '-e', 'const _SERVER_ADDR = "localhost"', '-e', u'const _HEAD_FILES = ["/sdcard/tests/xpc/netwerk/test/unit/head_channels.js", "/sdcard/tests/xpc/netwerk/test/unit/head_cache.js", "/sdcard/tests/xpc/netwerk/test/unit/head_cache2.js"];', '-e', 'const _JSDEBUGGER_PORT = 0;', '-e', u'const _TEST_FILE = ["test_simple.js"];', '-e', u'const _TEST_NAME = "netwerk/test/unit/test_simple.js"', '-e', '_execute_test(); quit(0);'] netwerk/test/unit/test_simple.js xpcw: cd /sdcard/tests/xpc/netwerk/test/unit 1:28.80 netwerk/test/unit/test_simple.js xpcw: xpcshell -r /sdcard/tests/xpc/c/httpd.manifest --greomni /data/local/xpcb/fennec-61.0a1.en-US.android-arm.apk -m -s -e const _HEAD_JS_PATH = "/sdcard/tests/xpc/head.js"; -e const _MOZINFO_JS_PATH = "/sdcard/tests/xpc/p/mozinfo.json"; -e const _TESTING_MODULES_DIR = "/sdcard/tests/xpc/m"; -f /sdcard/tests/xpc/head.js -e const _SERVER_ADDR = "localhost" -e const _HEAD_FILES = ["/sdcard/tests/xpc/netwerk/test/unit/head_channels.js", "/sdcard/tests/xpc/netwerk/test/unit/head_cache.js", "/sdcard/tests/xpc/netwerk/test/unit/head_cache2.js"]; -e const _JSDEBUGGER_PORT = 0; -e const _TEST_FILE = ["test_simple.js"]; -e const _TEST_NAME = "netwerk/test/unit/test_simple.js" -e _execute_test(); quit(0); 1:28.80 netwerk/test/unit/test_simple.js Segmentation fault 1:28.80 INFO <<<<<<< 1:29.41 adb DEBUG command_output: adb -s emulator-5554 wait-for-device pull /sdcard/tests/xpc/minidumps /tmp/tmp8LQPpQ/minidumps, timeout: None, timedout: None, exitcode: 0, output: pull: building file list... /sdcard/tests/xpc/minidumps/: 0 files pulled. 1:29.82 adb DEBUG shell_output: adb -s emulator-5554 wait-for-device shell /data/local/xpcb/cleardir /sdcard/tests/xpc/minidumps; echo rc=$?, timeout: None, root: False, timedout: None, exitcode: 0, output: 1:30.12 adb DEBUG get_process_list: [[1, '/init', 'root'], [2, 'kthreadd', 'root'], [3, 'ksoftirqd/0', 'root'], [5, 'kworker/0:0H', 'root'], [6, 'kworker/u2:0', 'root'], [7, 'migration/0', 'root'], [8, 'rcu_preempt', 'root'], [9, 'rcu_bh', 'root'], [10, 'rcu_sched', 'root'], [11, 'khelper', 'root'], [12, 'kworker/u2:1', 'root'], [203, 'writeback', 'root'], [205, 'bioset', 'root'], [207, 'kblockd', 'root'], [301, 'kworker/0:1', 'root'], [302, 'rpciod', 'root'], [312, 'kswapd0', 'root'], [367, 'fsnotify_mark', 'root'], [383, 'nfsiod', 'root'], [395, 'crypto', 'root'], [522, 'kpsmoused', 'root'], [527, 'kworker/0:2', 'root'], [541, 'binder', 'root'], [559, 'deferwq', 'root'], [560, '/sbin/ueventd', 'root'], [803, 'kworker/0:1H', 'root'], [804, 'jbd2/vda-8', 'root'], [806, 'ext4-rsv-conver', 'root'], [808, 'jbd2/vdb-8', 'root'], [809, 'ext4-rsv-conver', 'root'], [813, 'jbd2/vdc-8', 'root'], [814, 'ext4-rsv-conver', 'root'], [815, '/system/bin/logd', 'logd'], [816, '/system/bin/debuggerd', 'root'], [817, '/system/bin/debuggerd64', 'root'], [818, '/system/bin/vold', 'root'], [823, 'kauditd', 'root'], [824, 'debuggerd64:signaller', 'root'], [826, 'debuggerd:signaller', 'root'], [848, '/sbin/healthd', 'root'], [850, '/system/bin/lmkd', 'root'], [851, '/system/bin/servicemanager', 'system'], [852, '/system/bin/surfaceflinger', 'system'], [856, 'zygote64', 'root'], [857, 'zygote', 'root'], [858, '/system/bin/audioserver', 'audioserver'], [859, '/system/bin/cameraserver', 'cameraserver'], [860, '/system/bin/drmserver', 'drm'], [862, '/system/bin/installd', 'root'], [863, '/system/bin/keystore', 'keystore'], [864, 'media.codec', 'mediacodec'], [867, '/system/bin/mediadrmserver', 'media'], [868, 'media.extractor', 'mediaex'], [869, '/system/bin/mediaserver', 'media'], [870, '/system/bin/netd', 'root'], [872, '/system/bin/rild', 'radio'], [874, '/system/bin/fingerprintd', 'system'], [875, '/system/bin/gatekeeperd', 'system'], [876, '/system/xbin/perfprofd', 'root'], [1110, 'system_server', 'system'], [1238, 'com.android.inputmethod.latin', 'u0_a39'], [1284, '/system/bin/sdcard', 'media_rw'], [1349, '/system/bin/sdcard', 'media_rw'], [1417, 'com.android.phone', 'radio'], [1431, 'com.android.settings', 'system'], [1618, 'com.android.systemui', 'u0_a19'], [1692, 'android.ext.services', 'u0_a10'], [1744, 'android.process.media', 'u0_a8'], [1761, 'com.android.keychain', 'system'], [1776, 'com.android.launcher3', 'u0_a12'], [1790, 'android.process.acore', 'u0_a1'], [1818, 'com.android.printspooler', 'u0_a48'], [1856, 'com.android.quicksearchbox', 'u0_a50'], [1879, 'com.android.dialer', 'u0_a6'], [1899, 'com.android.managedprovisioning', 'u0_a13'], [1913, 'com.android.messaging', 'u0_a51'], [1931, 'com.android.providers.calendar', 'u0_a2'], [1989, '/sbin/adbd', 'root'], [2137, '/system/bin/sh', 'root'], [2139, 'ps', 'root']] 1:30.12 INFO INFO | Result summary: 1:30.12 INFO INFO | Passed: 0 1:30.12 INFO INFO | Failed: 1 1:30.12 INFO INFO | Todo: 0 1:30.12 INFO INFO | Retried: 0 1:30.12 SUITE_END xpcshell ~~~~~~~~ Ran 1 checks (1 tests) Expected results: 0 Unexpected results: 1 test: 1 (1 fail) Unexpected Results ------------------ FAIL netwerk/test/unit/test_simple.js - xpcshell return code: 139
Reporter | ||
Comment 1•6 years ago
|
||
Also: $ adb shell # cd /sdcard/tests/xpc/netwerk/test/unit # export LD_LIBRARY_PATH=/data/local/xpcb # /data/local/xpcb/xpcshell Segmentation fault
Reporter | ||
Comment 2•6 years ago
|
||
I see the same result on the x86-7.0 emulator with an x86 build.
Reporter | ||
Updated•6 years ago
|
Reporter | ||
Updated•6 years ago
|
Comment 3•6 years ago
|
||
snorp: Do you know whom might be able to investigate this?
Comment 4•6 years ago
|
||
Assignee | ||
Comment 5•6 years ago
|
||
(In reply to Bob Clary [:bc:] from comment #4) > Created attachment 8968465 [details] > tombstone_06 Looks to me like we're not allowed to open some files needed by the android system properties, used by the system linker. This probably just means that running executables like xpcshell is not going to work anymore due to security constraints. Are we running as root? Can we try disabling selinux?
Reporter | ||
Comment 6•6 years ago
|
||
We are running as root. We are running in permissive mode (definitely on emulator; bc may have different environment on some hardware?). Note that cppunit tests run fine for me on the same 7.0 emulator...why are they different?
Reporter | ||
Comment 7•6 years ago
|
||
We have wondered about the use of LD_LIBRARY_PATH, but I think that is okay: generic_arm64:/data/local/xpcb # mv libxul.so xx.so generic_arm64:/data/local/xpcb # ./xpcw . xpcw: cd . xpcw: xpcshell CANNOT LINK EXECUTABLE "/data/local/xpcb/xpcshell": library "libxul.so" not found Aborted 134|generic_arm64:/data/local/xpcb # mv xx.so libxul.so generic_arm64:/data/local/xpcb # ./xpcw . xpcw: cd . xpcw: xpcshell Segmentation fault 139|generic_arm64:/data/local/xpcb # cat xpcw #!/system/bin/sh export MOZ_CRASHREPORTER=1 export XPCSHELL_TEST_TEMP_DIR=/sdcard/tests/xpc/tmp export XPCOM_DEBUG_BREAK=stack-and-abort export XPCSHELL_MINIDUMP_DIR=/sdcard/tests/xpc/minidumps export MOZ_DISABLE_CONTENT_SANDBOX=1 export GRE_HOME=/data/data/org.mozilla.fennec_gbrown export MOZ_DISABLE_NONLOCAL_CONNECTIONS=1 export XPCSHELL_TEST_PROFILE_DIR=/sdcard/tests/xpc/p export HOME=/sdcard/tests/xpc/p export MOZ_CRASHREPORTER_NO_REPORT=1 export MOZ_DEVELOPER_REPO_DIR=/home/gbrown/src export MOZ_LINKER_CACHE=/data/local/xpcb export LD_LIBRARY_PATH=/data/local/xpcb export TMPDIR=/sdcard/tests/xpc/tmp cd $1 echo xpcw: cd $1 shift echo xpcw: xpcshell "$@" /data/local/xpcb/xpcshell "$@" generic_arm64:/data/local/xpcb #
Comment 8•6 years ago
|
||
adb*.py sets selinux to permissive. I just checked manually and my gs3 is running SELinux Permissive and I could reproduce the crash. kmag: Any ideas on this or should we give up trying to get xpcshell running on newer Androids?
Comment 9•6 years ago
|
||
Re-triaging per https://bugzilla.mozilla.org/show_bug.cgi?id=1473195 Needinfo :susheel if you think this bug should be re-triaged.
Updated•6 years ago
|
Updated•6 years ago
|
Updated•6 years ago
|
Comment 11•6 years ago
|
||
I only ran them temporarily on android-hw. gbrown would know more about android-em but it appear they are all green on emulators atm. gbrown: anything to add?
Reporter | ||
Comment 12•6 years ago
|
||
(In reply to Bob Clary [:bc:] from comment #11) > it appear they are all green on emulators atm On treeherder, xpcshell only runs on android 4.2 and 4.3 normally. The crash still occurs on Android 7.0 x86 (probably on arm and devices too): https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=0dca735dec8a40554d558c296fb36965d8d9a1f9
Comment 13•6 years ago
|
||
(In reply to Geoff Brown [:gbrown] from comment #12) > The crash still occurs on Android 7.0 x86 (probably on arm and devices too): > > https://treeherder.mozilla.org/#/ > jobs?repo=try&tier=1%2C2%2C3&revision=0dca735dec8a40554d558c296fb36965d8d9a1f9 In that case, setting GV P1 for a crash testing.
Reporter | ||
Updated•6 years ago
|
Comment 14•5 years ago
|
||
64=wontfix because this test crash doesn't block Focus 8.0.
Comment 15•5 years ago
|
||
Sending this bug to JS triage. Are xpcshell test crashes typically JS bugs? The xpcshell tests don't crash on Android 4.3 ARM emulator, but they seem to all crash in GeckoView on Android 7 (ARM devices and ARM and x86 emulators).
Comment 16•5 years ago
|
||
Steven, does the SpiderMonkey team own the xpcshell tests? They're failing on Android 7 (ARM devices and x86 emulator), but not on Android 4.3 ARM emulator. The test failure might be a file permissions problem.
Comment 17•5 years ago
|
||
Ted, do you know who might be a good person on our team to look at this? Do we have someone that knows xpcshell on our team?
Comment 18•5 years ago
|
||
In theory, xpcshell and the test-runner are owned by XPConnect https://wiki.mozilla.org/Modules/Core#XPConnect. These android-specific tooling falls between gaps.
I am unable to reproduce this myself. (Physical devices fail without rooting, and |./mach android-emulator --version 7.0| never finishes starting up and mounting an /sdcard).
Nothing seems to indicate SpiderMonkey is particularly related here. If I had to speculate, based on Comment 4 where we crash in |operator new| during initial loading, I would say that the allocator-interception code is failuring with the libc that exists on the Android 7 image. It is possible there is some odd configuration of how the xpcshell binary is generated that is making us hit it.
Someone with Android build system and linker understanding should probably look at this.
Comment 19•5 years ago
|
||
I'll reset the [geckoview] whiteboard priority so the GeckoView team will revisit this bug in our triage meeting.
Comment 20•5 years ago
|
||
James will take a look. This is probably some new permission restriction in Android 7.
Comment 21•5 years ago
|
||
FWIW I could not get the plain JS shell to run on newer Android versions, same symptom - SEGV on startup when running out of /data/local/tmp via adb shell (device not rooted or otherwise mucked with). But I never took it into a debugger to find out more.
Comment 22•5 years ago
|
||
I wonder if this is related: https://developer.android.com/about/versions/nougat/android-7.0-changes#ndk
Updated•5 years ago
|
Comment 23•5 years ago
|
||
This is a shot in the dark, but over in Bug 1525661, I found many inscrutable issues when running out of a profile directory that is not writable. Different emulators have different permissions, and I wonder if we're seeing something like this? I expect a debugger would reveal if this was the cause almost immediately.
Reporter | ||
Comment 24•5 years ago
|
||
(In reply to Geoff Brown [:gbrown] from comment #6)
Note that cppunit tests run fine for me on the same 7.0 emulator...why are
they different?
Actually, there are a few startup crashes on 7.0; glandium is having a look at those in bug 1530874.
Reporter | ||
Comment 25•5 years ago
|
||
Updated try run, now complete with tombstones (saved as artifacts):
Reporter | ||
Comment 26•5 years ago
|
||
I stumbled on this curiosity: If I build locally with this mozconfig, xpcshell does not crash on startup on the 7.0 x86 emulator, and many tests pass:
ac_add_options --enable-application=mobile/android
ac_add_options --target=x86_64
export MOZILLA_OFFICIAL=1
CC="/home/gbrown/.mozbuild/clang/bin/clang"
CXX="/home/gbrown/.mozbuild/clang/bin/clang++"
ac_add_options --with-android-ndk="/home/gbrown/.mozbuild/android-ndk-r17b"
mk_add_options MOZ_OBJDIR=/home/gbrown/objdirs/x86_64-debug
mk_add_options AUTOCLOBBER=1
If I uncomment MOZILLA_OFFICIAL, the startup crash returns.
I cannot demonstrate this on try: If I remove MOZILLA_OFFICIAL from the nightly mozconfig, gradle fails at build time:
Comment 27•5 years ago
|
||
(In reply to Geoff Brown [:gbrown] from comment #26)
I stumbled on this curiosity: If I build locally with this mozconfig, xpcshell does not crash on startup on the 7.0 x86 emulator, and many tests pass:
ac_add_options --enable-application=mobile/android
ac_add_options --target=x86_64export MOZILLA_OFFICIAL=1
CC="/home/gbrown/.mozbuild/clang/bin/clang"
CXX="/home/gbrown/.mozbuild/clang/bin/clang++"
ac_add_options --with-android-ndk="/home/gbrown/.mozbuild/android-ndk-r17b"
mk_add_options MOZ_OBJDIR=/home/gbrown/objdirs/x86_64-debug
mk_add_options AUTOCLOBBER=1If I uncomment MOZILLA_OFFICIAL, the startup crash returns.
I cannot demonstrate this on try: If I remove MOZILLA_OFFICIAL from the nightly mozconfig, gradle fails at build time:
The Gradle failure just has to do with dependencies:
[task 2019-03-11T19:12:55.311Z] 19:12:55 INFO - > Could not find com.squareup.leakcanary:leakcanary-android:1.4-beta1.
We could make that work if we wanted to; or just work around it here: https://searchfox.org/mozilla-central/source/mobile/android/app/build.gradle#246.
MOZILLA_OFFICIAL changes many things so I don't think it'll be all that helpful for narrowing down the behaviour. Is it really this hard to track down a startup crash in an xpcshell test on Android? (I ask honestly: I don't know, I've never tried.)
Updated•5 years ago
|
Reporter | ||
Comment 28•5 years ago
|
||
This goes away with the fix from bug 1530874.
Description
•