Closed Bug 1608932 Opened 4 years ago Closed 4 years ago

Unable to run mochitests or reftests locally in Android emulator, get "remote object '/sdcard/tests/logs/mochitest.log' does not exist" errors

Categories

(Firefox for Android Graveyard :: Testing, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 1607818

People

(Reporter: botond, Unassigned)

References

(Regression)

Details

(Keywords: regression)

Attachments

(1 file)

With recent m-c, I've been unable to run mochitests locally in an Android emulator.

I get output like this:

$ ./mach mochitest --keep-open=false dom/html/test/test_bug500885.html   
 0:00.25 adb INFO Using adb 1.0.41
 0:00.71 adb INFO adbd running as root
 0:00.82 adb INFO su -c setenforce 0 exitcode 0, stdout: None
 0:01.16 adb INFO su 0 supported
 0:01.39 adb INFO /system/bin/ls -1A supported
 0:01.50 adb INFO Native cp support: True
 0:01.62 adb INFO Native chmod -R support: True
 0:01.73 adb INFO Native chown -R support: True
 0:01.85 adb INFO Setting SELinux Permissive
 0:02.07 adb INFO Native pidof support: True
Re-install geckoview AndroidTest? (Y/n) n
 1:04.27 adb INFO Using adb 1.0.41
 1:04.73 adb INFO adbd running as root
 1:04.84 adb INFO su -c setenforce 0 exitcode 0, stdout: None
 1:05.19 adb INFO su 0 supported
 1:05.41 adb INFO /system/bin/ls -1A supported
 1:05.53 adb INFO Native cp support: True
 1:05.64 adb INFO Native chmod -R support: True
 1:05.75 adb INFO Native chown -R support: True
 1:05.86 adb INFO Setting SELinux Permissive
 1:06.09 adb INFO Native pidof support: True
 1:11.34 INFO Android sdk version '24'; will use this to filter manifests
 1:11.35 INFO Checking for ssltunnel processes...
 1:11.39 INFO Checking for xpcshell processes...
 1:11.41 SUITE_START: mochitest-plain - running 1 tests
 1:11.41 INFO Running manifest: dom/html/test/mochitest.ini
 1:11.41 INFO The following extra prefs will be set:
  plugin.load_flash_only=false
pk12util: PKCS12 IMPORT SUCCESSFUL
 1:14.52 INFO MochitestServer : launching [u'/home/botond/.mozbuild/android-device/host-utils-71.0a1.en-US.linux-x86_64/xpcshell', '-g', '/home/botond/.mozbuild/android-device/host-utils-71.0a1.en-US.linux-x86_64', '-f', '/home/botond/.mozbuild/android-device/host-utils-71.0a1.en-US.linux-x86_64/components/httpd.js', '-e', "const _PROFILE_PATH = '/tmp/tmpDQbZG9.mozrunner'; const _SERVER_PORT = '8888'; const _SERVER_ADDR = '10.242.24.73'; const _TEST_PREFIX = undefined; const _DISPLAY_RESULTS = false;", '-f', '/home/botond/dev/mozilla/out-of-tree-objdirs/refactoring-android-x86/_tests/testing/mochitest/server.js']
 1:14.52 INFO runtests.py | Server pid: 6673
 1:14.53 INFO runtests.py | Websocket server pid: 6676
 1:14.55 INFO runtests.py | SSL tunnel pid: 6679
 1:15.17 INFO runtests.py | Running with scheme: http
 1:15.17 INFO runtests.py | Running with e10s: True
 1:15.17 INFO runtests.py | Running with fission: False
 1:15.17 INFO runtests.py | Running with serviceworker_e10s: True
 1:15.17 INFO runtests.py | Running with socketprocess_e10s: False
 1:15.17 INFO runtests.py | Running tests: start.

 1:15.51 adb INFO Granting important runtime permissions to org.mozilla.geckoview.test
 1:16.78 adb INFO launch_application: am start -W -n org.mozilla.geckoview.test/org.mozilla.geckoview.test.TestRunnerActivity -a android.intent.action.MAIN --es env9 MOZ_UPLOAD_DIR=/sdcard/tests/mozlog --es env8 R_LOG_DESTINATION=stderr --es args "-no-remote -profile /sdcard/tests/profile//" --es env3 R_LOG_VERBOSE=1 --es env2 XPCOM_DEBUG_BREAK=stack --es env1 MOZ_WEBRENDER=0 --es env0 MOZ_CRASHREPORTER=1 --es env7 MOZ_CRASHREPORTER_SHUTDOWN=1 --es env6 MOZ_IN_AUTOMATION=1 --es env5 MOZ_DISABLE_NONLOCAL_CONNECTIONS=1 --es env4 DISABLE_UNSAFE_CPOW_WARNINGS=1 --ez use_multiprocess True --es env13 MOZ_HIDE_RESULTS_TABLE=1 --es env12 R_LOG_LEVEL=6 --es env11 MOZ_DEVELOPER_REPO_DIR=/home/botond/dev/mozilla/refactoring --es env10 MOZ_CRASHREPORTER_NO_REPORT=1 -d "http://mochi.test:8888/tests?autorun=1&closeWhenDone=1&logFile=%2Fsdcard%2Ftests%2Flogs%2Fmochitest.log&fileLevel=INFO&consoleLevel=INFO&hideResultsTable=1&manifestFile=tests.json&dumpOutputDirectory=%2Fsdcard%2Ftests"
remoteautomation.py | Application pid: 31848
Failed wait for remote log: /sdcard/tests/logs/mochitest.log missing?
remoteautomation.py | exception reading log: args: /home/botond/.mozbuild/android-sdk-linux/platform-tools/adb wait-for-device pull /sdcard/tests/logs/mochitest.log /tmp/tmpJnsFx4, exitcode: 1, stdout: adb: error: remote object '/sdcard/tests/logs/mochitest.log' does not exist
remoteautomation.py | exception reading log: args: /home/botond/.mozbuild/android-sdk-linux/platform-tools/adb wait-for-device pull /sdcard/tests/logs/mochitest.log /tmp/tmpbUbkCy, exitcode: 1, stdout: adb: error: remote object '/sdcard/tests/logs/mochitest.log' does not exist
wait for org.mozilla.geckoview.test complete; top activity=com.android.launcher3
remoteautomation.py | Application ran for: 0:00:35.358154
 1:51.32 CRASH: pid:6327. Test:remoteautomation.py. Minidump analysed:False. Signature:[None]
Crash dump filename: /tmp/tmpqrUA5a/29c75af1-fffc-a245-ef05-4279a3c0a035.dmp
MINIDUMP_STACKWALK not set, can't process dump.

 1:51.32 INFO Stopping web server
 1:51.33 INFO Stopping web socket server
 1:51.35 INFO Stopping ssltunnel
 1:51.38 WARNING leakcheck | refcount logging is off, so leaks can't be detected!
 1:51.38 INFO runtests.py | Running tests: end.
 1:52.61 INFO Buffered messages finished
0 INFO TEST-START | Shutdown
1 INFO Passed:  0
2 INFO Failed:  0
3 INFO Todo:    0
4 INFO Mode:    e10s
5 INFO SimpleTest FINISHED
 1:53.74 INFO Buffered messages finished
 1:53.74 SUITE_END
 1:53.74
Overall Summary
===============

mochitest-plain
~~~~~~~~~~~~~~~
Ran 0 checks ()
Expected results: 0
Unexpected results: 0
OK

Note that it didn't actually run anything.

I guess the reason has to do with the following errors?

adb: error: remote object '/sdcard/tests/logs/mochitest.log' does not exist

How can I diagnose this further?

I see the same output, with a current x86_64 build. I see the test app start briefly in the emulator. logcat shows a crash, apparently before any tests are run (so there is no log generated on the device).

I can run the geckoview test app manually (click on the app icon in the emulator), but all mochitests crash like this. The cause of the crash doesn't jump out at me from the logcat messages, but I do see:

E GeckoConsole: [JavaScript Error: "SyntaxError: JSON.parse: unexpected end of data at line 7059 column 6 of the JSON data"]

a little before the crash. :botond, do you see the same thing?

Flags: needinfo?(botond)
Attached file Logcat

Hmm, no, I don't see that error. I've attached my complete logcat.

Flags: needinfo?(botond)

Reftests are affected as well.

Summary: Unable to run mochitest locally in Android emulator, get "remote object '/sdcard/tests/logs/mochitest.log' does not exist" errors → Unable to run mochitests or reftests locally in Android emulator, get "remote object '/sdcard/tests/logs/mochitest.log' does not exist" errors
Blocks: 1556556
Keywords: regression

Regression window:

https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=370577749507ae071a1b9126f2df4c1533d094ed&tochange=155706ccc7e418948e534b52de33d85416455875

which is bug 1544147.

(This regression window, which required building from source at each step, was brought to you in part by the Toronto office's sccache cluster, which can do an Android clobber build in ~7 minutes.)

Flags: needinfo?(gsvelto)
Regressed by: 1544147
Has Regression Range: --- → yes

Oh, debugging this is going to be interesting. Bug 1544147 should have no effect on the tests unless a non-main process is failing very early during startup - before an IPC channel to the main process is established. Either bug 1544147 is causing those crashes, which would probably be a simple fix, or it's triggering a race when dealing with those crashes that we didn't know was there (a sadly really common occurrence in crash reporting code).

I'll make an Android build and investigate this later today or tomorrow at the latest, leaving the NI? for now.

It seems that when we start running a test the socket process gets closed in an abnormal way. Previously this would go unnoticed by with my change the code is trying to grab a minidump and something is going wrong at that point. That makes two issues to solve but I'll start with the code I changed because that should work regardless of when processes die (or why).

Flags: needinfo?(gsvelto)

It seems odd that tests are running fine in continuous integration, even mochitest/reftest on emulator; I would be interested in knowing what is different: can we change one test environment or the other to make them more alike and avoid future discrepancies?

Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → DUPLICATE
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: