797153 - Make remote xpcshell test runner not assume it's being run from the objdir

Status: RESOLVED FIXED

(Testing :: XPCShell Harness, defect)

Description

[Jonathan Griffin (:jgriffin)]

The remote xpcshell testrunner (which the b2g version extends) assumes that it will be run from the objdir, here:

http://mxr.mozilla.org/mozilla-central/source/testing/xpcshell/remotexpcshelltests.py#144

This means it can't be run from a test package, such as we'd use in CI.

Comment 1

[Andrew Halberstadt [:ahal]]

Attachment: Patch 1.0 - don't depend on running from objdir

This isn't a particularly elegant fix, but I'm not sure that an elegant fix is possible without re-writing how we do things in B2G.

I got this working with the following command line (with proper virtualenv activated):
> cd objdir-gecko
> make packaged-tests
> cd dist/test-package-stage/xpcshell
> python runtestsb2g.py --b2gpath $B2G_PATH --adbpath $ADB_PATH --xre-path $MOZ_HOST_BIN --emulator arm --manifest tests/xpcshell.ini --use-device-libs

Notes:
* Without --use-device-libs devicemanager times out when copying libxul.so
* I see 1058 pass, 33 fail. All 33 failures are TEST-UNEXPECTED-PASS and have the same weird log output: http://www.pastebin.mozilla.org/1961499

Seeing as these two issues are outside the scope of this bug, I propose getting this landed and I'll file new bugs for them.

Comment 2

[Jonathan Griffin (:jgriffin)]

Comment on attachment 685292 [details] [diff] [review]
Patch 1.0 - don't depend on running from objdir

Review of attachment 685292 [details] [diff] [review]:
-----------------------------------------------------------------

Awesome, thanks!  I wonder if the hang you saw while copying libxul.so is the same as the problem we used to see intermittently when installing gecko.

Comment 3

[Geoff Brown [:gbrown]]

Comment on attachment 685292 [details] [diff] [review]
Patch 1.0 - don't depend on running from objdir

Review of attachment 685292 [details] [diff] [review]:
-----------------------------------------------------------------

feedback+ with the localLib issue resolved.

::: testing/xpcshell/remotexpcshelltests.py
@@ +150,3 @@
>  
>          # Additional libraries may be found in a sub-directory such as "lib/armeabi-v7a"
>          localArmLib = os.path.join(localLib, "lib")

This one localLib reference was missed. As-is, it causes a failure for Fennec xpcshell tests; changing to ...join(self.localLib, "lib") resolves the error.

Comment 4

[Geoff Brown [:gbrown]]

(In reply to Andrew Halberstadt [:ahal] from comment #1)
> * I see 1058 pass, 33 fail. All 33 failures are TEST-UNEXPECTED-PASS and
> have the same weird log output: http://www.pastebin.mozilla.org/1961499

I find the 1058 passes as troubling as the failures! When I run Fennec xpcshell tests, there are a lot of failures. Is it possible that the tests are not running?  Running check-one-remote will show more verbose output:

/media/extra/objdir-native-droid$ make SOLO_FILE=test_simple.js -C netwerk/test check-one-remote
make: Entering directory `/media/extra/objdir-native-droid/netwerk/test'
/media/extra/objdir-native-droid/_virtualenv/bin/python -u /home/mozdev/src/config/pythonpath.py \
	  -I/home/mozdev/src/build \
	  -I/home/mozdev/src/build/mobile \
	  -I/home/mozdev/src/testing/mozbase/mozdevice/mozdevice \
	  /home/mozdev/src/testing/xpcshell/remotexpcshelltests.py \
	  --symbols-path=../../dist/crashreporter-symbols \
	  --build-info-json=../../mozinfo.json \
	  --test-path=test_simple.js \
	  --testing-modules-dir=../../_tests/modules \
	  --profile-name=fennec \
	  --verbose \
	   \
	  --dm_trans=adb \
	  --deviceIP= \
	  --objdir=../.. \
          --noSetup \
	  ../../_tests/xpcshell/netwerk/test/unit ../../_tests/xpcshell/netwerk/test/unit_ipc
...
using APK: ../../dist/fennec-20.0a1.en-US.android-arm.apk
TEST-INFO | profile dir is /data/local/tests/xpcshell/p
TEST-INFO | /media/extra/objdir-native-droid/_tests/xpcshell/netwerk/test/unit/test_simple.js | running test ...
TEST-INFO | /media/extra/objdir-native-droid/_tests/xpcshell/netwerk/test/unit/test_simple.js | full command: ['/data/local/tests/xpcshell/b/xpcw', '-r', '/data/local/tests/xpcshell/c/httpd.manifest', '--greomni', '/data/local/tests/xpcshell/b/fennec-20.0a1.en-US.android-arm.apk', '-m', '-n', '-s', '-e', 'const _HTTPD_JS_PATH = "/data/local/tests/xpcshell/c/httpd.js";', '-e', 'const _HEAD_JS_PATH = "/data/local/tests/xpcshell/head.js";', '-e', 'const _TESTING_MODULES_DIR = "/data/local/tests/xpcshell/m";', '-f', '/data/local/tests/xpcshell/head.js', '-e', 'const _SERVER_ADDR = "localhost"', '-e', 'const _HEAD_FILES = ["/data/local/tests/xpcshell/netwerk/test/unit/head_channels.js", "/data/local/tests/xpcshell/netwerk/test/unit/head_cache.js"];', '-e', 'const _TAIL_FILES = [];', '-e', 'const _TEST_FILE = ["test_simple.js"];', '-e', '_execute_test(); quit(0);']
TEST-INFO | /media/extra/objdir-native-droid/_tests/xpcshell/netwerk/test/unit/test_simple.js | current directory: '/data/local/tests/xpcshell/netwerk/test/unit'
TEST-INFO | /media/extra/objdir-native-droid/_tests/xpcshell/netwerk/test/unit/test_simple.js | environment: {'XPCOM_DEBUG_BREAK': 'stack-and-abort', 'GRE_HOME': '/data/data/org.mozilla.fennec_mozdev', 'XPCOM_MEM_LEAK_LOG': '/data/local/tests/xpcshell/p/leaks.log', 'XPCSHELL_TEST_PROFILE_DIR': '/data/local/tests/xpcshell/p', 'NSPR_LOG_MODULES': 'all:5', 'HOME': '/data/local/tests/xpcshell/p', 'MOZ_CRASHREPORTER_NO_REPORT': '1', 'MOZ_LINKER_CACHE': '/data/local/tests/xpcshell/b', 'NSPR_LOG_FILE': '/data/local/tests/xpcshell/nspr.log', 'LD_LIBRARY_PATH': '/data/local/tests/xpcshell/b', 'TMPDIR': '/data/local/tests/xpcshell/tmp', 'NS_TRACE_MALLOC_DISABLE_STACKS': '1'}
TEST-PASS | /media/extra/objdir-native-droid/_tests/xpcshell/netwerk/test/unit/test_simple.js | test passed (time: 914.944ms)
>>>>>>>
xpcw: cd /data/local/tests/xpcshell/netwerk/test/unit
xpcw: xpcshell -r /data/local/tests/xpcshell/c/httpd.manifest --greomni /data/local/tests/xpcshell/b/fennec-20.0a1.en-US.android-arm.apk -m -n -s -e const _HTTPD_JS_PATH = "/data/local/tests/xpcshell/c/httpd.js"; -e const _HEAD_JS_PATH = "/data/local/tests/xpcshell/head.js"; -e const _TESTING_MODULES_DIR = "/data/local/tests/xpcshell/m"; -f /data/local/tests/xpcshell/head.js -e const _SERVER_ADDR = "localhost" -e const _HEAD_FILES = ["/data/local/tests/xpcshell/netwerk/test/unit/head_channels.js", "/data/local/tests/xpcshell/netwerk/test/unit/head_cache.js"]; -e const _TAIL_FILES = []; -e const _TEST_FILE = ["test_simple.js"]; -e _execute_test(); quit(0);

TEST-INFO | (xpcshell/head.js) | test 1 pending

TEST-INFO | (xpcshell/head.js) | test 2 pending

TEST-INFO | (xpcshell/head.js) | test 2 finished

TEST-INFO | (xpcshell/head.js) | running event loop

TEST-PASS | /data/local/tests/xpcshell/netwerk/test/unit/head_channels.js | [ChannelListener.prototype.onStopRequest : 162] 10 == 10

TEST-PASS | test_simple.js | [checkRequest : 54] 0123456789 == 0123456789

TEST-INFO | (xpcshell/head.js) | test 1 finished

TEST-INFO | (xpcshell/head.js) | exiting test

TEST-PASS | (xpcshell/head.js) | 2 (+ 0) check(s) passed

TEST-INFO | (xpcshell/head.js) | 0 check(s) todo
<<<<<<<
INFO | Result summary:
INFO | Passed: 1
INFO | Failed: 0
INFO | Todo: 0

Comment 5

[Andrew Halberstadt [:ahal]]

(In reply to Geoff Brown [:gbrown] from comment #4)
> I find the 1058 passes as troubling as the failures!

I agree! And seeing as all the failures are TEST-UNEXPECTED-PASS it makes me think that for some reason all the tests are just getting marked as pass no matter what. I'll file a separate bug to investigate.

Comment 6

[Andrew Halberstadt [:ahal]]

Checked in with fix for the error that gbrown caught:
https://hg.mozilla.org/integration/mozilla-inbound/rev/3120e4f4ddb7

When/if we get xpcshell tests running for b2g we'll need this patch in beta and aurora. Won't check it in yet, but adding whiteboard flags to remember for when it is needed.

Comment 7

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/3120e4f4ddb7

Comment 8

[Andrew Halberstadt [:ahal]]

Getting xpcshell up and running is now a C2 (Dec 10th) priority. Given that I went ahead and landed this on beta and aurora:

https://hg.mozilla.org/releases/mozilla-beta/rev/783577f813bb
https://hg.mozilla.org/releases/mozilla-aurora/rev/73b4d37ab9cd