Closed Bug 797153 Opened 7 years ago Closed 7 years ago

Make remote xpcshell test runner not assume it's being run from the objdir

Categories

(Testing :: XPCShell Harness, defect)

defect
Not set

Tracking

(firefox18 fixed, firefox19 fixed, firefox20 fixed)

RESOLVED FIXED
Tracking Status
firefox18 --- fixed
firefox19 --- fixed
firefox20 --- fixed

People

(Reporter: jgriffin, Assigned: ahal)

References

Details

Attachments

(1 file)

The remote xpcshell testrunner (which the b2g version extends) assumes that it will be run from the objdir, here:

http://mxr.mozilla.org/mozilla-central/source/testing/xpcshell/remotexpcshelltests.py#144

This means it can't be run from a test package, such as we'd use in CI.
Assignee: nobody → ahalberstadt
Blocks: 809233
Blocks: 810547
This isn't a particularly elegant fix, but I'm not sure that an elegant fix is possible without re-writing how we do things in B2G.

I got this working with the following command line (with proper virtualenv activated):
> cd objdir-gecko
> make packaged-tests
> cd dist/test-package-stage/xpcshell
> python runtestsb2g.py --b2gpath $B2G_PATH --adbpath $ADB_PATH --xre-path $MOZ_HOST_BIN --emulator arm --manifest tests/xpcshell.ini --use-device-libs

Notes:
* Without --use-device-libs devicemanager times out when copying libxul.so
* I see 1058 pass, 33 fail. All 33 failures are TEST-UNEXPECTED-PASS and have the same weird log output: http://www.pastebin.mozilla.org/1961499

Seeing as these two issues are outside the scope of this bug, I propose getting this landed and I'll file new bugs for them.
Attachment #685292 - Flags: review?(jgriffin)
Attachment #685292 - Flags: feedback?(gbrown)
Comment on attachment 685292 [details] [diff] [review]
Patch 1.0 - don't depend on running from objdir

Review of attachment 685292 [details] [diff] [review]:
-----------------------------------------------------------------

Awesome, thanks!  I wonder if the hang you saw while copying libxul.so is the same as the problem we used to see intermittently when installing gecko.
Attachment #685292 - Flags: review?(jgriffin) → review+
Comment on attachment 685292 [details] [diff] [review]
Patch 1.0 - don't depend on running from objdir

Review of attachment 685292 [details] [diff] [review]:
-----------------------------------------------------------------

feedback+ with the localLib issue resolved.

::: testing/xpcshell/remotexpcshelltests.py
@@ +150,3 @@
>  
>          # Additional libraries may be found in a sub-directory such as "lib/armeabi-v7a"
>          localArmLib = os.path.join(localLib, "lib")

This one localLib reference was missed. As-is, it causes a failure for Fennec xpcshell tests; changing to ...join(self.localLib, "lib") resolves the error.
Attachment #685292 - Flags: feedback?(gbrown) → feedback+
(In reply to Andrew Halberstadt [:ahal] from comment #1)
> * I see 1058 pass, 33 fail. All 33 failures are TEST-UNEXPECTED-PASS and
> have the same weird log output: http://www.pastebin.mozilla.org/1961499

I find the 1058 passes as troubling as the failures! When I run Fennec xpcshell tests, there are a lot of failures. Is it possible that the tests are not running?  Running check-one-remote will show more verbose output:

/media/extra/objdir-native-droid$ make SOLO_FILE=test_simple.js -C netwerk/test check-one-remote
make: Entering directory `/media/extra/objdir-native-droid/netwerk/test'
/media/extra/objdir-native-droid/_virtualenv/bin/python -u /home/mozdev/src/config/pythonpath.py \
	  -I/home/mozdev/src/build \
	  -I/home/mozdev/src/build/mobile \
	  -I/home/mozdev/src/testing/mozbase/mozdevice/mozdevice \
	  /home/mozdev/src/testing/xpcshell/remotexpcshelltests.py \
	  --symbols-path=../../dist/crashreporter-symbols \
	  --build-info-json=../../mozinfo.json \
	  --test-path=test_simple.js \
	  --testing-modules-dir=../../_tests/modules \
	  --profile-name=fennec \
	  --verbose \
	   \
	  --dm_trans=adb \
	  --deviceIP= \
	  --objdir=../.. \
          --noSetup \
	  ../../_tests/xpcshell/netwerk/test/unit ../../_tests/xpcshell/netwerk/test/unit_ipc
...
using APK: ../../dist/fennec-20.0a1.en-US.android-arm.apk
TEST-INFO | profile dir is /data/local/tests/xpcshell/p
TEST-INFO | /media/extra/objdir-native-droid/_tests/xpcshell/netwerk/test/unit/test_simple.js | running test ...
TEST-INFO | /media/extra/objdir-native-droid/_tests/xpcshell/netwerk/test/unit/test_simple.js | full command: ['/data/local/tests/xpcshell/b/xpcw', '-r', '/data/local/tests/xpcshell/c/httpd.manifest', '--greomni', '/data/local/tests/xpcshell/b/fennec-20.0a1.en-US.android-arm.apk', '-m', '-n', '-s', '-e', 'const _HTTPD_JS_PATH = "/data/local/tests/xpcshell/c/httpd.js";', '-e', 'const _HEAD_JS_PATH = "/data/local/tests/xpcshell/head.js";', '-e', 'const _TESTING_MODULES_DIR = "/data/local/tests/xpcshell/m";', '-f', '/data/local/tests/xpcshell/head.js', '-e', 'const _SERVER_ADDR = "localhost"', '-e', 'const _HEAD_FILES = ["/data/local/tests/xpcshell/netwerk/test/unit/head_channels.js", "/data/local/tests/xpcshell/netwerk/test/unit/head_cache.js"];', '-e', 'const _TAIL_FILES = [];', '-e', 'const _TEST_FILE = ["test_simple.js"];', '-e', '_execute_test(); quit(0);']
TEST-INFO | /media/extra/objdir-native-droid/_tests/xpcshell/netwerk/test/unit/test_simple.js | current directory: '/data/local/tests/xpcshell/netwerk/test/unit'
TEST-INFO | /media/extra/objdir-native-droid/_tests/xpcshell/netwerk/test/unit/test_simple.js | environment: {'XPCOM_DEBUG_BREAK': 'stack-and-abort', 'GRE_HOME': '/data/data/org.mozilla.fennec_mozdev', 'XPCOM_MEM_LEAK_LOG': '/data/local/tests/xpcshell/p/leaks.log', 'XPCSHELL_TEST_PROFILE_DIR': '/data/local/tests/xpcshell/p', 'NSPR_LOG_MODULES': 'all:5', 'HOME': '/data/local/tests/xpcshell/p', 'MOZ_CRASHREPORTER_NO_REPORT': '1', 'MOZ_LINKER_CACHE': '/data/local/tests/xpcshell/b', 'NSPR_LOG_FILE': '/data/local/tests/xpcshell/nspr.log', 'LD_LIBRARY_PATH': '/data/local/tests/xpcshell/b', 'TMPDIR': '/data/local/tests/xpcshell/tmp', 'NS_TRACE_MALLOC_DISABLE_STACKS': '1'}
TEST-PASS | /media/extra/objdir-native-droid/_tests/xpcshell/netwerk/test/unit/test_simple.js | test passed (time: 914.944ms)
>>>>>>>
xpcw: cd /data/local/tests/xpcshell/netwerk/test/unit
xpcw: xpcshell -r /data/local/tests/xpcshell/c/httpd.manifest --greomni /data/local/tests/xpcshell/b/fennec-20.0a1.en-US.android-arm.apk -m -n -s -e const _HTTPD_JS_PATH = "/data/local/tests/xpcshell/c/httpd.js"; -e const _HEAD_JS_PATH = "/data/local/tests/xpcshell/head.js"; -e const _TESTING_MODULES_DIR = "/data/local/tests/xpcshell/m"; -f /data/local/tests/xpcshell/head.js -e const _SERVER_ADDR = "localhost" -e const _HEAD_FILES = ["/data/local/tests/xpcshell/netwerk/test/unit/head_channels.js", "/data/local/tests/xpcshell/netwerk/test/unit/head_cache.js"]; -e const _TAIL_FILES = []; -e const _TEST_FILE = ["test_simple.js"]; -e _execute_test(); quit(0);

TEST-INFO | (xpcshell/head.js) | test 1 pending

TEST-INFO | (xpcshell/head.js) | test 2 pending

TEST-INFO | (xpcshell/head.js) | test 2 finished

TEST-INFO | (xpcshell/head.js) | running event loop

TEST-PASS | /data/local/tests/xpcshell/netwerk/test/unit/head_channels.js | [ChannelListener.prototype.onStopRequest : 162] 10 == 10

TEST-PASS | test_simple.js | [checkRequest : 54] 0123456789 == 0123456789

TEST-INFO | (xpcshell/head.js) | test 1 finished

TEST-INFO | (xpcshell/head.js) | exiting test

TEST-PASS | (xpcshell/head.js) | 2 (+ 0) check(s) passed

TEST-INFO | (xpcshell/head.js) | 0 check(s) todo
<<<<<<<
INFO | Result summary:
INFO | Passed: 1
INFO | Failed: 0
INFO | Todo: 0
(In reply to Geoff Brown [:gbrown] from comment #4)
> I find the 1058 passes as troubling as the failures!

I agree! And seeing as all the failures are TEST-UNEXPECTED-PASS it makes me think that for some reason all the tests are just getting marked as pass no matter what. I'll file a separate bug to investigate.
Checked in with fix for the error that gbrown caught:
https://hg.mozilla.org/integration/mozilla-inbound/rev/3120e4f4ddb7

When/if we get xpcshell tests running for b2g we'll need this patch in beta and aurora. Won't check it in yet, but adding whiteboard flags to remember for when it is needed.
Whiteboard: [automation-needed-in-aurora][automation-needed-in-beta]
Blocks: 787578
https://hg.mozilla.org/mozilla-central/rev/3120e4f4ddb7
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla20
Getting xpcshell up and running is now a C2 (Dec 10th) priority. Given that I went ahead and landed this on beta and aurora:

https://hg.mozilla.org/releases/mozilla-beta/rev/783577f813bb
https://hg.mozilla.org/releases/mozilla-aurora/rev/73b4d37ab9cd
Whiteboard: [automation-needed-in-aurora][automation-needed-in-beta]
Target Milestone: mozilla20 → ---
You need to log in before you can comment on or make changes to this bug.