797162 - Make it possible to run b2g xpcshell tests from objdir/_tests/xpcshell

Status: RESOLVED FIXED

(Testing :: XPCShell Harness, defect, P1)

Description

[Jonathan Griffin (:jgriffin)]

It isn't possible right now to directly run xpcshell tests on b2g from the objdir/_tests/xpcshell folder, since the test runner requires several files that aren't present there, like automation.py and b2gautomation.py.

On desktop, this isn't much of an issue since the tests are kicked off with a make target, which uses pythonpath to insert the path to the necessary files.  On B2G, using a make target is more awkward, since there are a number of different parameters that need to be specified depending on whether you want to run the tests on a device or an emulator.

Let's fix this by copying automation.py etc into objdir/_tests/xpcshell as we do for mochitests.

The alternative would be to write a make target that requires a number of environment variables to run successfully, ala http://mxr.mozilla.org/mozilla-central/source/testing/testsuite-targets.mk#263.  I'm marginally in favor of the copy-file approach, since it seems less opaque.

Comment 1

[Jonathan Griffin (:jgriffin)]

Attachment: Add make target for running B2G xpcshell tests locally

Comment 2

[Jonathan Griffin (:jgriffin)]

@marshall_law, you can use this patch locally for running xpcshell tests on B2G. To use it:

1 - define the B2G_HOME var, which should point to your B2G repo
2 - either have adb on your path, or set ADB_PATH with the location of adb
3 - build B2G, and then go to objdir-gecko, and run:

make xpcshell-tests-b2g

to run all the tests, or

TEST_PATH=path/to/test make xpcshell-tests-b2g to run a specific test.  TEST_PATH is relative to objdir-gecko/_tests/xpcshell, so e.g.,

TEST_PATH=chrome/test/unit/test_crlf.js make xpcshell-tests-b2g

This will work on either an emulator or a device, provided you have it running and it shows up in 'adb devices'.  It will likely fall over if you have both an emulator running and a connected device.

One of the first things that the testrunner does is to copy all of the xpcshell test files over to the device, which can take a very long time.  You'll save yourself enormous amounts of time by installing busybox before you run this, so all the files can be transferred as a single zip.  This script explains how to do that:  https://github.com/mihneadb/Negatus/blob/master/setup-tools.sh

I'll transcribe this to MDN after this lands.

@mihnea, if you want to adapt test.sh to use this make target, that would be cool.  If you don't have time, I'll do it.

Comment 3

[Marshall Culpepper [:marshall_law]]

(In reply to Jonathan Griffin (:jgriffin) from comment #2)
> @marshall_law, you can use this patch locally for running xpcshell tests on
> B2G. 

\o/ I'll start playing with it today :)

> You'll save yourself enormous amounts of time by installing busybox before you run this

FYI, we distribute a prebuilt version of busybox in gaia that I use for my local testing of updates (busybox includes a lightweight http server). In your script, you should be able to find it in $B2G_HOME/gaia/busybox-armv6l. I actually have a zsh function I wrote for my profile for pushing this, if it helps any:

https://github.com/marshall/dotfiles/blob/master/mozilla/b2g.zsh#L17

Comment 4

[Marshall Culpepper [:marshall_law]]

My mistake, the busybox path in gaia should be $B2G_HOME/gaia/build/busybox-armv6l

Comment 5

[Andrew Overholt [:overholt]]

Blocking all xpcshell tests is a blocker.

Comment 6

[Marshall Culpepper [:marshall_law]]

To clarify, this blocks running xpcshell tests on device/emulator for gecko devs, but not running them on desktop AFAIK.

Comment 7

[Marshall Culpepper [:marshall_law]]

Attachment: xpcshell-tests-b2g errors

I ran into a few errors when running this command in objdir-gecko/_tests/xpcshell:

B2G_HOME=/Users/mculpepper/Code/B2G-emulator TEST_PATH=toolkit/mozapps/update/test_timermanager make xpcshell-tests-b2g

- If /data/local/tests doesn't exist on the device, the python script will crash the first time it's run, but it succeeds on subsequent attempts. STR: just adb shell rm -r /data/local/tests and try running 'make xpcshell-tests-b2g'

- Something is trying to push "self.options.localAPK" to the device, which is apparently None. This error happens every time for me.

This attached log has the exception from each error and debug logging I added so you can see each adb command to verify..

Comment 8

[Andrew Halberstadt [:ahal]]

FYI jgriffin is PTO until Tuesday.

Devicemanager (https://github.com/mozilla/mozbase/commits/master/mozdevice) has undergone a substantial refactoring in the last week or two in attempt to get it into a usable state before every harness and its nephew depends on it. I wouldn't be surprised if there was a recent regression that is causing those errors. I don't know anything about the xpcshell harness, but I'll see if I can isolate the problem in devicemanager.

Comment 9

[Mihnea Dobrescu-Balaur (:mihneadb)]

(In reply to Jonathan Griffin (:jgriffin) from comment #2)

> @mihnea, if you want to adapt test.sh to use this make target, that would be
> cool.  If you don't have time, I'll do it.

I will have time to look at this this weekend.

Comment 10

[Jonathan Griffin (:jgriffin)]

@marshall_law, that problem you were seeing is being addressed by bug 799288.

Comment 11

[Mihnea Dobrescu-Balaur (:mihneadb)]

Attachment: patch for B2G repo to include xpcshell tests in test.sh

This is what I came up with. I wasn't able to test it because I don't have the Makefiles in my objdir-gecko folder after I do ./config.sh emulator && ./build.sh.

Am I missing something?

Comment 12

[Mihnea Dobrescu-Balaur (:mihneadb)]

Attachment: patch for B2G repo to include xpcshell tests in test.sh

Forgot to export ADB_PATH.

Comment 13

[Mihnea Dobrescu-Balaur (:mihneadb)]

Attachment: add xpcshell to test.sh

Ok, figured there was no need to reset ADB_PATH to the one in the PATH (if it exists), since the make target does that anyway.

Comment 14

[Alex Keybl [:akeybl]]

Blocks a P1/blocker. Marking this as a P1/blocker.

Comment 15

[Jonathan Griffin (:jgriffin)]

Comment on attachment 671565 [details] [diff] [review]
add xpcshell to test.sh

Review of attachment 671565 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks Mihnea.  I think this is right.  We'll have to wait until the other patch lands to land this.

Comment 16

[(not currently active) Ted Mielczarek]

Comment on attachment 668177 [details] [diff] [review]
Add make target for running B2G xpcshell tests locally

Review of attachment 668177 [details] [diff] [review]:
-----------------------------------------------------------------

r- because I don't want to add more uses of pythonpath.py but otherwise looks ok.

::: testing/testsuite-targets.mk
@@ +280,5 @@
> +	$(PYTHON) -u $(topsrcdir)/config/pythonpath.py \
> +	  -I$(topsrcdir)/build \
> +	  -I$(topsrcdir)/build/mobile \
> +	  -I$(topsrcdir)/testing/marionette/client/marionette \
> +	  -I$(topsrcdir)/testing/mozbase/mozdevice/mozdevice \

This is horrible. Can you just add this stuff to the virtualenv instead?

@@ +297,5 @@
> +xpcshell-tests-b2g:
> +	@if [ "${B2G_HOME}" = "" ]; then \
> +		echo "Please set the B2G_HOME variable"; exit 1; \
> +	elif [ ! -f "${ADB_PATH}" ]; then \
> +		echo "Please set the ADB_PATH variable"; exit 1; \

This seems redundant given that you're setting ADB_PATH above. Can't the harness just look for adb in $PATH? Seems like that'd be nicer than this setup.

Comment 17

[Jonathan Griffin (:jgriffin)]

Attachment: Add make target for running B2G xpcshell tests locally, v0.2

This adds marionette and related dependencies to the in-tree virtualenv, and updates the remote xpcshell targets to use them.  This also updates remotexpcshelltests.py and runtestsb2g.py to expect devicemanager and related classes to belong to the mozdevice package.

> @@ +297,5 @@
> > +xpcshell-tests-b2g:
> > +	@if [ "${B2G_HOME}" = "" ]; then \
> > +		echo "Please set the B2G_HOME variable"; exit 1; \
> > +	elif [ ! -f "${ADB_PATH}" ]; then \
> > +		echo "Please set the ADB_PATH variable"; exit 1; \
> 
> This seems redundant given that you're setting ADB_PATH above. Can't the 
> harness just look for adb in $PATH? Seems like that'd be nicer than this 
> setup.

It isn't redundant since adb isn't guaranteed to be on the path.  If it is, great, if not, we allow someone to pass the path directly to the script.  The test.sh script that will invoke this from the B2G directory will pass this environment variable.

Comment 18

[Jonathan Griffin (:jgriffin)]

Pushed to try, just in case:  https://tbpl.mozilla.org/?tree=Try&rev=fbd5bc33f77d.

We're not running xpcshell test on Fennec, so this won't have any impact there.

Comment 19

[Mozilla RelEng Bot]

Try run for fbd5bc33f77d is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=fbd5bc33f77d
Results (out of 16 total builds):
    success: 16
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-fbd5bc33f77d

Comment 20

[Jonathan Griffin (:jgriffin)]

Try run green, review ping to :ted

Comment 21

[(not currently active) Ted Mielczarek]

Comment on attachment 673959 [details] [diff] [review]
Add make target for running B2G xpcshell tests locally, v0.2

Review of attachment 673959 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/testsuite-targets.mk
@@ +286,5 @@
> +	  --no-clean \
> +	  --objdir=$(DEPTH) \
> +	  $$EXTRA_XPCSHELL_ARGS \
> +	  --b2gpath=${B2G_HOME} \
> +	  $(TEST_PATH_ARG) $(EXTRA_TEST_ARGS)

I can't wait till we can move this all into mach and get rid of these huge horrible makefile blocks.

@@ +293,5 @@
> +xpcshell-tests-b2g:
> +	@if [ "${B2G_HOME}" = "" ]; then \
> +		echo "Please set the B2G_HOME variable"; exit 1; \
> +	elif [ ! -f "${ADB_PATH}" ]; then \
> +		echo "Please set the ADB_PATH variable"; exit 1; \

I still think this should probably just be tested in runtestsb2g.py, but I'm not about to r- you over that.

Comment 22

[Jonathan Griffin (:jgriffin)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/c0d55648fbf5

Comment 23

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/c0d55648fbf5

Comment 24

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/f1780c4b50a5