Make it possible to run b2g xpcshell tests from objdir/_tests/xpcshell

RESOLVED FIXED in Firefox 18

Status

P1
normal
RESOLVED FIXED
6 years ago
6 years ago

People

(Reporter: jgriffin, Assigned: jgriffin)

Tracking

unspecified
mozilla19
All
Gonk (Firefox OS)
Points:
---
Bug Flags:
in-testsuite +

Firefox Tracking Flags

(blocking-basecamp:+, firefox18 fixed, firefox19 fixed)

Details

Attachments

(3 attachments, 3 obsolete attachments)

(Assignee)

Description

6 years ago
It isn't possible right now to directly run xpcshell tests on b2g from the objdir/_tests/xpcshell folder, since the test runner requires several files that aren't present there, like automation.py and b2gautomation.py.

On desktop, this isn't much of an issue since the tests are kicked off with a make target, which uses pythonpath to insert the path to the necessary files.  On B2G, using a make target is more awkward, since there are a number of different parameters that need to be specified depending on whether you want to run the tests on a device or an emulator.

Let's fix this by copying automation.py etc into objdir/_tests/xpcshell as we do for mochitests.

The alternative would be to write a make target that requires a number of environment variables to run successfully, ala http://mxr.mozilla.org/mozilla-central/source/testing/testsuite-targets.mk#263.  I'm marginally in favor of the copy-file approach, since it seems less opaque.
(Assignee)

Updated

6 years ago
Assignee: nobody → jgriffin
(Assignee)

Comment 1

6 years ago
Created attachment 668177 [details] [diff] [review]
Add make target for running B2G xpcshell tests locally
Attachment #668177 - Flags: review?(ted.mielczarek)
(Assignee)

Comment 2

6 years ago
@marshall_law, you can use this patch locally for running xpcshell tests on B2G. To use it:

1 - define the B2G_HOME var, which should point to your B2G repo
2 - either have adb on your path, or set ADB_PATH with the location of adb
3 - build B2G, and then go to objdir-gecko, and run:

make xpcshell-tests-b2g

to run all the tests, or

TEST_PATH=path/to/test make xpcshell-tests-b2g to run a specific test.  TEST_PATH is relative to objdir-gecko/_tests/xpcshell, so e.g.,

TEST_PATH=chrome/test/unit/test_crlf.js make xpcshell-tests-b2g

This will work on either an emulator or a device, provided you have it running and it shows up in 'adb devices'.  It will likely fall over if you have both an emulator running and a connected device.

One of the first things that the testrunner does is to copy all of the xpcshell test files over to the device, which can take a very long time.  You'll save yourself enormous amounts of time by installing busybox before you run this, so all the files can be transferred as a single zip.  This script explains how to do that:  https://github.com/mihneadb/Negatus/blob/master/setup-tools.sh

I'll transcribe this to MDN after this lands.

@mihnea, if you want to adapt test.sh to use this make target, that would be cool.  If you don't have time, I'll do it.
(In reply to Jonathan Griffin (:jgriffin) from comment #2)
> @marshall_law, you can use this patch locally for running xpcshell tests on
> B2G. 

\o/ I'll start playing with it today :)

> You'll save yourself enormous amounts of time by installing busybox before you run this

FYI, we distribute a prebuilt version of busybox in gaia that I use for my local testing of updates (busybox includes a lightweight http server). In your script, you should be able to find it in $B2G_HOME/gaia/busybox-armv6l. I actually have a zsh function I wrote for my profile for pushing this, if it helps any:

https://github.com/marshall/dotfiles/blob/master/mozilla/b2g.zsh#L17
My mistake, the busybox path in gaia should be $B2G_HOME/gaia/build/busybox-armv6l
Blocks: 787578
blocking-basecamp: --- → ?
Blocking all xpcshell tests is a blocker.
blocking-basecamp: ? → +
To clarify, this blocks running xpcshell tests on device/emulator for gecko devs, but not running them on desktop AFAIK.
Created attachment 668523 [details]
xpcshell-tests-b2g errors

I ran into a few errors when running this command in objdir-gecko/_tests/xpcshell:

B2G_HOME=/Users/mculpepper/Code/B2G-emulator TEST_PATH=toolkit/mozapps/update/test_timermanager make xpcshell-tests-b2g

- If /data/local/tests doesn't exist on the device, the python script will crash the first time it's run, but it succeeds on subsequent attempts. STR: just adb shell rm -r /data/local/tests and try running 'make xpcshell-tests-b2g'

- Something is trying to push "self.options.localAPK" to the device, which is apparently None. This error happens every time for me.

This attached log has the exception from each error and debug logging I added so you can see each adb command to verify..
FYI jgriffin is PTO until Tuesday.

Devicemanager (https://github.com/mozilla/mozbase/commits/master/mozdevice) has undergone a substantial refactoring in the last week or two in attempt to get it into a usable state before every harness and its nephew depends on it. I wouldn't be surprised if there was a recent regression that is causing those errors. I don't know anything about the xpcshell harness, but I'll see if I can isolate the problem in devicemanager.
(In reply to Jonathan Griffin (:jgriffin) from comment #2)

> @mihnea, if you want to adapt test.sh to use this make target, that would be
> cool.  If you don't have time, I'll do it.

I will have time to look at this this weekend.
(Assignee)

Comment 10

6 years ago
@marshall_law, that problem you were seeing is being addressed by bug 799288.
Created attachment 671246 [details] [diff] [review]
patch for B2G repo to include xpcshell tests in test.sh

This is what I came up with. I wasn't able to test it because I don't have the Makefiles in my objdir-gecko folder after I do ./config.sh emulator && ./build.sh.

Am I missing something?
Attachment #671246 - Flags: feedback?(jgriffin)
Created attachment 671249 [details] [diff] [review]
patch for B2G repo to include xpcshell tests in test.sh

Forgot to export ADB_PATH.
Attachment #671246 - Attachment is obsolete: true
Attachment #671246 - Flags: feedback?(jgriffin)
Attachment #671249 - Flags: feedback?(jgriffin)
Created attachment 671565 [details] [diff] [review]
add xpcshell to test.sh

Ok, figured there was no need to reset ADB_PATH to the one in the PATH (if it exists), since the make target does that anyway.
Attachment #671249 - Attachment is obsolete: true
Attachment #671249 - Flags: feedback?(jgriffin)
Attachment #671565 - Flags: feedback?(jgriffin)
Blocks a P1/blocker. Marking this as a P1/blocker.
Priority: -- → P1
(Assignee)

Comment 15

6 years ago
Comment on attachment 671565 [details] [diff] [review]
add xpcshell to test.sh

Review of attachment 671565 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks Mihnea.  I think this is right.  We'll have to wait until the other patch lands to land this.
Attachment #671565 - Flags: feedback?(jgriffin) → feedback+
Comment on attachment 668177 [details] [diff] [review]
Add make target for running B2G xpcshell tests locally

Review of attachment 668177 [details] [diff] [review]:
-----------------------------------------------------------------

r- because I don't want to add more uses of pythonpath.py but otherwise looks ok.

::: testing/testsuite-targets.mk
@@ +280,5 @@
> +	$(PYTHON) -u $(topsrcdir)/config/pythonpath.py \
> +	  -I$(topsrcdir)/build \
> +	  -I$(topsrcdir)/build/mobile \
> +	  -I$(topsrcdir)/testing/marionette/client/marionette \
> +	  -I$(topsrcdir)/testing/mozbase/mozdevice/mozdevice \

This is horrible. Can you just add this stuff to the virtualenv instead?

@@ +297,5 @@
> +xpcshell-tests-b2g:
> +	@if [ "${B2G_HOME}" = "" ]; then \
> +		echo "Please set the B2G_HOME variable"; exit 1; \
> +	elif [ ! -f "${ADB_PATH}" ]; then \
> +		echo "Please set the ADB_PATH variable"; exit 1; \

This seems redundant given that you're setting ADB_PATH above. Can't the harness just look for adb in $PATH? Seems like that'd be nicer than this setup.
Attachment #668177 - Flags: review?(ted.mielczarek) → review-
(Assignee)

Comment 17

6 years ago
Created attachment 673959 [details] [diff] [review]
Add make target for running B2G xpcshell tests locally, v0.2

This adds marionette and related dependencies to the in-tree virtualenv, and updates the remote xpcshell targets to use them.  This also updates remotexpcshelltests.py and runtestsb2g.py to expect devicemanager and related classes to belong to the mozdevice package.

> @@ +297,5 @@
> > +xpcshell-tests-b2g:
> > +	@if [ "${B2G_HOME}" = "" ]; then \
> > +		echo "Please set the B2G_HOME variable"; exit 1; \
> > +	elif [ ! -f "${ADB_PATH}" ]; then \
> > +		echo "Please set the ADB_PATH variable"; exit 1; \
> 
> This seems redundant given that you're setting ADB_PATH above. Can't the 
> harness just look for adb in $PATH? Seems like that'd be nicer than this 
> setup.

It isn't redundant since adb isn't guaranteed to be on the path.  If it is, great, if not, we allow someone to pass the path directly to the script.  The test.sh script that will invoke this from the B2G directory will pass this environment variable.
Attachment #668177 - Attachment is obsolete: true
Attachment #673959 - Flags: review?(ted)
(Assignee)

Comment 18

6 years ago
Pushed to try, just in case:  https://tbpl.mozilla.org/?tree=Try&rev=fbd5bc33f77d.

We're not running xpcshell test on Fennec, so this won't have any impact there.
OS: Windows 2000 → Gonk

Comment 19

6 years ago
Try run for fbd5bc33f77d is complete.
Detailed breakdown of the results available here:
    https://tbpl.mozilla.org/?tree=Try&rev=fbd5bc33f77d
Results (out of 16 total builds):
    success: 16
Builds (or logs if builds failed) available at:
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/jgriffin@mozilla.com-fbd5bc33f77d
(Assignee)

Comment 20

6 years ago
Try run green, review ping to :ted
Comment on attachment 673959 [details] [diff] [review]
Add make target for running B2G xpcshell tests locally, v0.2

Review of attachment 673959 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/testsuite-targets.mk
@@ +286,5 @@
> +	  --no-clean \
> +	  --objdir=$(DEPTH) \
> +	  $$EXTRA_XPCSHELL_ARGS \
> +	  --b2gpath=${B2G_HOME} \
> +	  $(TEST_PATH_ARG) $(EXTRA_TEST_ARGS)

I can't wait till we can move this all into mach and get rid of these huge horrible makefile blocks.

@@ +293,5 @@
> +xpcshell-tests-b2g:
> +	@if [ "${B2G_HOME}" = "" ]; then \
> +		echo "Please set the B2G_HOME variable"; exit 1; \
> +	elif [ ! -f "${ADB_PATH}" ]; then \
> +		echo "Please set the ADB_PATH variable"; exit 1; \

I still think this should probably just be tested in runtestsb2g.py, but I'm not about to r- you over that.
Attachment #673959 - Flags: review?(ted) → review+
https://hg.mozilla.org/mozilla-central/rev/c0d55648fbf5
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Flags: in-testsuite+
Resolution: --- → FIXED
https://hg.mozilla.org/releases/mozilla-aurora/rev/f1780c4b50a5
status-firefox18: --- → fixed
status-firefox19: --- → fixed
(Assignee)

Updated

6 years ago
Whiteboard: [automation-needed-in-aurora]
(Assignee)

Updated

6 years ago
Whiteboard: [automation-needed-in-aurora]
You need to log in before you can comment on or make changes to this bug.