828069 - Turn on xpcshell tests for Android 2.2

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Geoff Brown [:gbrown]]

We can turn on xpcshell tests on tegras (Android 2.2), for a very limited subset of tests. Running Android xpcshell tests in continuous integration will help prevent breakage of the Android xpcshell harness and clear the way for running additional Android xpcshell tests.

(xpcshell tests on panda / Android 4.0 fail, due to bug 827554.)

The very-limited, initial set of known-to-pass tests are listed in a separate manifest, testing/xpcshell/xpcshell_android.ini.

Running locally on a tegra over sut, this test takes about 6 minutes to run, most of which is setup (copying files to the tegra).

A key difference between xpcshell tests and existing remote tests like mochitests and reftests is that the Android xpcshell tests will copy files to both /mnt/sdcard/tests and /data/local. All /data/local files are copied to /data/local/xpcb, a directory created by the test harness via sut.


Running locally, remote xpcshell tests are normally run as "make xpcshell-tests-remote", from the Android objdir, which executes:

<path to objdir>/_virtualenv/bin/python -u <path to src>/testing/xpcshell/remotexpcshelltests.py \
	    --manifest=./_tests/xpcshell/xpcshell_android.ini \
	    --build-info-json=./mozinfo.json \
	    --no-logfiles \
	    --testing-modules-dir=<path to objdir>/_tests/modules \
	    --dm_trans=sut \
	    --deviceIP=<device IP> \
	    --objdir=. \
	    --symbols-path=./dist/crashreporter-symbols

Comment 1

[Justin Wood (:Callek)]

What would help me make this a realty is knowing _that_ it works and a basic "steps to make it work" for doing this from a packaged tests, rather than an android build objdir

e.g.

http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android/1357669276/

taking:
	fennec-21.0a1.en-US.android-arm.apk
	fennec-21.0a1.en-US.android-arm.tests.zip

And laying out steps to do it with that.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

I did this:
wget fennec.apk
wget fennec.tests.zip
unzip fennec.tests.zip
mkdir fennec
mv fennec.apk fennec
cd fennec
unzip fennec.apk
cd ..
cd xpcshell
python remotexpcshelltests.py --app=org.mozilla.fennec --deviceIP=192.168.1.69 --manifest=tests/xpcshell_android.ini --no-logfiles --build-info-json=mozinfo.json --testing-modules-dir=tests/modules --local-lib-dir=../fennec/ --apk=../fennec/fennec-21.0a1.en-US.android-arm.apk 

we already do the first 3 steps, I assume the others can be handled without much trouble.

Comment 3

[Geoff Brown [:gbrown]]

Thanks Joel!

We should also be able to add --symbols-path=<path-to-crashreporter-symbols>, optionally.

The expected output, except perhaps for some differences in paths, is:

Pushing libmozalloc.so..
Pushing libxul.so..
This is a big file, it could take a while.
Pushing libplc4.so..
Pushing libnss3.so..
Pushing libnssutil3.so..
Pushing libplds4.so..
Pushing libxpcom.so..
Pushing libsmime3.so..
Pushing libfreebl3.so..
Pushing libnssckbi.so..
Pushing libssl3.so..
Pushing libsoftokn3.so..
Pushing libmozsqlite3.so..
Pushing libnspr4.so..
pushing directory: tests/modules to /mnt/sdcard/tests/xpcshell/m
pushing /media/extra/xpcstage/xpcshell/tests
pushing directory: /media/extra/xpcstage/xpcshell/tests to /mnt/sdcard/tests/xpcshell
TEST-INFO | /media/extra/xpcstage/xpcshell/tests/gfx/tests/unit/test_nsIScriptableRegion.js | running test ...
TEST-PASS | /media/extra/xpcstage/xpcshell/tests/gfx/tests/unit/test_nsIScriptableRegion.js | test passed (time: 752.509ms)
INFO | Result summary:
INFO | Passed: 1
INFO | Failed: 0
INFO | Todo: 0

Comment 4

[Justin Wood (:Callek)]

Attachment: [m-c] fixup remotexpcshelltests.py

So this patch gets things working on the foopies for me. I added the python version check; even though the tree in general does the check in configure, we don't do configure on test machines.

I'm ok with dropping the py ver check was just a "hey since I learned its not useful to use anything less, and it fails with an odd message, just make it obvious why it fails" type reasoning.

For clarity the py2.7 req is the subprocess.check_output call.

I was able to run this on a foopy with the following command by hand:
[cltbld@foopy109 tests]$ python2.7 xpcshell/remotexpcshelltests.py --deviceIP 10.250.51.249 --xre-path ../hostutils/xre --manifest=xpcshell/tests/xpcshell_android.ini  --symbols-path=http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android-armv6/1364306903/fennec-22.0a1.en-US.android-arm-armv6.crashreporter-symbols.zip --build-info-json=xpcshell/mozinfo.json  --testing-modules-dir=xpcshell/tests/modules --local-lib-dir=../fennec --apk=../fennec-22.0a1.en-US.android-arm-armv6.apk --no-logfiles

Which resulted in a single test being run (and passing)

Comment 5

[Geoff Brown [:gbrown]]

Comment on attachment 729922 [details] [diff] [review]
[m-c] fixup remotexpcshelltests.py

Review of attachment 729922 [details] [diff] [review]:
-----------------------------------------------------------------

I was a little troubled by the py version check at first, but I think your reasoning is sound -- I'm inclined to keep it in.

Really glad this works -- thanks!

Comment 6

[William Lachance (:wlach)]

Comment on attachment 729922 [details] [diff] [review]
[m-c] fixup remotexpcshelltests.py

># HG changeset patch
># Parent c9bf19d37fe084efd285fb985784a0d92a02f978
>diff --git a/testing/xpcshell/remotexpcshelltests.py b/testing/xpcshell/remotexpcshelltests.py
>--- a/testing/xpcshell/remotexpcshelltests.py
>+++ b/testing/xpcshell/remotexpcshelltests.py
>@@ -152,7 +152,11 @@ class XPCShellRemote(xpcshell.XPCShellTe
> 
>     def setupTestDir(self):
>         print 'pushing %s' % self.xpcDir
>-        self.device.pushDir(self.xpcDir, self.remoteScriptsDir, retryLimit=10)
>+        try:
>+            self.device.pushDir(self.xpcDir, self.remoteScriptsDir, retryLimit=10)
>+        except TypeError:
>+            # Foopies have an older mozdevice ver without retryLimit
>+            self.device.pushDir(self.xpcDir, self.remoteScriptsDir)

I'm a bit confused about this. Shouldn't we be using the copy of mozdevice that's in tree under `testing/mozbase`? What version of mozdevice that the foopy is using shouldn't affect anything here. Do we need to add it onto the path somehow, as we do here for example: http://mxr.mozilla.org/mozilla-central/source/testing/xpcshell/Makefile.in#58 (not sure if that's the right place, it just stuck out at me).

Comment 7

[Justin Wood (:Callek)]

Landed to inbound:

https://hg.mozilla.org/integration/mozilla-inbound/rev/123017ccf036

Comment 8

[Justin Wood (:Callek)]

(In reply to William Lachance (:wlach) from comment #6)
> I'm a bit confused about this. Shouldn't we be using the copy of mozdevice
> that's in tree under `testing/mozbase`? What version of mozdevice that the
> foopy is using shouldn't affect anything here. Do we need to add it onto the
> path somehow, as we do here for example:
> http://mxr.mozilla.org/mozilla-central/source/testing/xpcshell/Makefile.
> in#58 (not sure if that's the right place, it just stuck out at me).

The foopies current use the mozdevice installed on it for all its tests. We don't have an objdir nor a configure run in order to use anything like the venv python or the include path like in that Makefile.

I admit it would be best to migrate this test code to use the in-tree mozdevice though, but that (imo) will be a future improvement.

This is setup in my patch to allow us to use the newer mozdevice code as soon as it's available, either by fixing us to use the in-tree mozdevice, or when we update the mozdevice on foopies.

Comment 9

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/123017ccf036

Comment 10

[Justin Wood (:Callek)]

Attachment: [custom] v0.9

f? to joel for the mere fact of "does this command look right"

This seems to do it with local testing/etc. and a dump master output.

Comment 11

[Justin Wood (:Callek)]

Attachment: [configs] v0.9

this is feedback only because I need to figure out how to limit this to cedar.

Per earlier in this bug there are currently issues preventing it from working on pandas, and this patch as it stands rolls it out to all tegras to every branch. Not what we want (since it needed a harness patch that I just landed to m-c today)

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 730492 [details] [diff] [review]
[custom] v0.9

Review of attachment 730492 [details] [diff] [review]:
-----------------------------------------------------------------

no red flags here!

Comment 13

[Justin Wood (:Callek)]

Attachment: [configs] v1.0

Limited to cedar, and now with comments!

Comment 14

[Aki Sasaki (not active)]

Comment on attachment 730867 [details] [diff] [review]
[configs] v1.0

The trains comment will need to be marked MERGE DAY when this rolls out to m-c.

Comment 15

[Justin Wood (:Callek)]

http://hg.mozilla.org/build/buildbot-configs/rev/d79ec2834f7b

http://hg.mozilla.org/build/buildbotcustom/rev/9edbe1e77f80

Should deploy later today.

Comment 16

[Kim Moir [:kmoir] ET]

In the mobile meeting today, it was requested to enable xpcshell tests on other branches.

Comment 17

[Justin Wood (:Callek)]

Attachment: [configs] more places

As per this weeks mobile meeting, they want this on for the other branches now as well.

This is green on cedar.

Comment 18

[Justin Wood (:Callek)]

(In reply to Justin Wood (:Callek) from comment #17)
> Created attachment 733700 [details] [diff] [review]
> [configs] more places

Forgot to mention, extra changes were pep8 violation fixes for stuff I caused :(

Comment 19

[Aki Sasaki (not active)]

Comment on attachment 733700 [details] [diff] [review]
[configs] more places

r=me if you add a MERGE DAY comment.

Comment 20

[Justin Wood (:Callek)]

Comment on attachment 733700 [details] [diff] [review]
[configs] more places

Review of attachment 733700 [details] [diff] [review]:
-----------------------------------------------------------------

http://hg.mozilla.org/build/buildbot-configs/rev/05d6b0ef6a50

Complete with MERGE DAY comment

Comment 21

[Rail Aliiev [:rail]]

in production