Last Comment Bug 668351 - add make targets to run netwerk xpcshell tests on Android
: add make targets to run netwerk xpcshell tests on Android
Status: RESOLVED FIXED
[mobile-testing][xpcshell]
:
Product: Core
Classification: Components
Component: XPConnect (show other bugs)
: Trunk
: All Android
: -- normal (vote)
: mozilla9
Assigned To: Geoff Brown [:gbrown]
:
Mentors:
Depends on: 668349
Blocks:
  Show dependency treegraph
 
Reported: 2011-06-29 14:23 PDT by Geoff Brown [:gbrown]
Modified: 2011-08-22 04:47 PDT (History)
4 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
patch to add make targets for remote xpcshell tests (4.79 KB, patch)
2011-07-26 15:33 PDT, Geoff Brown [:gbrown]
no flags Details | Diff | Review
patch to add make targets for remote xpcshell tests (4.73 KB, patch)
2011-08-09 09:19 PDT, Geoff Brown [:gbrown]
jmaher: review+
Details | Diff | Review
patch to add make targets for remote xpcshell tests (6.74 KB, patch)
2011-08-11 22:23 PDT, Geoff Brown [:gbrown]
no flags Details | Diff | Review
patch to add make targets for remote xpcshell tests (6.70 KB, patch)
2011-08-11 22:42 PDT, Geoff Brown [:gbrown]
no flags Details | Diff | Review
patch to add make targets for remote xpcshell tests (6.70 KB, patch)
2011-08-17 11:34 PDT, Geoff Brown [:gbrown]
gbrown: review+
Details | Diff | Review

Description Geoff Brown [:gbrown] 2011-06-29 14:23:00 PDT
On desktop, make -C netwerk/tests/ xpcshell-tests runs a suite of tests via runxpcshelltests.py. A similar, convenient facility should be available for running xpcshell tests on Android devices.
Comment 1 Geoff Brown [:gbrown] 2011-07-26 15:33:44 PDT
Created attachment 548612 [details] [diff] [review]
patch to add make targets for remote xpcshell tests

I tested this with:

cd src/mozilla-central/objdir-droid
make -j9 -s
make package
make -C netwerk/test/ xpcshell-tests-remote
make SOLO_FILE=test_simple.js -C netwerk/test/ check-one-remote

and had no problem.


I also tried:

cd src/mozilla-central/objdir-droid
make -j9 -s
make package
make xpcshell-tests-remote

This executed many tests successfully but eventually one of the tests hung. Do we need to handle remote testing of the global set of xpcshell tests?
Comment 2 Brad Lassey [:blassey] (use needinfo?) 2011-07-27 08:49:30 PDT
(In reply to comment #1)
> This executed many tests successfully but eventually one of the tests hung.
> Do we need to handle remote testing of the global set of xpcshell tests?

Let's just worry about the network tests here and file a follow up bug to get the manifests for other tests properly annotated. Also, be sure to file a bug for any test that gets marked skip-if or fail-if.
Comment 3 Joel Maher (:jmaher) 2011-07-27 12:05:16 PDT
Comment on attachment 548612 [details] [diff] [review]
patch to add make targets for remote xpcshell tests

Review of attachment 548612 [details] [diff] [review]:
-----------------------------------------------------------------

why are we adding targets in rules.mk and testsuite-targets.mk?  It would be nicer to just put stuff in testsuite-targets.mk.  It seems like we have 3 copies of this command we are putting in the .mk files.
Comment 4 Geoff Brown [:gbrown] 2011-07-27 14:44:12 PDT
(In reply to comment #3)
> why are we adding targets in rules.mk and testsuite-targets.mk?  It would be
> nicer to just put stuff in testsuite-targets.mk.  It seems like we have 3
> copies of this command we are putting in the .mk files.

I am mirroring the existing make targets. Both rules.mk and testsuite-targets.mk have "xpcshell-tests" targets with comments on each pointing to the other:

rules.mk: # See also testsuite-targets.mk 'xpcshell-tests' target for global execution.

testsuite-targets.mk: # See also config/rules.mk 'xpcshell-tests' target for local execution.

I assume there are "historical reasons" for this arrangement, I don't want to change the "xpcshell-tests" targets, and I want to provide "-remote" equivalents to the "xpcshell-tests" targets. 

I am open to alternatives, but not sure of what to do...
Comment 5 Geoff Brown [:gbrown] 2011-08-09 09:18:36 PDT
The split between rules.mk and testsuite-targets.mk is more clear to me now. There are slightly different invocations of the xpcshell-tests target:

1. The "global" invocation, used to run the full set of 1200+ tests, spread across many directories but all referenced by testing/xpcshell/xpcshell.ini. Use "make xpcshell-tests" in $objdir for this. In this case, xpcshelltests.py is called with --manifest=... <master xpcshell.ini>

2. The "local" invocation, used to run a single directory of tests. Use "make -C <directory> xpcshell-tests" in $objdir for this. In this case, xpcshelltests.py is called with a list of directories for arguments, and it searches those directories for manifests.

The remote versions in this patch mirror the existing targets:

make xpcshell-tests
make xpcshell-tests-remote

make -C netwerk/test xpcshell-tests
make -C netwerk/test xpcshell-tests-remote

make SOLO_FILE=test_simple.js -C netwerk/test check-one
make SOLO_FILE=test_simple.js -C netwerk/test check-one-remote
Comment 6 Geoff Brown [:gbrown] 2011-08-09 09:19:53 PDT
Created attachment 551790 [details] [diff] [review]
patch to add make targets for remote xpcshell tests

Minor update to avoid the symbolic link -- not needed or used for xpcshell tests.
Comment 8 Brad Lassey [:blassey] (use needinfo?) 2011-08-10 15:22:53 PDT
backed out due to orange:
http://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=d9456378c12d

config/rules.mk needs to match js/src/config/rules.mk. The simple follow up didn't solve the problem because the rules.mk in this push references packager.mk which does not exist in js/src
Comment 9 Geoff Brown [:gbrown] 2011-08-11 22:10:54 PDT
(In reply to Brad Lassey [:blassey][blassey@mozilla.com] from comment #8)
> config/rules.mk needs to match js/src/config/rules.mk. The simple follow up
> didn't solve the problem because the rules.mk in this push references
> packager.mk which does not exist in js/src

I tried a few things and came to the conclusion that including package-name.mk was just a bad idea. The package name was needed so that the --apk argument could be passed to remotexpcshelltests.py, and then that .apk could be pushed to the device. I have made changes now so that rules.mk does not include package-name.mk and does not pass the --apk argument; remotexpcshelltests.py is updated (in bug 668349) to search for an apk in $objdir, if no --apk argument is provided.
Comment 10 Brad Lassey [:blassey] (use needinfo?) 2011-08-11 22:18:39 PDT
why can't you use $(ANDROID_PACKAGE_NAME) without including packager.mk
Comment 11 Geoff Brown [:gbrown] 2011-08-11 22:23:28 PDT
Created attachment 552595 [details] [diff] [review]
patch to add make targets for remote xpcshell tests

Updated to sync config/ and js/src/config/ versions of rules.mk. --apk argument no longer specified.
Comment 12 Geoff Brown [:gbrown] 2011-08-11 22:27:05 PDT
(In reply to Brad Lassey [:blassey][blassey@mozilla.com] from comment #10)
> why can't you use $(ANDROID_PACKAGE_NAME) without including packager.mk

$(ANDROID_PACKAGE_NAME) is something like "org.mozilla.fennec"; we need the name of the APK file, like "fennec-8.0a1.en-US.eabi-arm.apk".
Comment 13 Geoff Brown [:gbrown] 2011-08-11 22:42:18 PDT
Created attachment 552600 [details] [diff] [review]
patch to add make targets for remote xpcshell tests
Comment 14 Geoff Brown [:gbrown] 2011-08-17 11:34:16 PDT
Created attachment 553854 [details] [diff] [review]
patch to add make targets for remote xpcshell tests

updated for bitrot
Comment 15 Geoff Brown [:gbrown] 2011-08-17 12:09:25 PDT
Comment on attachment 553854 [details] [diff] [review]
patch to add make targets for remote xpcshell tests

r=jmaher
Comment 16 Geoff Brown [:gbrown] 2011-08-18 10:32:22 PDT
Pushed to try -- looks good to me.

http://hg.mozilla.org/try/pushloghtml?changeset=ff5c3f0b0f6a
http://tbpl.mozilla.org/?tree=Try&rev=ff5c3f0b0f6a
Comment 18 Ed Morley [:emorley] 2011-08-21 14:34:01 PDT
The 5 patches in comment 17 have been pushed together to try:
http://tbpl.allizom.org/?tree=Try&usebuildbot=1&rev=95446695ead0

Presuming green, I'll push to inbound shortly afterwards :-)
Comment 20 Kyle Huey [:khuey] (khuey@mozilla.com) 2011-08-22 04:47:37 PDT
http://hg.mozilla.org/mozilla-central/rev/ea0fb4946b29

Note You need to log in before you can comment on or make changes to this bug.