668351 - add make targets to run netwerk xpcshell tests on Android

Status: RESOLVED FIXED

(Core :: XPConnect, defect)

Description

[Geoff Brown [:gbrown] (pto Apr 11-18)]

On desktop, make -C netwerk/tests/ xpcshell-tests runs a suite of tests via runxpcshelltests.py. A similar, convenient facility should be available for running xpcshell tests on Android devices.

Comment 1

[Geoff Brown [:gbrown] (pto Apr 11-18)]

Attachment: patch to add make targets for remote xpcshell tests

I tested this with:

cd src/mozilla-central/objdir-droid
make -j9 -s
make package
make -C netwerk/test/ xpcshell-tests-remote
make SOLO_FILE=test_simple.js -C netwerk/test/ check-one-remote

and had no problem.


I also tried:

cd src/mozilla-central/objdir-droid
make -j9 -s
make package
make xpcshell-tests-remote

This executed many tests successfully but eventually one of the tests hung. Do we need to handle remote testing of the global set of xpcshell tests?

Comment 2

[Brad Lassey [:blassey] (use needinfo?)]

(In reply to comment #1)
> This executed many tests successfully but eventually one of the tests hung.
> Do we need to handle remote testing of the global set of xpcshell tests?

Let's just worry about the network tests here and file a follow up bug to get the manifests for other tests properly annotated. Also, be sure to file a bug for any test that gets marked skip-if or fail-if.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 548612 [details] [diff] [review]
patch to add make targets for remote xpcshell tests

Review of attachment 548612 [details] [diff] [review]:
-----------------------------------------------------------------

why are we adding targets in rules.mk and testsuite-targets.mk?  It would be nicer to just put stuff in testsuite-targets.mk.  It seems like we have 3 copies of this command we are putting in the .mk files.

Comment 4

[Geoff Brown [:gbrown] (pto Apr 11-18)]

(In reply to comment #3)
> why are we adding targets in rules.mk and testsuite-targets.mk?  It would be
> nicer to just put stuff in testsuite-targets.mk.  It seems like we have 3
> copies of this command we are putting in the .mk files.

I am mirroring the existing make targets. Both rules.mk and testsuite-targets.mk have "xpcshell-tests" targets with comments on each pointing to the other:

rules.mk: # See also testsuite-targets.mk 'xpcshell-tests' target for global execution.

testsuite-targets.mk: # See also config/rules.mk 'xpcshell-tests' target for local execution.

I assume there are "historical reasons" for this arrangement, I don't want to change the "xpcshell-tests" targets, and I want to provide "-remote" equivalents to the "xpcshell-tests" targets. 

I am open to alternatives, but not sure of what to do...

Comment 5

[Geoff Brown [:gbrown] (pto Apr 11-18)]

The split between rules.mk and testsuite-targets.mk is more clear to me now. There are slightly different invocations of the xpcshell-tests target:

1. The "global" invocation, used to run the full set of 1200+ tests, spread across many directories but all referenced by testing/xpcshell/xpcshell.ini. Use "make xpcshell-tests" in $objdir for this. In this case, xpcshelltests.py is called with --manifest=... <master xpcshell.ini>

2. The "local" invocation, used to run a single directory of tests. Use "make -C <directory> xpcshell-tests" in $objdir for this. In this case, xpcshelltests.py is called with a list of directories for arguments, and it searches those directories for manifests.

The remote versions in this patch mirror the existing targets:

make xpcshell-tests
make xpcshell-tests-remote

make -C netwerk/test xpcshell-tests
make -C netwerk/test xpcshell-tests-remote

make SOLO_FILE=test_simple.js -C netwerk/test check-one
make SOLO_FILE=test_simple.js -C netwerk/test check-one-remote

Comment 6

[Geoff Brown [:gbrown] (pto Apr 11-18)]

Attachment: patch to add make targets for remote xpcshell tests

Minor update to avoid the symbolic link -- not needed or used for xpcshell tests.

Comment 7

[Geoff Brown [:gbrown] (pto Apr 11-18)]

http://hg.mozilla.org/integration/mozilla-inbound/rev/f2e8ee893fe4

Comment 8

[Brad Lassey [:blassey] (use needinfo?)]

backed out due to orange:
http://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=d9456378c12d

config/rules.mk needs to match js/src/config/rules.mk. The simple follow up didn't solve the problem because the rules.mk in this push references packager.mk which does not exist in js/src

Comment 9

[Geoff Brown [:gbrown] (pto Apr 11-18)]

(In reply to Brad Lassey [:blassey][[e-mail]] from comment #8)
> config/rules.mk needs to match js/src/config/rules.mk. The simple follow up
> didn't solve the problem because the rules.mk in this push references
> packager.mk which does not exist in js/src

I tried a few things and came to the conclusion that including package-name.mk was just a bad idea. The package name was needed so that the --apk argument could be passed to remotexpcshelltests.py, and then that .apk could be pushed to the device. I have made changes now so that rules.mk does not include package-name.mk and does not pass the --apk argument; remotexpcshelltests.py is updated (in bug 668349) to search for an apk in $objdir, if no --apk argument is provided.

Comment 10

[Brad Lassey [:blassey] (use needinfo?)]

why can't you use $(ANDROID_PACKAGE_NAME) without including packager.mk

Comment 11

[Geoff Brown [:gbrown] (pto Apr 11-18)]

Attachment: patch to add make targets for remote xpcshell tests

Updated to sync config/ and js/src/config/ versions of rules.mk. --apk argument no longer specified.

Comment 12

[Geoff Brown [:gbrown] (pto Apr 11-18)]

(In reply to Brad Lassey [:blassey][[e-mail]] from comment #10)
> why can't you use $(ANDROID_PACKAGE_NAME) without including packager.mk

$(ANDROID_PACKAGE_NAME) is something like "org.mozilla.fennec"; we need the name of the APK file, like "fennec-8.0a1.en-US.eabi-arm.apk".

Comment 13

[Geoff Brown [:gbrown] (pto Apr 11-18)]

Attachment: patch to add make targets for remote xpcshell tests

Comment 14

[Geoff Brown [:gbrown] (pto Apr 11-18)]

Attachment: patch to add make targets for remote xpcshell tests

updated for bitrot

Comment 15

[Geoff Brown [:gbrown] (pto Apr 11-18)]

Comment on attachment 553854 [details] [diff] [review]
patch to add make targets for remote xpcshell tests

r=jmaher

Comment 16

[Geoff Brown [:gbrown] (pto Apr 11-18)]

Pushed to try -- looks good to me.

http://hg.mozilla.org/try/pushloghtml?changeset=ff5c3f0b0f6a
http://tbpl.mozilla.org/?tree=Try&rev=ff5c3f0b0f6a

Comment 17

[Geoff Brown [:gbrown] (pto Apr 11-18)]

A group of patches is ready for check-in, ideally all together:

https://bug661282.bugzilla.mozilla.org/attachment.cgi?id=548872
https://bug678385.bugzilla.mozilla.org/attachment.cgi?id=554510
https://bug668349.bugzilla.mozilla.org/attachment.cgi?id=553856
https://bug668349.bugzilla.mozilla.org/attachment.cgi?id=553857
https://bug668351.bugzilla.mozilla.org/attachment.cgi?id=553854

Comment 18

[Ed Morley [:emorley]]

The 5 patches in comment 17 have been pushed together to try:
http://tbpl.allizom.org/?tree=Try&usebuildbot=1&rev=95446695ead0

Presuming green, I'll push to inbound shortly afterwards :-)

Comment 19

[Ed Morley [:emorley]]

http://hg.mozilla.org/integration/mozilla-inbound/rev/ea0fb4946b29

Comment 20

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

http://hg.mozilla.org/mozilla-central/rev/ea0fb4946b29