710445 - Add support for packaging tests in a specific directory

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

From bug 701076 comment 30 :

Per f2f discussion with tfair, it will be really helpful to add support for packaging tests in a specific directory.

(instead of having the individual files listed in Makefile.in)

bug 701076 comment 31 :

We might not be able to make this work so smoothly, but we will try to make it simpler.  For what :gkw is looking to do, we think it could work for him to generate a set of .java files and put them in the build tree (mobile/android/base/test/), rebuild and run the tests.  

Some issues to resolve:
 * removing the files hardcoded in the makefile.in
 * allowing for specific files to be run via the runtestsremote.py (just run the new files only)
 * figuring out how all this works with the robocop.ini

Comment 1

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

workaround via IRC:

jmaherXX: gkw: you can just hack the robocop.ini file for now
jmaherXX: gkw: but we detect all tests in the mobile/android/base/test file
jmaherXX: gkw: put your test file in that folder ^
jmaherXX: then edit the robocop.ini in that folder to just have your tests in there

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

What would be the goal for resolving this?  Right now we detect all files in the mobile/android/base/tests folder and roll those into robocop.apk.

If you are generating files, could you add them automatically to the robocop.ini file or create a unique .ini file that you could use?  If so, we could work on either:
* specifying which .ini to use for the test
* using keywords or tags in the ini file for filtering

Comment 3

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

> If you are generating files, could you add them automatically to the
> robocop.ini file or create a unique .ini file that you could use?  If so, we
> could work on either:
> * specifying which .ini to use for the test
> * using keywords or tags in the ini file for filtering

I'd prefer the first option, it sounds more straightforward.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: allow for custom manifest files (1.0)

this patch solves a few problems:
1) allows for --robocop to specify the .ini file used.  This is great as we can now run a set of tests without much hacking.  this also solves a lot of the problems for running mochitests vs talos
2) allow for robocop-path.  This is useful for 'make mochitest-robotium' where we would need to install the robocop.apk
3) allow for test-path.  This allows us to specify a single test inside the manifest file (i.e. robocop.ini).  An example would be --test-path=testLoad

Comment 5

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Comment on attachment 586538 [details] [diff] [review]
allow for custom manifest files (1.0)

Looks good. It'll be nice to have some documentation (probably separate from this) describing how to run your own tests with robocop too.

Comment 6

[Geoff Brown [:gbrown]]

Comment on attachment 586538 [details] [diff] [review]
allow for custom manifest files (1.0)

Review of attachment 586538 [details] [diff] [review]:
-----------------------------------------------------------------

Good stuff! I especially appreciate the test-path support -- that will be very helpful.

What is the plan for robocop.apk installation via SUT?

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

thanks, I was trying to give general support to everything.

robocop.apk will be installed as fennec.apk is installed, during the setup steps (in installApp.py).  I left the adb installation in the script so 'make mochitest-robotium' would run smoother.

Comment 8

[Ed Morley [:emorley]]

Landed (https://hg.mozilla.org/integration/mozilla-inbound/rev/b5686f25b258) but that push was backed out again (https://hg.mozilla.org/integration/mozilla-inbound/rev/879883efec3c) for mochitest/reftest failures on Android & Windows:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=8ec01f6f316f

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

inbound:
https://hg.mozilla.org/integration/mozilla-inbound/rev/aea8ac101dfe

Comment 10

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/aea8ac101dfe