856614 - Add a unit testing framework for fennec code

Status: RESOLVED DUPLICATE of bug 903528

(Testing :: Mochitest, defect)

Description

[Kartikaya Gupta (email:[e-mail])]

It seems to me like a lot of our testing woes result from trying to use robocop to do things that would be better tested with a unit test framework. Robocop is good as a functional testing framework (i.e. simulate user inputs and test UI output) but sometimes we want to do more of a white-box testing of individual pieces of code.

I think if we add a unit test framework that allows in-process testing of code (i.e. no need for reflection) and that can optionally be built into our dev builds (e.g. via a mozconfig option) and run from a menu item, that would help a lot. When adding a new feature we usually test it on a device anyway, and if we can quickly run a set of regression unit tests just by selecting a menu item I think it could help prevent a lot of regressions.

Comment 1

[Geoff Brown [:gbrown]]

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #0)
> It seems to me like a lot of our testing woes result from trying to use robocop 
> to do things that would be better tested with a unit test framework.

There was a discussion about white-box testing GeckoInputConnection a while back and my first thought was that we could use reflection in Robocop to do that, but then when I started to write it, I realized that the reflection approach gets complicated and unwieldy pretty quickly; I'm with you from that perspective. 

> I think if we add a unit test framework that allows in-process testing of
> code (i.e. no need for reflection) 

Would all the test code be in Java? Are there particular classes/APIs you have in mind that need testing? I'm just looking for examples to help us visualize how this would work. 

The menu approach sounds convenient for developers, but keep in mind we would probably want to run this in continuous integration also.

Comment 2

[Kartikaya Gupta (email:[e-mail])]

(In reply to Geoff Brown [:gbrown] from comment #1)
> Would all the test code be in Java? Are there particular classes/APIs you
> have in mind that need testing? I'm just looking for examples to help us
> visualize how this would work. 

Yeah, I'm imagining this would be in Java. TBH I haven't thought of specific examples that would benefit from this; it just came up while I was talking to margaret last week. I think she mentioned that a lot of the content provider tests would be better suited to a unit test framework since they run in a "headless" kind of environment anyway.

> The menu approach sounds convenient for developers, but keep in mind we
> would probably want to run this in continuous integration also.

Yeah, I imagine it wouldn't be too hard to have multiple entry points to run the tests. We could probably trigger it via an intent for CI purposes.

Comment 3

[:Margaret Leibovic]

(In reply to Kartikaya Gupta (email:[e-mail]) from comment #2)
> (In reply to Geoff Brown [:gbrown] from comment #1)
> > Would all the test code be in Java? Are there particular classes/APIs you
> > have in mind that need testing? I'm just looking for examples to help us
> > visualize how this would work. 
> 
> Yeah, I'm imagining this would be in Java. TBH I haven't thought of specific
> examples that would benefit from this; it just came up while I was talking
> to margaret last week. I think she mentioned that a lot of the content
> provider tests would be better suited to a unit test framework since they
> run in a "headless" kind of environment anyway.

When I saw this bug, I looked into ContentProviderTest, but I found that it does still need an Activity, and the reflection situation isn't quite as bad as I remembered. I'm cc'ing Lucas in case he has any ideas about what would have made writing those tests easier.

The difficult thing about unit testing is that if we create another Java-only framework, we'll still have difficulty testing the JS side of things.

Comment 4

[Richard Newman [:rnewman]]

We have a decent sized suite of tests in the background services repo. We use JUnit 4 to test pure-Java stuff, and the Android JUnit 3 InstrumentationTestCase system to run unit tests on a live device. Both of these sets can be run through Maven or through Eclipse.

The former (and, when we get a virtual machine setup, the latter) runs on ci.mozilla.org.

It should be relatively straightforward to get something similar working for Fennec, and then to integrate it with TBPL, at which point we can make sure our background services tests run, too.

See Bug 709353.

Comment 5

[Richard Newman [:rnewman]]

The workflow for running the Android instrumentation tests, by the way, is either:

* Plug phone into laptop
* Hit button in Eclipse, or type `mvn integration-test`.

The latter produces a suite of reports, can print JUnit format, etc. etc.

I don't see much need to make our test suite available from UI; it can be triggered remotely.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

I was working with :gbrown yesterday to use specialpowers to send data from a content file in .js and using the androidbridge to send data to java.  This seems to work pretty well.

I have hesitations about a new harness- this is why I would like to look at specific test scenarios and see if we can make mochitests or robocop tests from them without hacking too much.

This is a good topic and subject to bring up- we should always evaluate our tools to see if they meet our needs.  Maybe it is time for a new harness, maybe we need to wait a bit.

Comment 7

[Mark Finkle (:mfinkle) (use needinfo?)]

(In reply to Joel Maher (:jmaher) from comment #6)

> I have hesitations about a new harness- this is why I would like to look at
> specific test scenarios and see if we can make mochitests or robocop tests
> from them without hacking too much.

I think we want a way to test Java code, without needing to load Gecko.

Comment 8

[Richard Newman [:rnewman]]

(In reply to Joel Maher (:jmaher) from comment #6)

> I was working with :gbrown yesterday to use specialpowers to send data from
> a content file in .js and using the androidbridge to send data to java. 
> This seems to work pretty well.

See also Nick's work in Bug 870908.


(In reply to Mark Finkle (:mfinkle) from comment #7)

> I think we want a way to test Java code, without needing to load Gecko.

Most definitely. Probably a majority of our Java code doesn't need or want to wait hundreds of msec for Gecko just to use JNI or messages to poke at Java objects.

We have 261 tests in android-sync that run headless, and 276 tests that don't even need *Android*, let alone a UI. All of Fennec's DB tests, unit tests, etc. should be using similar mechanisms.

Comment 9

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Forward duping as unit testing landed (sans TBPL support) with the Services' test suite.