717042 - Robocop: "OK" displayed even when failures detected

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[Geoff Brown [:gbrown]]

Robocop's Asserter does not call the Robotium/junit assert framework when an assertion fails, so failed Robocop assertions go unnoticed by Robotium, and Robotium cheerfully reports "OK" for all tests. I worry that this could cause confusion. Consider:

...
    org.mozilla.fennec_mozdev.tests.testNewTab:.
    Test results for InstrumentationTestRunner=.
    Time: 13.223
     
    OK (1 test)
     
     
    INFO | automation.py | Application pid: 0
    0 INFO SimpleTest START
    1 INFO TEST-START | testNewTab
    2 INFO TEST-PASS | testNewTab | Awesomebar url is fine - http://mochi.test:8888/tests/robocop/robocop_blank_01.html should equal http://mochi.test:8888/tests/robocop/robocop_blank_01.html
    3 INFO TEST-PASS | testNewTab | Number of tabs has increased - 2 should equal 2
    4 INFO TEST-UNEXPECTED-FAIL | testNewTab | URL is still fine - got http://mochi.test:8888/tests/robocop/robocop_blank_01.html, expected http://mochi.test:8888/tests/robocop/robocop_blank_02.html
    5 INFO TEST-PASS | testNewTab | Number of tabs has increased - 3 should equal 3
    6 INFO TEST-END | testNewTab | finished in 11572ms
    7 INFO TEST-START | Shutdown
    8 INFO Passed: 3
    9 INFO Failed: 1
    10 INFO Todo: 0
    11 INFO SimpleTest FINISHED
    INFO | automation.py | Application ran for: 0:00:29.013596
    INFO | automation.py | Reading PID log: /tmp/tmpoG7hW6pidlog
    WARNING | automationutils.processLeakLog() | refcount logging is off, so leaks can't be detected!
     
    INFO | runtests.py | Running tests: end.

Comment 1

[Geoff Brown [:gbrown]]

By calling the junit assert/fail, we get this instead:

...

org.mozilla.fennec_mozdev.tests.testNewTab:
Failure in testNewTab:
junit.framework.AssertionFailedError: 4 INFO TEST-UNEXPECTED-FAIL | testNewTab | URL is still fine - got http://mochi.test:8888/tests/robocop/robocop_blank_01.html, expected http://mochi.test:8888/tests/robocop/robocop_blank_02.html
	at org.mozilla.fennec_mozdev.FennecNativeAssert._logMochitestResult(FennecNativeAssert.java:192)
	at org.mozilla.fennec_mozdev.FennecNativeAssert.ok(FennecNativeAssert.java:227)
	at org.mozilla.fennec_mozdev.FennecNativeAssert.is(FennecNativeAssert.java:237)
	at org.mozilla.fennec_mozdev.tests.testNewTab.testNewTab(testNewTab.java:35)
	at java.lang.reflect.Method.invokeNative(Native Method)
	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
	at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:186)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:169)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:154)
	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:525)
	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1547)

Test results for InstrumentationTestRunner=.F
Time: 11.815

FAILURES!!!
Tests run: 1,  Failures: 1,  Errors: 0


INFO | automation.py | Application pid: 0
0 INFO SimpleTest START
1 INFO TEST-START | testNewTab
2 INFO TEST-PASS | testNewTab | Awesomebar url is fine - http://mochi.test:8888/tests/robocop/robocop_blank_01.html should equal http://mochi.test:8888/tests/robocop/robocop_blank_01.html
3 INFO TEST-PASS | testNewTab | Number of tabs has increased - 2 should equal 2
4 INFO TEST-UNEXPECTED-FAIL | testNewTab | URL is still fine - got http://mochi.test:8888/tests/robocop/robocop_blank_01.html, expected http://mochi.test:8888/tests/robocop/robocop_blank_02.html
5 INFO TEST-END | testNewTab | finished in 10618ms
6 INFO TEST-START | Shutdown
7 INFO Passed: 2
8 INFO Failed: 1
9 INFO Todo: 0
10 INFO SimpleTest FINISHED
INFO | automation.py | Application ran for: 0:00:28.054138
INFO | automation.py | Reading PID log: /tmp/tmpHyopC4pidlog
WARNING | automationutils.processLeakLog() | refcount logging is off, so leaks can't be detected!

INFO | runtests.py | Running tests: end.

Comment 2

[Geoff Brown [:gbrown]]

Note these differences:
 - stack trace of failure
 - "FAILURES" instead of "OK"
 - the test case ends immediately -- test code following the assertion is not executed

Comment 3

[Geoff Brown [:gbrown]]

Attachment: simple change to call junit fail when failure detected

I couldn't resist cleaning up some dead code at the same time; I'm happy to split this out if desired.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 587478 [details] [diff] [review]
simple change to call junit fail when failure detected

Review of attachment 587478 [details] [diff] [review]:
-----------------------------------------------------------------

thanks for the cleanup.  I am sure somebody somewhere who is big on junit or robotium proper would ask for a bunch of other things, but this is designed for our needs so r=me!

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

inbound:
https://hg.mozilla.org/integration/mozilla-inbound/rev/74b7b6b595a1

Comment 6

[Matt Brubeck (:mbrubeck)]

https://hg.mozilla.org/mozilla-central/rev/74b7b6b595a1