764527 - robocop harness misses some failures

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[Kartikaya Gupta (email:[e-mail])]

Attachment: Catch exceptions in runTest

I was trying to run robocop locally and I saw this failure go by in the logcat, but it wasn't detected by the robocop harness.

06-13 14:35:37.431 I/TestRunner(31067): failed: testBookmark(org.mozilla.fennec_kats.tests.testBookmark)
06-13 14:35:37.431 I/TestRunner(31067): ----- begin exception -----
06-13 14:35:37.441 I/TestRunner(31067):
06-13 14:35:37.441 I/TestRunner(31067): junit.framework.AssertionFailedError: The text: Bookmark is not found!
06-13 14:35:37.441 I/TestRunner(31067):     at junit.framework.Assert.fail(Assert.java:47)
06-13 14:35:37.441 I/TestRunner(31067):     at junit.framework.Assert.assertTrue(Assert.java:20)
06-13 14:35:37.441 I/TestRunner(31067):     at com.jayway.android.robotium.solo.Clicker.clickOnText(Clicker.java:279)
06-13 14:35:37.441 I/TestRunner(31067):     at com.jayway.android.robotium.solo.Solo.clickOnText(Solo.java:823)
06-13 14:35:37.441 I/TestRunner(31067):     at org.mozilla.fennec_kats.tests.testBookmark.toggleBookmark(testBookmark.java:174)
06-13 14:35:37.441 I/TestRunner(31067):     at org.mozilla.fennec_kats.tests.testBookmark.setUpBookmark(testBookmark.java:180)
06-13 14:35:37.441 I/TestRunner(31067):     at org.mozilla.fennec_kats.tests.testBookmark.runMenuTest(testBookmark.java:57)
06-13 14:35:37.441 I/TestRunner(31067):     at org.mozilla.fennec_kats.tests.testBookmark.testBookmark(testBookmark.java:50)
06-13 14:35:37.441 I/TestRunner(31067):     at java.lang.reflect.Method.invokeNative(Native Method)
06-13 14:35:37.441 I/TestRunner(31067):     at java.lang.reflect.Method.invoke(Method.java:491)
06-13 14:35:37.441 I/TestRunner(31067):     at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
06-13 14:35:37.441 I/TestRunner(31067):     at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
06-13 14:35:37.441 I/TestRunner(31067):     at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:186)
06-13 14:35:37.441 I/TestRunner(31067):     at junit.framework.TestCase.runBare(TestCase.java:127)
06-13 14:35:37.441 I/TestRunner(31067):     at junit.framework.TestResult$1.protect(TestResult.java:106)
06-13 14:35:37.441 I/TestRunner(31067):     at junit.framework.TestResult.runProtected(TestResult.java:124)
06-13 14:35:37.441 I/TestRunner(31067):     at junit.framework.TestResult.run(TestResult.java:109)
06-13 14:35:37.441 I/TestRunner(31067):     at junit.framework.TestCase.run(TestCase.java:118)
06-13 14:35:37.441 I/TestRunner(31067):     at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:169)
06-13 14:35:37.441 I/TestRunner(31067):     at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:154)
06-13 14:35:37.441 I/TestRunner(31067):     at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:525)
06-13 14:35:37.441 I/TestRunner(31067):     at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1547)
06-13 14:35:37.441 I/TestRunner(31067): ----- end exception -----
06-13 14:35:37.441 I/TestRunner(31067): finished: testBookmark(org.mozilla.fennec_kats.tests.testBookmark)
06-13 14:35:37.441 I/stdout  (31058): org.mozilla.fennec_kats.tests.testBookmark:
06-13 14:35:37.441 I/stdout  (31058): Failure in testBookmark:
06-13 14:35:37.441 I/stdout  (31058): junit.framework.AssertionFailedError: The text: Bookmark is not found!
06-13 14:35:37.441 I/stdout  (31058):   at com.jayway.android.robotium.solo.Clicker.clickOnText(Clicker.java:279)
06-13 14:35:37.441 I/stdout  (31058):   at com.jayway.android.robotium.solo.Solo.clickOnText(Solo.java:823)
06-13 14:35:37.441 I/stdout  (31058):   at org.mozilla.fennec_kats.tests.testBookmark.toggleBookmark(testBookmark.java:174)
06-13 14:35:37.441 I/stdout  (31058):   at org.mozilla.fennec_kats.tests.testBookmark.setUpBookmark(testBookmark.java:180)
06-13 14:35:37.441 I/stdout  (31058):   at org.mozilla.fennec_kats.tests.testBookmark.runMenuTest(testBookmark.java:57)
06-13 14:35:37.441 I/stdout  (31058):   at org.mozilla.fennec_kats.tests.testBookmark.testBookmark(testBookmark.java:50)
06-13 14:35:37.441 I/stdout  (31058):   at java.lang.reflect.Method.invokeNative(Native Method)
06-13 14:35:37.441 I/stdout  (31058):   at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
06-13 14:35:37.441 I/stdout  (31058):   at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
06-13 14:35:37.441 I/stdout  (31058):   at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:186)
06-13 14:35:37.441 I/stdout  (31058):   at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:169)
06-13 14:35:37.441 I/stdout  (31058):   at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:154)
06-13 14:35:37.441 I/stdout  (31058):   at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:525)
06-13 14:35:37.441 I/stdout  (31058):   at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1547)
06-13 14:35:37.441 I/Process (  290): Sending signal. PID: 31067 SIG: 9
06-13 14:35:37.441 I/ActivityManager(  290): Force stopping package org.mozilla.fennec_kats uid=10103
06-13 14:35:37.451 D/AndroidRuntime(31058): Shutting down VM
06-13 14:35:37.451 I/stdout  (31058): Test results for InstrumentationTestRunner=.F
06-13 14:35:37.451 I/stdout  (31058): Time: 52.22
06-13 14:35:37.451 I/stdout  (31058): FAILURES!!!
06-13 14:35:37.451 I/stdout  (31058): Tests run: 1,  Failures: 1,  Errors: 0

Robocop, however, happily reported success:

INFO | automation.py | Application pid: 0
0 INFO SimpleTest START
1 INFO TEST-START | testBookmark
2 INFO TEST-PASS | testBookmark | checking that bookmarks list exists - bookmarks list exists
3 INFO TEST-PASS | testBookmark | bookmarks list has 4 children (the default bookmarks) - 4 should equal 4
4 INFO TEST-PASS | testBookmark | Find default bookmark - Default bookmark for about:firefox found
5 INFO TEST-PASS | testBookmark | Find default bookmark - Default bookmark for about:home found
6 INFO TEST-PASS | testBookmark | Find default bookmark - Default bookmark for https://addons.mozilla.org/en-US/android/ found
7 INFO TEST-PASS | testBookmark | Find default bookmark - Default bookmark for http://support.mozilla.org/en-US/mobile found
8 INFO TEST-PASS | testBookmark | bookmarks list has 5 children (the default bookmarks and the new one) - 5 should equal 5
9 INFO TEST-PASS | testBookmark | Page is not bookmarked initially - false should equal false
10 INFO TEST-PASS | testBookmark | Awesomebar URL typed properly - http://mochi.test:8888/tests/robocop/robocop_blank_01.html should equal http://mochi.test:8888/tests/robocop/robocop_blank_01.html
11 INFO TEST-END | testBookmark | finished in 51357ms
12 INFO TEST-START | Shutdown
13 INFO Passed: 9
14 INFO Failed: 0
15 INFO Todo: 0
16 INFO SimpleTest FINISHED

Note that the number of "TEST-PASS" lines is fewer than it should be - on current inbound tbpl, I see "Passed: 15" for testBookmark. The exception thrown from the code just early-exited the test and the failure wasn't picked up. The attached patch should fix that.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 632816 [details] [diff] [review]
Catch exceptions in runTest

Review of attachment 632816 [details] [diff] [review]:
-----------------------------------------------------------------

looks good, and will extend to PixelTest as well.

Comment 2

[Kartikaya Gupta (email:[e-mail])]

https://hg.mozilla.org/integration/mozilla-inbound/rev/ed4548cac45f

Comment 3

[Ryan VanderMeulen [:RyanVM]]

http://mozillamemes.tumblr.com/post/19498220636/try-server-takes-the-beatings-so-mozilla-inbound

This turned robocop perma-orange. Backed out.
https://hg.mozilla.org/integration/mozilla-inbound/rev/124cfff79386

https://tbpl.mozilla.org/php/getParsedLog.php?id=12635808&tree=Mozilla-Inbound

2 INFO TEST-UNEXPECTED-FAIL | testSearchSuggestions | Exception caught - java.lang.RuntimeException: Error setting SuggestClient
I/Robocop ( 5049): 2 INFO TEST-UNEXPECTED-FAIL | testSearchSuggestions | Exception caught - java.lang.RuntimeException: Error setting SuggestClient
I/TestRunner( 5049): junit.framework.AssertionFailedError: 2 INFO TEST-UNEXPECTED-FAIL | testSearchSuggestions | Exception caught - java.lang.RuntimeException: Error setting SuggestClient

Comment 4

[Kartikaya Gupta (email:[e-mail])]

Thanks for the prompt backout, entirely my fault. I should have anticipated this happening.

Filed bug 764699 for the exception that was revealed.

Comment 5

[Kartikaya Gupta (email:[e-mail])]

Bug 764699 is fixed now, and the try run is showing green: https://tbpl.mozilla.org/?tree=Try&rev=281cd9438ca1 (except for the piles of timeouts and infra failures, which were happening before anyway).

Re-landed on inbound:

https://hg.mozilla.org/integration/mozilla-inbound/rev/66b5079b0dbe

Comment 6

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/66b5079b0dbe