822823 - Fix cases where robocop runTests returns 0 when a failure has occurred

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Ed Morley [:emorley]]

We only dump the logcat when a non-zero exit code was returned, to avoid log bloat across all of the robocop tests.

However, there are instances where an error has occurred, but runTests returns 0.

eg:
https://tbpl.mozilla.org/php/getParsedLog.php?id=17851959&tree=Firefox&full=1#error0
https://tbpl.mozilla.org/php/getParsedLog.php?id=17881486&tree=Firefox&full=1#error0

{
1 INFO TEST-START | testBookmarksTab
2 INFO TEST-PASS | testBookmarksTab | Inserted at:  - content://org.mozilla.fennec.db.browser/bookmarks/10?profile=default
3 INFO TEST-PASS | testBookmarksTab | checking that bookmarks list exists - android.widget.ListView@484353e8
4 INFO TEST-UNEXPECTED-FAIL | testBookmarksTab | bookmarks list has 5 children (defaults + a folder) - got 0, expected 5
Exception caught during test!
junit.framework.AssertionFailedError: 4 INFO TEST-UNEXPECTED-FAIL | testBookmarksTab | bookmarks list has 5 children (defaults + a folder) - got 0, expected 5
	at junit.framework.Assert.fail(Assert.java:47)
	at org.mozilla.fennec.FennecMochitestAssert._logMochitestResult(FennecMochitestAssert.java:107)
	at org.mozilla.fennec.FennecMochitestAssert.ok(FennecMochitestAssert.java:136)
	at org.mozilla.fennec.FennecMochitestAssert.is(FennecMochitestAssert.java:142)
	at org.mozilla.fennec.tests.testBookmarksTab.testList(testBookmarksTab.java:68)
	at org.mozilla.fennec.tests.testBookmarksTab.testBookmarksTab(testBookmarksTab.java:52)
	at java.lang.reflect.Method.invokeNative(Native Method)
	at java.lang.reflect.Method.invoke(Method.java:521)
	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:204)
	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:194)
	at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:186)
	at org.mozilla.fennec.tests.BaseTest.runTest(BaseTest.java:120)
	at junit.framework.TestCase.runBare(TestCase.java:127)
	at junit.framework.TestResult$1.protect(TestResult.java:106)
	at junit.framework.TestResult.runProtected(TestResult.java:124)
	at junit.framework.TestResult.run(TestResult.java:109)
	at junit.framework.TestCase.run(TestCase.java:118)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:169)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:154)
	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:520)
	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1447)
5 INFO TEST-UNEXPECTED-FAIL | testBookmarksTab | Exception caught - junit.framework.AssertionFailedError: 4 INFO TEST-UNEXPECTED-FAIL | testBookmarksTab | bookmarks list has 5 children (defaults + a folder) - got 0, expected 5
}

and 
{
1 INFO TEST-START | testThumbnails
2 INFO TEST-PASS | testThumbnails | Awesomebar URL typed properly - http://mochi.test:8888/tests/robocop/robocop_404.sjs?type=changeColor should equal http://mochi.test:8888/tests/robocop/robocop_404.sjs?type=changeColor
3 INFO TEST-PASS | testThumbnails | Awesomebar URL typed properly - http://mochi.test:8888/tests/robocop/robocop_404.sjs?type=do404 should equal http://mochi.test:8888/tests/robocop/robocop_404.sjs?type=do404
4 INFO TEST-PASS | testThumbnails | Awesomebar URL typed properly - about:home should equal about:home
5 INFO TEST-UNEXPECTED-FAIL | testThumbnails | Top site thumbnail updated for HTTP 200 - got 0, expected -16711936
Exception caught during test!
junit.framework.AssertionFailedError: 5 INFO TEST-UNEXPECTED-FAIL | testThumbnails | Top site thumbnail updated for HTTP 200 - got 0, expected -16711936
	at junit.framework.Assert.fail(Assert.java:47)
	at org.mozilla.fennec.FennecMochitestAssert._logMochitestResult(FennecMochitestAssert.java:107)
	at org.mozilla.fennec.FennecMochitestAssert.ok(FennecMochitestAssert.java:136)
	at org.mozilla.fennec.FennecMochitestAssert.is(FennecMochitestAssert.java:142)
	at org.mozilla.fennec.tests.testThumbnails.testThumbnails(testThumbnails.java:57)
	at java.lang.reflect.Method.invokeNative(Native Method)
	at java.lang.reflect.Method.invoke(Method.java:521)
	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:204)
	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:194)
	at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:186)
	at org.mozilla.fennec.tests.BaseTest.runTest(BaseTest.java:120)
	at junit.framework.TestCase.runBare(TestCase.java:127)
	at junit.framework.TestResult$1.protect(TestResult.java:106)
	at junit.framework.TestResult.runProtected(TestResult.java:124)
	at junit.framework.TestResult.run(TestResult.java:109)
	at junit.framework.TestCase.run(TestCase.java:118)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:169)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:154)
	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:520)
	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1447)
6 INFO TEST-UNEXPECTED-FAIL | testThumbnails | Exception caught - junit.framework.AssertionFailedError: 5 INFO TEST-UNEXPECTED-FAIL | testThumbnails | Top site thumbnail updated for HTTP 200 - got 0, expected -16711936
}

Comment 1

[Geoff Brown [:gbrown]]

Attachment: dump logcat if log data contains "TEST-UNEXPECTED-FAIL" or if no Shutdown line is found

When tests fail but there is no crash, am instrument ... returns 0, and I could not find a way to change that. Instead, this patch scans the test output for TEST-UNEXPECTED-FAIL, dumping logcat if found. For additional safety, we dump logcat if the end-of-test Shutdown message is missing.

Sample log: https://tbpl.mozilla.org/php/getParsedLog.php?id=18654619&tree=Try&full=1#error0

Comment 2

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/927268cb7811

Comment 3

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/927268cb7811