1105522 - Robocop exception handling is skipped in UITests

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Testing, defect)

Description

[Geoff Brown [:gbrown]]

In bug 1085837, a NullPointerException appears in the logcat for a Robocop test but does not fail the test with an appropriate message in the main test log. Notice that the exception stack trace does not include BaseTest.runTest:

04:08:13     INFO -  11-26 03:51:42.233 I/TestRunner( 1355): failed: testAppMenuPathways(org.mozilla.gecko.tests.testAppMenuPathways)
04:08:13     INFO -  11-26 03:51:42.246 I/TestRunner( 1355): ----- begin exception -----
04:08:13     INFO -  11-26 03:51:42.303 I/TestRunner( 1355): 
04:08:13     INFO -  11-26 03:51:42.303 I/TestRunner( 1355): java.lang.NullPointerException
04:08:13     INFO -  11-26 03:51:42.303 I/TestRunner( 1355): 	at org.mozilla.gecko.tests.components.AppMenuComponent.assertMenuItemIsDisabledAndVisible(AppMenuComponent.java:128)
04:08:13     INFO -  11-26 03:51:42.303 I/TestRunner( 1355): 	at org.mozilla.gecko.tests.testAppMenuPathways._testSaveAsPDFPathway(testAppMenuPathways.java:26)
04:08:13     INFO -  11-26 03:51:42.303 I/TestRunner( 1355): 	at org.mozilla.gecko.tests.testAppMenuPathways.testAppMenuPathways(testAppMenuPathways.java:21)
04:08:13     INFO -  11-26 03:51:42.303 I/TestRunner( 1355): 	at java.lang.reflect.Method.invokeNative(Native Method)
04:08:13     INFO -  11-26 03:51:42.303 I/TestRunner( 1355): 	at java.lang.reflect.Method.invoke(Method.java:507)
04:08:13     INFO -  11-26 03:51:42.303 I/TestRunner( 1355): 	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:204)
04:08:13     INFO -  11-26 03:51:42.303 I/TestRunner( 1355): 	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:194)
04:08:13     INFO -  11-26 03:51:42.303 I/TestRunner( 1355): 	at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:186)
04:08:13     INFO -  11-26 03:51:42.303 I/TestRunner( 1355): 	at junit.framework.TestCase.runBare(TestCase.java:127)
04:08:13     INFO -  11-26 03:51:42.303 I/TestRunner( 1355): 	at junit.framework.TestResult$1.protect(TestResult.java:106)
04:08:13     INFO -  11-26 03:51:42.303 I/TestRunner( 1355): 	at junit.framework.TestResult.runProtected(TestResult.java:124)
04:08:13     INFO -  11-26 03:51:42.303 I/TestRunner( 1355): 	at junit.framework.TestResult.run(TestResult.java:109)
04:08:13     INFO -  11-26 03:51:42.303 I/TestRunner( 1355): 	at junit.framework.TestCase.run(TestCase.java:118)
04:08:13     INFO -  11-26 03:51:42.303 I/TestRunner( 1355): 	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:169)
04:08:13     INFO -  11-26 03:51:42.303 I/TestRunner( 1355): 	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:154)
04:08:13     INFO -  11-26 03:51:42.303 I/TestRunner( 1355): 	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:529)
04:08:13     INFO -  11-26 03:51:42.303 I/TestRunner( 1355): 	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1448)
04:08:13     INFO -  11-26 03:51:42.303 I/TestRunner( 1355): ----- end exception -----

Is the stack misleading, or is BaseTest.runTest no longer invoked?

Comment 1

[Geoff Brown [:gbrown]]

Oh, I see! testAppMenuPathways (and others) extend UITest, which inherits from BaseRobocopTest, bypassing BaseTest and the exception handling in runTest. This is probably a bad thing....

Comment 2

[Geoff Brown [:gbrown]]

Attachment: implement runTest in UITest

This just copies BaseTest.runTest into UITest. (Cannot be implemented in BaseRobocopTest without moving mSolo too.)

Comment 3

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Comment on attachment 8529914 [details] [diff] [review]
implement runTest in UITest

Review of attachment 8529914 [details] [diff] [review]:
-----------------------------------------------------------------

(In reply to Geoff Brown [:gbrown] from comment #2)
> This just copies BaseTest.runTest into UITest. (Cannot be implemented in
> BaseRobocopTest without moving mSolo too.)

Seems do-able - I filed bug 1106900 and bug 1106903.

Comment 4

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=9dd49c68402d

https://hg.mozilla.org/integration/mozilla-inbound/rev/c292aca6244e

Comment 5

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/c292aca6244e

Comment 6

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Thanks for pushing this forward, Geoff! Hearts!