1605256 - Expose harness exceptions differently in our junit tests

Status: RESOLVED FIXED

(GeckoView :: General, enhancement)

Description

[[ex-Mozilla] Agi Sferro | :agi]

Sometimes our harness fail even before running any test code. In this case the test is not at fault and we shouldn't mark it as a failure for the specific test but rather for the test harness.

As an example, this test failure:

 INSTRUMENTATION_STATUS: stack=org.mozilla.geckoview.test.util.UiThreadUtils$TimeoutException: Timed out after 30000ms
   at org.mozilla.geckoview.test.util.UiThreadUtils$TimeoutRunnable.run(UiThreadUtils.java:58)
   at android.os.Handler.handleCallback(Handler.java:751)
   at android.os.Handler.dispatchMessage(Handler.java:95)
   at org.mozilla.geckoview.test.util.UiThreadUtils.waitForCondition(UiThreadUtils.java:161)
   at org.mozilla.geckoview.test.rule.GeckoSessionTestRule.openSession(GeckoSessionTestRule.java:1135)
   at org.mozilla.geckoview.test.rule.GeckoSessionTestRule.prepareStatement(GeckoSessionTestRule.java:1085)
   at org.mozilla.geckoview.test.rule.GeckoSessionTestRule$2.lambda$evaluate$0$GeckoSessionTestRule$2(GeckoSessionTestRule.java:1264)
   at org.mozilla.geckoview.test.rule.-$$Lambda$GeckoSessionTestRule$2$mzZNnl5Bu5F2_4xGxj0DHU4J33I.run(lambda)
   at android.app.Instrumentation$SyncRunnable.run(Instrumentation.java:1950)
   at android.os.Handler.handleCallback(Handler.java:751)
   at android.os.Handler.dispatchMessage(Handler.java:95)
   at android.os.Looper.loop(Looper.java:154)
   at android.app.ActivityThread.main(ActivityThread.java:6077)
   at java.lang.reflect.Method.invoke(Native Method)
   at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:866)
   at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:756)

it's not a test failure because we fail to even start the main session in prepareStatement.

Comment 1

[[ex-Mozilla] Agi Sferro | :agi]

We could probably expose the failing exception to our test logger and throw a TestHarnessException in these cases.

Comment 2

[[ex-Mozilla] Agi Sferro | :agi]

Attachment: Bug 1605256 - Add exception type to message.

This commit does two things:

• It exposes the error that caused the test to fail in our logs, so it would
  appear like so:

TEST-UNEXPECTED-FAIL | org.mozilla.geckoview.test.ExtensionActionTest.setIconSvg[#pageAction] | java.lang.AssertionError: We shouldn't get here

• It catches exceptions that are raised before any test code is run and outputs
  a different message (since those failures do not depend on the test at all),
  like so:

ERROR runjunit.py | The previous test failed because of an error in the test harness | org.mozilla.geckoview.test.rule.TestHarnessException: java.lang.RuntimeException: We shouldn't get here

This will hopefully help with spurious intermittents and understand when a
failure changes.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8b6baaf1e987
Add exception type to message. r=snorp

Comment 4

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/8b6baaf1e987

Comment 5

[Geoff Brown [:gbrown]]

TestHarnessException is showing up in a new group of junit failures in bug 1581345.