Robocop test failures don't always supply screenshot artifacts

RESOLVED WONTFIX

Status

()

Firefox for Android
Testing
RESOLVED WONTFIX
2 years ago
2 years ago

People

(Reporter: mcomella, Assigned: gbrown)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

The failures in my try push for bug 1118268 did not include screenshots:
  https://treeherder.mozilla.org/#/jobs?repo=try&revision=bdc973a55aef

I've seen this issue before but I don't know when it happens and when it doesn't.

Geoff said he'd take a look.
(Assignee)

Comment 1

2 years ago
Our code requests a robotium screenshot at http://hg.mozilla.org/mozilla-central/annotate/d07dbd40dcd2/mobile/android/tests/browser/robocop/src/org/mozilla/gecko/tests/BaseRobocopTest.java#l192. It looks like that code was hit:

11:43:57  WARNING -  TEST-UNEXPECTED-FAIL | testPrivateBrowsing | GeckoEventExpecter - blockForEvent timeout: Tab:Added
11:43:57     INFO -  0 ERROR Exception caught during test! - junit.framework.AssertionFailedError: TEST-UNEXPECTED-FAIL | testPrivateBrowsing | GeckoEventExpecter - blockForEvent timeout: Tab:Added
11:43:57     INFO -  	at junit.framework.Assert.fail(Assert.java:50)
11:43:57     INFO -  	at org.mozilla.gecko.FennecMochitestAssert._logMochitestResult(FennecMochitestAssert.java:128)
11:43:57     INFO -  	at org.mozilla.gecko.FennecMochitestAssert.ok(FennecMochitestAssert.java:150)
11:43:57     INFO -  	at org.mozilla.gecko.FennecNativeActions$GeckoEventExpecter.blockForEvent(FennecNativeActions.java:96)
11:43:57     INFO -  	at org.mozilla.gecko.FennecNativeActions$GeckoEventExpecter.blockForEvent(FennecNativeActions.java:80)
11:43:57     INFO -  	at org.mozilla.gecko.tests.PixelTest.addTab(PixelTest.java:64)
11:43:57     INFO -  	at org.mozilla.gecko.tests.testPrivateBrowsing.addTab(testPrivateBrowsing.java:19)
11:43:57     INFO -  	at org.mozilla.gecko.tests.testPrivateBrowsing.testPrivateBrowsing(testPrivateBrowsing.java:30)
11:43:57     INFO -  	at java.lang.reflect.Method.invokeNative(Native Method)
11:43:57     INFO -  	at java.lang.reflect.Method.invoke(Method.java:525)
11:43:57     INFO -  	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
11:43:57     INFO -  	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
11:43:57     INFO -  	at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192)
11:43:57     INFO -  	at org.mozilla.gecko.tests.BaseRobocopTest.runTest(BaseRobocopTest.java:188)
11:43:57     INFO -  	at junit.framework.TestCase.runBare(TestCase.java:134)
11:43:57     INFO -  	at junit.framework.TestResult$1.protect(TestResult.java:115)
11:43:57     INFO -  	at junit.framework.TestResult.runProtected(TestResult.java:133)
11:43:57     INFO -  	at junit.framework.TestResult.run(TestResult.java:118)
11:43:57     INFO -  	at junit.framework.TestCase.run(TestCase.java:124)
11:43:57     INFO -  	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:191)
11:43:57     INFO -  	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:176)
11:43:57     INFO -  	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:554)
11:43:57     INFO -  	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1701)
11:43:57  WARNING -  TEST-UNEXPECTED-FAIL | testPrivateBrowsing | Exception caught - junit.framework.AssertionFailedError: TEST-UNEXPECTED-FAIL | testPrivateBrowsing | GeckoEventExpecter - blockForEvent timeout: Tab:Added
11:43:57     INFO -  TEST-OK | testPrivateBrowsing | took 281711ms
11:43:57     INFO -  TEST-START | Shutdown

...so I wonder if there is a limitation in Robotium.
(In reply to Geoff Brown [:gbrown] from comment #2)
> https://github.com/robotiumtech/robotium/issues/579?

Seems likely (though we have a semi-custom menu implementation so maybe we get around this).

Another possibility, from the javadoc [1]:
 Requires write permission (android.permission.WRITE_EXTERNAL_STORAGE) in AndroidManifest.xml of the application under test.

Maybe Sebastian's permission's changes are related (though I'd think we'd Android 6 emulators for this to be a problem).

[1]: https://robotium.googlecode.com/svn/doc/com/robotium/solo/Solo.html#takeScreenshot%28java.lang.String%29
This might just be fixed if we moved to the Toolbar API (bug 1126061), but that's a long way off.

If it's robotium's API's problem, is there another way we can take a screenshot during a failure?
(Assignee)

Comment 5

2 years ago
One approach to consider is taking a screenshot on the emulator host. Linux screentopng seems to reliably produce an accurate screenshot of the emulator content as well as the desktop environment the emulator is running in. We don't have a good way of triggering the screenshot (or any action on the host) from a test, but I hope to pursue this idea in bug 1178554.
Mentor: gbrown
See Also: → bug 1178554
(Assignee)

Comment 6

2 years ago
For bug 1235689, https://treeherder.mozilla.org/#/jobs?repo=try&revision=56e1c685cb1f also has failures without screenshots. I verified that takeScreenshot is called. I suspect (but am not sure) that the Page menu is open at the time of the failure.
(Assignee)

Updated

2 years ago
See Also: → bug 1181020
(Assignee)

Comment 8

2 years ago
Bug 1118268 (an intermittent testPrivateBrowsing failure) almost never provides a robocop screenshot. https://treeherder.mozilla.org/#/jobs?repo=try&revision=508f1b370e46&selectedJob=18886333 reproduced this failure with additional logging on top of robotium 5.5.4.

It shows that takeScreenshot() returns early at https://github.com/RobotiumTech/robotium/blob/master/robotium-solo/src/main/java/com/robotium/solo/ScreenshotTaker.java#L79

	public void takeScreenshot(final String name, final int quality) {
		View decorView = getScreenshotView();
		if(decorView == null) 
			return;
(Assignee)

Comment 9

2 years ago
I can't think of anything to do here, except bug 1178554, or hoping that robotium will find a way to address this.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.