Closed Bug 1244906 Opened 9 years ago Closed 9 years ago

Robocop test failures don't always supply screenshot artifacts

Categories

(Firefox for Android Graveyard :: Testing, defect)

Product:
Firefox for Android Graveyard
Component:
Testing
Platform:
All
Android
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: mcomella, Assigned: gbrown)

References

Details

The failures in my try push for bug 1118268 did not include screenshots: https://treeherder.mozilla.org/#/jobs?repo=try&revision=bdc973a55aef I've seen this issue before but I don't know when it happens and when it doesn't. Geoff said he'd take a look.
Our code requests a robotium screenshot at http://hg.mozilla.org/mozilla-central/annotate/d07dbd40dcd2/mobile/android/tests/browser/robocop/src/org/mozilla/gecko/tests/BaseRobocopTest.java#l192. It looks like that code was hit: 11:43:57 WARNING - TEST-UNEXPECTED-FAIL | testPrivateBrowsing | GeckoEventExpecter - blockForEvent timeout: Tab:Added 11:43:57 INFO - 0 ERROR Exception caught during test! - junit.framework.AssertionFailedError: TEST-UNEXPECTED-FAIL | testPrivateBrowsing | GeckoEventExpecter - blockForEvent timeout: Tab:Added 11:43:57 INFO - at junit.framework.Assert.fail(Assert.java:50) 11:43:57 INFO - at org.mozilla.gecko.FennecMochitestAssert._logMochitestResult(FennecMochitestAssert.java:128) 11:43:57 INFO - at org.mozilla.gecko.FennecMochitestAssert.ok(FennecMochitestAssert.java:150) 11:43:57 INFO - at org.mozilla.gecko.FennecNativeActions$GeckoEventExpecter.blockForEvent(FennecNativeActions.java:96) 11:43:57 INFO - at org.mozilla.gecko.FennecNativeActions$GeckoEventExpecter.blockForEvent(FennecNativeActions.java:80) 11:43:57 INFO - at org.mozilla.gecko.tests.PixelTest.addTab(PixelTest.java:64) 11:43:57 INFO - at org.mozilla.gecko.tests.testPrivateBrowsing.addTab(testPrivateBrowsing.java:19) 11:43:57 INFO - at org.mozilla.gecko.tests.testPrivateBrowsing.testPrivateBrowsing(testPrivateBrowsing.java:30) 11:43:57 INFO - at java.lang.reflect.Method.invokeNative(Native Method) 11:43:57 INFO - at java.lang.reflect.Method.invoke(Method.java:525) 11:43:57 INFO - at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214) 11:43:57 INFO - at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199) 11:43:57 INFO - at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:192) 11:43:57 INFO - at org.mozilla.gecko.tests.BaseRobocopTest.runTest(BaseRobocopTest.java:188) 11:43:57 INFO - at junit.framework.TestCase.runBare(TestCase.java:134) 11:43:57 INFO - at junit.framework.TestResult$1.protect(TestResult.java:115) 11:43:57 INFO - at junit.framework.TestResult.runProtected(TestResult.java:133) 11:43:57 INFO - at junit.framework.TestResult.run(TestResult.java:118) 11:43:57 INFO - at junit.framework.TestCase.run(TestCase.java:124) 11:43:57 INFO - at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:191) 11:43:57 INFO - at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:176) 11:43:57 INFO - at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:554) 11:43:57 INFO - at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1701) 11:43:57 WARNING - TEST-UNEXPECTED-FAIL | testPrivateBrowsing | Exception caught - junit.framework.AssertionFailedError: TEST-UNEXPECTED-FAIL | testPrivateBrowsing | GeckoEventExpecter - blockForEvent timeout: Tab:Added 11:43:57 INFO - TEST-OK | testPrivateBrowsing | took 281711ms 11:43:57 INFO - TEST-START | Shutdown ...so I wonder if there is a limitation in Robotium.
(In reply to Geoff Brown [:gbrown] from comment #2) > https://github.com/robotiumtech/robotium/issues/579? Seems likely (though we have a semi-custom menu implementation so maybe we get around this). Another possibility, from the javadoc [1]: Requires write permission (android.permission.WRITE_EXTERNAL_STORAGE) in AndroidManifest.xml of the application under test. Maybe Sebastian's permission's changes are related (though I'd think we'd Android 6 emulators for this to be a problem). [1]: https://robotium.googlecode.com/svn/doc/com/robotium/solo/Solo.html#takeScreenshot%28java.lang.String%29
This might just be fixed if we moved to the Toolbar API (bug 1126061), but that's a long way off. If it's robotium's API's problem, is there another way we can take a screenshot during a failure?
One approach to consider is taking a screenshot on the emulator host. Linux screentopng seems to reliably produce an accurate screenshot of the emulator content as well as the desktop environment the emulator is running in. We don't have a good way of triggering the screenshot (or any action on the host) from a test, but I hope to pursue this idea in bug 1178554.
Mentor: gbrown
See Also: → 1178554
For bug 1235689, https://treeherder.mozilla.org/#/jobs?repo=try&revision=56e1c685cb1f also has failures without screenshots. I verified that takeScreenshot is called. I suspect (but am not sure) that the Page menu is open at the time of the failure.
See Also: → 1181020
Bug 1118268 (an intermittent testPrivateBrowsing failure) almost never provides a robocop screenshot. https://treeherder.mozilla.org/#/jobs?repo=try&revision=508f1b370e46&selectedJob=18886333 reproduced this failure with additional logging on top of robotium 5.5.4. It shows that takeScreenshot() returns early at https://github.com/RobotiumTech/robotium/blob/master/robotium-solo/src/main/java/com/robotium/solo/ScreenshotTaker.java#L79 public void takeScreenshot(final String name, final int quality) { View decorView = getScreenshotView(); if(decorView == null) return;
I can't think of anything to do here, except bug 1178554, or hoping that robotium will find a way to address this.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → WONTFIX
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.