Open Bug 1178554 Opened 5 years ago Updated 3 years ago

Make screenshot-on-fail work for Android emulator mochitests

Categories

(Testing :: Mochitest, defect)

defect
Not set

Tracking

(Not tracked)

People

(Reporter: jgriffin, Unassigned)

References

Details

Ahal recently enabled --screenshot-on-fail for mochitests, so that whenever they fail for any reason, we create a screenshot.  This works well for desktop platforms, but doesn't seem to be honored by Android emulator tests, per this try run:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=bb42ee341afc

It seems like it probably wouldn't be hard to hook up for this case.
I know screentopng works on the android emulator hosts, and actually gives a better screenshot in most circumstances than any on-device screenshot I am aware of. I would verify the platform at http://hg.mozilla.org/mozilla-central/annotate/c26dbd63604d/build/automationutils.py#l421 and also check that MOZ_UPLOAD_DIR is set.
Actually, now I see that dumpScreen is called via killAndGetStack, which probably is not used on Android...we probably need to hook dumpScreen in somewhere else.
Assignee: nobody → gbrown
--screenshot-on-fail is wired up in the OutputHandler in desktop Mochitest:
https://hg.mozilla.org/mozilla-central/annotate/f5e3bacfb60e/testing/mochitest/runtests.py#l2513

It processes the structured log output and calls dumpScreen when it sees an unexpected result. Unfortunately the Android harnesses never got ported to mozbase so they're not using any of this code. :-(
Thanks Ted. I was confusing screenshot-on-fail with the dumpScreen in killAndGetStack. Neither are supported on Android.

There's no synchronization between the js mochitest and the OutputHandler, right? I mean, a test fails, writes a failure message, and the next test is started; the harness' OutputHandler processes the failure message and triggers a screen dump very shortly after the failure, but the screenshot could occur after the next test has started.

On Android this is a lot worse, because the python harness fetches the log from device at intervals of 60 seconds. Taking a screenshot up to 60 seconds after a failure might not have much value, and also might be confusing (I think the natural assumption is that the screenshot reflects the state of the browser at the time of the error).

I am tempted to WONTFIX but open another bug for screenshot-on-timeout and on crash.
Yes, the JS test runner will simply continue on after printing a test failure, so it's not fantastic.

Our logging story on Android kind of sucks in general. Is there a simple way we could stream logs from the device to the harness? My simplest idea would be to add a TCP server as a logging destination in the JS harness, and make the Python harness listen on a port and accept log messages over it.
Another option is to install busybox on the emulator, and then use 'adb shell tail -f' to stream the log to the harness.
An attempt at the simpler problem of screenshot-on-timeout: https://treeherder.mozilla.org/#/jobs?repo=try&revision=b3d684002bf3. To properly detect no-output timeouts, I needed to drastically reduce the devicemanager timeout used in launchProcess so that waitForFinish is actually running for most of the robocop test duration; that breaks waitForFinish for any test that launches another activity (maybe just testFilePicker). Maybe waitForFinish could be improved, to keep waiting (temporarily) if the top activity is not fennec but fennec is still running, or something like that.
See Also: → 1244906
See also bug 1212460 and the confusion there around log timestamps. That's not obviously or directly related to this bug, but I feel like they go together: Perhaps we need a re-design of the harness' process management, logging, etc with more attention to real-time considerations.
I added screenshot-on-harness-timeout support in bug 1319196.
See Also: → 1319196
I'd like to follow-up on comment 8, some day...but I'm not finding time for this currently.
Assignee: gbrown → nobody
You need to log in before you can comment on or make changes to this bug.