937902 - Allow redirection of logcat and screenshots when running robocop tests locally

Status: RESOLVED WONTFIX

(Testing :: Mochitest, defect)

Description

[Geoff Brown [:gbrown]]

When running robocop tests, if an error is detected, runtestsremote.py will dump logcat and any screenshots found to stdout. This works great on tbpl, and I don't want to change a thing there. 

But when running tests locally, there's too much data dumped too quickly. The user asks for this test to run, all this stuff scrolls by, and at the end, whether the test(s) passed or failed, logcat messages are displayed. Screenshots are dumped in base64, so need to be copied and decoded -- why not just provide the decoded file(s)?

By default, local runs should redirect logcat(s) and screenshot(s) to files.

Comment 1

[Geoff Brown [:gbrown]]

Attachment: wip - optionally redirect logcat, screenshots

mozdev@ubuntu:~/objdirs/droid$ make mochitest-robocop
Error deleting /data/anr/traces.txt
Mochi-Remote INFO | Device info: {}
Mochi-Remote INFO | Test root: /mnt/sdcard/tests
Mochi-Remote INFO | Android sdk version '12'; will use this to filter manifests
MochitestServer : launching [u'/home/mozdev/objdirs/firefox/dist/bin/xpcshell', '-g', '/home/mozdev/objdirs/firefox/dist/bin', '-v', '170', '-f', '/home/mozdev/objdirs/droid/_tests/testing/mochitest/httpd.js', '-e', "const _PROFILE_PATH = '/tmp/tmp1hUzUI'; const _SERVER_PORT = '8888'; const _SERVER_ADDR = '192.168.0.50'; const _TEST_PREFIX = undefined; const _DISPLAY_RESULTS = false;", '-f', './server.js']
runtests.py | Server pid: 32119
runtests.py | Running tests: start.

Robocop derived process name: org.mozilla.gecko
INFO | automation.py | Application pid: 0
0 INFO SimpleTest START
1 INFO TEST-START | testBookmarksPage
2 INFO TEST-PASS | testBookmarksPage | Inserting/updating a new bookmark - Inserting/updating the bookmark with the title = Browser Blank Page 01 and the url = http://mochi.test:8888/tests/robocop/robocop_blank_01.html
3 INFO TEST-PASS | testBookmarksPage | waiting for urlbar text to gain focus - urlbar text gained focus
4 INFO TEST-PASS | testBookmarksPage | waiting for urlbar text to gain focus - urlbar text gained focus
5 INFO TEST-PASS | testBookmarksPage | Checking that about:firefox displayed as a bookmark - about:firefox displayed
6 INFO TEST-PASS | testBookmarksPage | waiting for urlbar text to gain focus - urlbar text gained focus
7 INFO TEST-PASS | testBookmarksPage | Checking that http://support.mozilla.org/en-US/products/mobile displayed as a bookmark - http://support.mozilla.org/en-US/products/mobile displayed
8 INFO TEST-PASS | testBookmarksPage | waiting for urlbar text to gain focus - urlbar text gained focus
9 INFO TEST-PASS | testBookmarksPage | Checking that https://addons.mozilla.org/en-US/android/ displayed as a bookmark - https://addons.mozilla.org/en-US/android/ displayed
10 INFO TEST-PASS | testBookmarksPage | waiting for urlbar text to gain focus - urlbar text gained focus
11 INFO TEST-PASS | testBookmarksPage | Checking that the context menu option is present - Open in New Tab is present
12 INFO TEST-PASS | testBookmarksPage | Checking that the context menu option is present - Open in Private Tab is present
13 INFO TEST-PASS | testBookmarksPage | Checking that the context menu option is present - Edit is present
14 INFO TEST-PASS | testBookmarksPage | Checking that the context menu option is present - Remove is present
EventExpecter: no longer listening for Tab:Added
15 INFO TEST-PASS | testBookmarksPage | Checking that the tab is not changed - The tab was not changed
16 INFO TEST-PASS | testBookmarksPage | waiting for urlbar text to gain focus - urlbar text gained focus
EventExpecter: no longer listening for Tab:Added
17 INFO TEST-PASS | testBookmarksPage | Checking that the tab is not changed - The tab was not changed
18 INFO TEST-PASS | testBookmarksPage | waiting for urlbar text to gain focus - urlbar text gained focus
19 INFO TEST-PASS | testBookmarksPage | waiting for urlbar text to gain focus - urlbar text gained focus
20 INFO TEST-PASS | testBookmarksPage | Checking that the value is correct - The value = New bookmark title is correct
21 INFO TEST-PASS | testBookmarksPage | Checking that the value is correct - The value = www.NewBookmark.url is correct
22 INFO TEST-PASS | testBookmarksPage | Checking that the value is correct - The value = newBookmarkKeyword is correct
waitForText timeout on BOOKMARKS
23 INFO TEST-PASS | testBookmarksPage | waiting for urlbar text to gain focus - urlbar text gained focus
Exception caught during test!
junit.framework.AssertionFailedError: ViewPager is not found!
	at junit.framework.Assert.fail(Assert.java:47)
	at junit.framework.Assert.assertTrue(Assert.java:20)
	at com.jayway.android.robotium.solo.Waiter.waitForAndGetView(Waiter.java:501)
	at com.jayway.android.robotium.solo.Solo.getView(Solo.java:2033)
	at org.mozilla.gecko.tests.AboutHomeTest.openAboutHomeTab(AboutHomeTest.java:232)
	at org.mozilla.gecko.tests.AboutHomeTest.getDisplayedBookmark(AboutHomeTest.java:103)
	at org.mozilla.gecko.tests.AboutHomeTest.openBookmarkContextMenu(AboutHomeTest.java:91)
	at org.mozilla.gecko.tests.testBookmarksPage.testBookmarksPage(testBookmarksPage.java:55)
	at java.lang.reflect.Method.invokeNative(Native Method)
	at java.lang.reflect.Method.invoke(Method.java:491)
	at android.test.InstrumentationTestCase.runMethod(InstrumentationTestCase.java:214)
	at android.test.InstrumentationTestCase.runTest(InstrumentationTestCase.java:199)
	at android.test.ActivityInstrumentationTestCase2.runTest(ActivityInstrumentationTestCase2.java:186)
	at org.mozilla.gecko.tests.BaseTest.runTest(BaseTest.java:150)
	at junit.framework.TestCase.runBare(TestCase.java:127)
	at junit.framework.TestResult$1.protect(TestResult.java:106)
	at junit.framework.TestResult.runProtected(TestResult.java:124)
	at junit.framework.TestResult.run(TestResult.java:109)
	at junit.framework.TestCase.run(TestCase.java:118)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:169)
	at android.test.AndroidTestRunner.runTest(AndroidTestRunner.java:154)
	at android.test.InstrumentationTestRunner.onStart(InstrumentationTestRunner.java:525)
	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:1547)
24 INFO TEST-UNEXPECTED-FAIL | testBookmarksPage | Exception caught - junit.framework.AssertionFailedError: ViewPager is not found!
25 INFO TEST-END | testBookmarksPage | finished in 141367ms
26 INFO TEST-START | Shutdown
27 INFO Passed: 22
28 INFO Failed: 1
29 INFO Todo: 0
30 INFO SimpleTest FINISHED
INFO | automation.py | Application ran for: 0:02:27.515063
INFO | zombiecheck | Reading PID log: /tmp/tmpMkklOqpidlog
/data/anr/traces.txt not found
WARNING | leakcheck | refcount logging is off, so leaks can't be detected!
runtests.py | Running tests: end.
Mochi-Remote INFO | Logcat saved to: /home/mozdev/objdirs/droid/mochitest-logcat-testBookmarksPage
Mochi-Remote INFO | Device info: {}
Mochi-Remote INFO | Test root: /mnt/sdcard/tests
Mochi-Remote INFO | SCREENSHOT: FOUND: [/mnt/sdcard/Robotium-Screenshots/robocop-screenshot.jpg]
Mochi-Remote INFO | Screenshot saved to: /home/mozdev/objdirs/droid/mochitest-screenshot-testBookmarksPage1.jpg
Mochi-Remote INFO | SCREENSHOT: TOTAL PRINTED: [1]
INFO | runtests.py | Test summary: start.
0 INFO SimpleTest START
54 INFO TEST-START | Shutdown
55 INFO Passed: 22
56 INFO Failed: 1
57 INFO Todo: 0
58 INFO SimpleTest FINISHED
INFO | runtests.py | Test summary: end.
Mochi-Remote INFO | Logcat saved to: /home/mozdev/objdirs/droid/mochitest-logcat-final
Mochi-Remote INFO | Device info: {}
Mochi-Remote INFO | Test root: /mnt/sdcard/tests
make: *** [mochitest-robocop] Error 1
mozdev@ubuntu:~/objdirs/droid$ ls -l mochitest-*
-rw-rw-r-- 1 mozdev mozdev 387993 Nov 12 16:57 mochitest-logcat-final
-rw-rw-r-- 1 mozdev mozdev 387993 Nov 12 16:57 mochitest-logcat-testBookmarksPage
-rw-rw-r-- 1 mozdev mozdev  42891 Nov 12 16:57 mochitest-screenshot-testBookmarksPage1.jpg

Comment 2

[Geoff Brown [:gbrown]]

This patch has been neglected for too long now and there is some related work in bug 929654 which looks promising, so I think I'll abandon this path.