719443 - create a robocop test type for 'android'

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

in bug 715215, we install the robocop.apk.  Now we need to create a test type robocop to replace browser-chrome.

This would require a command similar to browser-chrome, but without the "--browser-chrome --test-path=mobile" and instead a "--robocop=robocop.ini".

Comment 1

[[e-mail] (:bhearsum)]

Hal, I know you're working on one part of this, so I'm opportunistically assigning this to you. Throw it back if that's wrong.

Comment 2

[Hal Wine [:hwine] use NI!]

(In reply to Joel Maher (:jmaher) from comment #0)
> Now we need to create a test
> type robocop to replace browser-chrome.
> 
> This would require a command similar to browser-chrome, but without the
> "--browser-chrome --test-path=mobile" and instead a "--robocop=robocop.ini".

Joel - once bug 719697 get's resolved, please post the final exact command line that works, and I'll pick this up.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

how close are we to landing this.  I am testing a final patch for bug 719697, so I wanted to make sure we were ready to land this as soon as my patch lands.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

This command gets the tests running:
python runtestsremote.py --app=org.mozilla.fennec_jmaher --xre-path=/home/jmaher/mozilla/inbound/obj-i686-pc-linux-gnu/dist/bin/ --deviceIP=192.168.1.100 --robocop=robocop.ini

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Here is some output:
jmaher@jmaher-MacBookPro:~/Downloads/robo/mochitest$ python runtestsremote.py --app=org.mozilla.fennec --xre-path=/home/jmaher/mozilla/inbound/obj-i686-pc-linux-gnu/dist/bin/ --deviceIP=192.168.1.100 --robocop=robocop.ini
unable to execute 'cp' on device; consider installing busybox from Android Market
NOT running as root
adbd cannot run as root in production builds
reconnecting socket
removing file: /sdcard/fennec_ids.txt
removing file: /sdcard/robotium.config
INFO | runtests.py | Installing extension at /home/jmaher/Downloads/robo/mochitest/extensions/specialpowers to /tmp/tmpyZsYvd.
INFO | runtests.py | Installing extension at /home/jmaher/Downloads/robo/mochitest/extensions/workerbootstrap to /tmp/tmpyZsYvd.
INFO | runtests.py | Installing extension at /home/jmaher/Downloads/robo/mochitest/extensions/worker to /tmp/tmpyZsYvd.
INFO | runtests.py | Installing extension at /home/jmaher/Downloads/robo/mochitest/extensions/[e-mail] to /tmp/tmpyZsYvd.
pushing directory: /tmp/tmpyZsYvd to /mnt/sdcard/tests/profile
args: ['/home/jmaher/mozilla/inbound/obj-i686-pc-linux-gnu/dist/bin/xpcshell', '-g', '/home/jmaher/mozilla/inbound/obj-i686-pc-linux-gnu/dist/bin', '-v', '170', '-f', './httpd.js', '-e', "const _PROFILE_PATH = '/tmp/tmpL83YT4';const _SERVER_PORT = '8888'; const _SERVER_ADDR ='192.168.1.109';", '-f', './server.js']
INFO | runtests.py | Server pid: 25167
pushing directory: /tmp/tmpyZsYvd to /mnt/sdcard/tests/profile
INFO | runtests.py | Running tests: start.

FIRE PROC: '"MOZ_CRASHREPORTER=1,XPCOM_DEBUG_BREAK=stack,MOZ_CRASHREPORTER_NO_REPORT=1,NO_EM_RESTART=1,MOZ_PROCESS_LOG=/tmp/tmpToaiFopidlog,XPCOM_MEM_BLOAT_LOG=/tmp/tmpyZsYvd/runtests_leaks.log" am instrument -w -e class org.mozilla.fennec.tests.testAwesomebar org.mozilla.roboexample.test/android.test.InstrumentationTestRunner'
INFO | automation.py | Application pid: 0
0 INFO SimpleTest START
1 INFO TEST-START | testAwesomebar
2 INFO TEST-PASS | testAwesomebar | Awesomebar URL typed properly - http://mochi.test:8888/tests/robocop/robocop_blank_01.html should equal http://mochi.test:8888/tests/robocop/robocop_blank_01.html
3 INFO TEST-PASS | testAwesomebar | Awesomebar URL stayed the same - http://mochi.test:8888/tests/robocop/robocop_blank_01.html should equal http://mochi.test:8888/tests/robocop/robocop_blank_01.html
4 INFO TEST-END | testAwesomebar | finished in 13923ms
5 INFO TEST-START | Shutdown
6 INFO Passed: 2
7 INFO Failed: 0
8 INFO Todo: 0
9 INFO SimpleTest FINISHED
INFO | automation.py | Application ran for: 0:01:05.333367
INFO | automation.py | Reading PID log: /tmp/tmpToaiFopidlog
getting files in '/mnt/sdcard/tests/profile/minidumps/'
WARNING | automationutils.processLeakLog() | refcount logging is off, so leaks can't be detected!

INFO | runtests.py | Running tests: end.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

I have seen output from this test type in android staging.  Are we done with this, Are we waiting on the signing issue in bug 719697 to be verified and closed out?

Comment 7

[Hal Wine [:hwine] use NI!]

(In reply to Joel Maher (:jmaher) from comment #6)
> I have seen output from this test type in android staging.  Are we done with
> this, Are we waiting on the signing issue in bug 719697 to be verified and
> closed out?

correct

Comment 8

[Hal Wine [:hwine] use NI!]

Attachment: add robocop aware test

does clean config and runs on tegra okay. Naming is set to trigger the correct download behavior in the steps, per bug 715215

Comment 9

[Hal Wine [:hwine] use NI!]

Comment on attachment 594365 [details] [diff] [review]
add robocop aware test

tested in dev with tegra for full set of android tests. Works fine there.

r+ from aki & catlee in irc

supporting code committed with bug 715215

Comment 10

[Hal Wine [:hwine] use NI!]

http://hg.mozilla.org/build/buildbot-configs/rev/a3dc674bc562

Comment 11

[Armen [:armenzg]]

This is live now as of 1300 PST.

Comment 12

[Armen [:armenzg]]

I see the tests running:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=robocop

Unfortunately I believe the exit codes are not working properly.
I see [1]:
> 8 INFO TEST-UNEXPECTED-FAIL | test_bug720538 | Checking bottom-left corner of viewport - Color rgba(255,255,255,255) not close enough to expected rgb(0,128,0)
but the exit code is success.

Is this expected?

[1] https://tbpl.mozilla.org/php/getParsedLog.php?id=9158753&tree=Mozilla-Inbound&full=1#error0

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

we can fix this in robocop and return the proper error code.  It isn't expected for the test to fail, but the tool worked as it should.

Comment 14

[Hal Wine [:hwine] use NI!]

Also, if there are branches that do not have robocop builds, but do execute this (modified) android suite. Those branches will fail red/orange, and we should just hide that column in tbpl until the robocop changes land on the branch.

(We don't expect this to happen, I just want to document the workaround if needed.)

Comment 15

[Armen [:armenzg]]

Is this bug then FIXED?

Should now the focus go to bug 723667?

(In reply to Joel Maher (:jmaher) from comment #13)
> we can fix this in robocop and return the proper error code.  It isn't
> expected for the test to fail, but the tool worked as it should.

Do you have a Talos bug for this I can follow?

Comment 16

[Hal Wine [:hwine] use NI!]

Based on Armen's comment that this test is running, and Joel's comment that this is a robocop issue (not config), marking this fixed.

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

the bug to track the log stuff is bug 717018