create a robocop test type for 'android'

RESOLVED FIXED

Status

Release Engineering
General
RESOLVED FIXED
6 years ago
4 years ago

People

(Reporter: jmaher, Assigned: hwine)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [android_tier_1])

Attachments

(1 attachment)

(Reporter)

Description

6 years ago
in bug 715215, we install the robocop.apk.  Now we need to create a test type robocop to replace browser-chrome.

This would require a command similar to browser-chrome, but without the "--browser-chrome --test-path=mobile" and instead a "--robocop=robocop.ini".
Hal, I know you're working on one part of this, so I'm opportunistically assigning this to you. Throw it back if that's wrong.
Assignee: nobody → hwine
(Assignee)

Comment 2

6 years ago
(In reply to Joel Maher (:jmaher) from comment #0)
> Now we need to create a test
> type robocop to replace browser-chrome.
> 
> This would require a command similar to browser-chrome, but without the
> "--browser-chrome --test-path=mobile" and instead a "--robocop=robocop.ini".

Joel - once bug 719697 get's resolved, please post the final exact command line that works, and I'll pick this up.
Depends on: 715215, 719697
Whiteboard: [android_tier_1]
(Reporter)

Comment 3

6 years ago
how close are we to landing this.  I am testing a final patch for bug 719697, so I wanted to make sure we were ready to land this as soon as my patch lands.
(Reporter)

Comment 4

6 years ago
This command gets the tests running:
python runtestsremote.py --app=org.mozilla.fennec_jmaher --xre-path=/home/jmaher/mozilla/inbound/obj-i686-pc-linux-gnu/dist/bin/ --deviceIP=192.168.1.100 --robocop=robocop.ini
(Reporter)

Comment 5

6 years ago
Here is some output:
jmaher@jmaher-MacBookPro:~/Downloads/robo/mochitest$ python runtestsremote.py --app=org.mozilla.fennec --xre-path=/home/jmaher/mozilla/inbound/obj-i686-pc-linux-gnu/dist/bin/ --deviceIP=192.168.1.100 --robocop=robocop.ini
unable to execute 'cp' on device; consider installing busybox from Android Market
NOT running as root
adbd cannot run as root in production builds
reconnecting socket
removing file: /sdcard/fennec_ids.txt
removing file: /sdcard/robotium.config
INFO | runtests.py | Installing extension at /home/jmaher/Downloads/robo/mochitest/extensions/specialpowers to /tmp/tmpyZsYvd.
INFO | runtests.py | Installing extension at /home/jmaher/Downloads/robo/mochitest/extensions/workerbootstrap to /tmp/tmpyZsYvd.
INFO | runtests.py | Installing extension at /home/jmaher/Downloads/robo/mochitest/extensions/worker to /tmp/tmpyZsYvd.
INFO | runtests.py | Installing extension at /home/jmaher/Downloads/robo/mochitest/extensions/roboextender@mozilla.org to /tmp/tmpyZsYvd.
pushing directory: /tmp/tmpyZsYvd to /mnt/sdcard/tests/profile
args: ['/home/jmaher/mozilla/inbound/obj-i686-pc-linux-gnu/dist/bin/xpcshell', '-g', '/home/jmaher/mozilla/inbound/obj-i686-pc-linux-gnu/dist/bin', '-v', '170', '-f', './httpd.js', '-e', "const _PROFILE_PATH = '/tmp/tmpL83YT4';const _SERVER_PORT = '8888'; const _SERVER_ADDR ='192.168.1.109';", '-f', './server.js']
INFO | runtests.py | Server pid: 25167
pushing directory: /tmp/tmpyZsYvd to /mnt/sdcard/tests/profile
INFO | runtests.py | Running tests: start.

FIRE PROC: '"MOZ_CRASHREPORTER=1,XPCOM_DEBUG_BREAK=stack,MOZ_CRASHREPORTER_NO_REPORT=1,NO_EM_RESTART=1,MOZ_PROCESS_LOG=/tmp/tmpToaiFopidlog,XPCOM_MEM_BLOAT_LOG=/tmp/tmpyZsYvd/runtests_leaks.log" am instrument -w -e class org.mozilla.fennec.tests.testAwesomebar org.mozilla.roboexample.test/android.test.InstrumentationTestRunner'
INFO | automation.py | Application pid: 0
0 INFO SimpleTest START
1 INFO TEST-START | testAwesomebar
2 INFO TEST-PASS | testAwesomebar | Awesomebar URL typed properly - http://mochi.test:8888/tests/robocop/robocop_blank_01.html should equal http://mochi.test:8888/tests/robocop/robocop_blank_01.html
3 INFO TEST-PASS | testAwesomebar | Awesomebar URL stayed the same - http://mochi.test:8888/tests/robocop/robocop_blank_01.html should equal http://mochi.test:8888/tests/robocop/robocop_blank_01.html
4 INFO TEST-END | testAwesomebar | finished in 13923ms
5 INFO TEST-START | Shutdown
6 INFO Passed: 2
7 INFO Failed: 0
8 INFO Todo: 0
9 INFO SimpleTest FINISHED
INFO | automation.py | Application ran for: 0:01:05.333367
INFO | automation.py | Reading PID log: /tmp/tmpToaiFopidlog
getting files in '/mnt/sdcard/tests/profile/minidumps/'
WARNING | automationutils.processLeakLog() | refcount logging is off, so leaks can't be detected!

INFO | runtests.py | Running tests: end.
(Reporter)

Comment 6

6 years ago
I have seen output from this test type in android staging.  Are we done with this, Are we waiting on the signing issue in bug 719697 to be verified and closed out?
(Assignee)

Comment 7

6 years ago
(In reply to Joel Maher (:jmaher) from comment #6)
> I have seen output from this test type in android staging.  Are we done with
> this, Are we waiting on the signing issue in bug 719697 to be verified and
> closed out?

correct
(Assignee)

Comment 8

6 years ago
Created attachment 594365 [details] [diff] [review]
add robocop aware test

does clean config and runs on tegra okay. Naming is set to trigger the correct download behavior in the steps, per bug 715215
(Assignee)

Comment 9

6 years ago
Comment on attachment 594365 [details] [diff] [review]
add robocop aware test

tested in dev with tegra for full set of android tests. Works fine there.

r+ from aki & catlee in irc

supporting code committed with bug 715215
Attachment #594365 - Flags: review+
Attachment #594365 - Flags: checked-in+
(Assignee)

Comment 10

6 years ago
http://hg.mozilla.org/build/buildbot-configs/rev/a3dc674bc562

Comment 11

6 years ago
This is live now as of 1300 PST.

Comment 12

6 years ago
I see the tests running:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=robocop

Unfortunately I believe the exit codes are not working properly.
I see [1]:
> 8 INFO TEST-UNEXPECTED-FAIL | test_bug720538 | Checking bottom-left corner of viewport - Color rgba(255,255,255,255) not close enough to expected rgb(0,128,0)
but the exit code is success.

Is this expected?

[1] https://tbpl.mozilla.org/php/getParsedLog.php?id=9158753&tree=Mozilla-Inbound&full=1#error0
(Reporter)

Comment 13

6 years ago
we can fix this in robocop and return the proper error code.  It isn't expected for the test to fail, but the tool worked as it should.
(Assignee)

Comment 14

6 years ago
Also, if there are branches that do not have robocop builds, but do execute this (modified) android suite. Those branches will fail red/orange, and we should just hide that column in tbpl until the robocop changes land on the branch.

(We don't expect this to happen, I just want to document the workaround if needed.)

Comment 15

6 years ago
Is this bug then FIXED?

Should now the focus go to bug 723667?

(In reply to Joel Maher (:jmaher) from comment #13)
> we can fix this in robocop and return the proper error code.  It isn't
> expected for the test to fail, but the tool worked as it should.

Do you have a Talos bug for this I can follow?
(Assignee)

Comment 16

6 years ago
Based on Armen's comment that this test is running, and Joel's comment that this is a robocop issue (not config), marking this fixed.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
(Reporter)

Comment 17

6 years ago
the bug to track the log stuff is bug 717018
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.