Closed Bug 575777 Opened 9 years ago Closed 9 years ago

add support for running android tests on multiple devices

Categories

(Tamarin Graveyard :: Build Config, defect)

x86
macOS
defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: dschaffe, Unassigned)

Details

(Whiteboard: buildbot, build-speedup has-patch)

Attachments

(2 files, 3 obsolete files)

we can detect all connected android devices with  "adb devices" and target a specific device by using "adb -s <id> shell ...".  We should run adb devices and set python threads number to the number of devices connected in runtests.  this will speedup the acceptance tests and allow dynamically adding n numbers of devices to a run.
awesome!
Whiteboard: buildbot, build-speedup
remaining work is to update buildbot and write the setup code.  before we run we deploy avmshell and android_runner.sh to all devices in /data/app.  should do something like:
for each device:
adb -s <id> root
adb -s <id> push avmshell /data/app
adb -s <id> shell chmod 777 /data/app/avmshell
adb -s <id> push android_runner.sh /data/app
adb -s <id> shell chmod 777 /data/app/android_runner.sh
Attachment #454960 - Flags: review?(cpeyer)
Comment on attachment 454960 [details] [diff] [review]
patch: android use all connected devices

Looks good - few minor nits:
- try/except in runtests() should catch explicit exceptions.  The only exception that could be thrown would be a ValueError due to a bad cast - also add an error message in the except
- maybe add some comments in preProcessTests explaining why we set the threads
Attachment #454960 - Flags: review?(cpeyer) → review+
- modify the android_shell to remove the EXITCODE from the output
- add an adb deployer script that will deploy the passed in shell to all of the devices connected to the machine. Will also verify that the correct version of the shell was deployed, matching rev:hash from expected (relies on patch in bug# 562075)
- run-acceptance-generic-adb.sh works just like the other generic scripts (local, ssh, adb)
- use a local run-smokes.sh script

Still to do:
- update getconfig() in runtests so that a proper config string is returned, currently I think it is hardcoded:   
   arm-android-mac-tvm-unknown-Dinterp
 Should not contain _mac_ and we should be able to figure out what type of build it is (release, releasedebugger, etc)

Running tests:
1) hook up N number of android devices to machine

Running smokes:
./run-smoketests.sh <rev>

Running acceptance:
../all/run-acceptance-generic-adb.sh <rev> <shellname> <vmargs> <config> <scriptargs>

example:
../all/run-acceptance-generic-adb.sh 4895 'avmshell' '' '' 'as3/AbcDecoder/'

This will deploy the release build# 4895 on all connected devices, confirm that the shell was deployed properly and then run all of the as3/AbcDecoder/ testcases using all of the connected devices (runtests.py runs with as many threads as there are devices connected to the machine)
Attachment #455436 - Flags: review?(dschaffe)
Whiteboard: buildbot, build-speedup → buildbot, build-speedup has-patch
Comment on attachment 455436 [details] [diff] [review]
generic adb script for running acceptance

also I thought we may want to add as a cleanup step looking for/killing zombie avmshell processes on the device.  in adb the android shell supports ps, grep, awk, kill.  this should work:

 pslist=`adb -s <id> shell ps | grep avmshell | awk '{print $2}'`
if [ "$pslist != "" ]
then
adb -s <id> shell kill $pslist
Attachment #455436 - Flags: review?(dschaffe) → review+
Attached patch fix android_shell (obsolete) — Splinter Review
the android_shell was echoing out additional information which was causing some tests to fail that compare the output of the shell against a known .out file
Attachment #455446 - Flags: review?(dschaffe)
Attachment #455446 - Flags: review?(dschaffe) → review+
Attached patch rolled up patchSplinter Review
This combines all patches together.

New things to look at/review:
1) modified android_shell.sh to call a single device IF no options are passed to the script or the only option passed is '-Dversion'
2) tweaked test/utils/runtestBase.py so that if the self.avm is a shell script and contains android, then modify the cputype to be 'arm' and override self.osname to be 'android'.

These 2 changes allow runtests.py to properly determine the config string

3) change the buildbot config to use the run-acceptance-generic-adb script
Attachment #455462 - Flags: review?(dschaffe)
Attachment #454960 - Attachment is obsolete: true
Attachment #455436 - Attachment is obsolete: true
Attachment #455446 - Attachment is obsolete: true
Attachment #455462 - Flags: review?(dschaffe) → review+
Comment on attachment 455462 [details] [diff] [review]
rolled up patch

Pushed patch and follow up changes as:
4904:bf8d82bfe976 
4905:28e4dcae4acd 
4906:177642156314
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Flags: in-testsuite-
Flags: flashplayer-triage+
add the following android acceptance passes:
- release interp
- release jit
- debug

This change should still keep the android test phase under 30 minutes.
Attachment #457056 - Flags: review?(dschaffe)
Attachment #457056 - Flags: review?(dschaffe) → review+
Comment on attachment 457056 [details] [diff] [review]
additional acceptance runs

patch pushed as 4915:6770a721a013
You need to log in before you can comment on or make changes to this bug.