485568 - tamarin acceptance tests should work easily with the winmo emulator

Status: VERIFIED FIXED

(Tamarin Graveyard :: Virtual Machine, defect)

Description

[Dan Schaffer]

will write a python script and add code into runtestBase.py to handle running tests in the emulator with minimal setup.

Comment 1

[Dan Schaffer]

Attachment: patch for winmo emulator

Comment 2

[Dan Schaffer]

steps to run:
$ export AVM=tamarin-redux/platform/win32/Windows Mobile 6 Professional SDK (ARMV4I)/Release/avmshell.exe
$ ./runtests.py --threads=2

pre-reqs:
1) build Release/Winmobile 6 version of wmrunner.exe (from utils/wmremote/.sln)
by default looks for wmrunner.exe in utils/wmremote/Windows Mobile 6
Professional SDK (ARMV4I)/Release/wmrunner.exe
or you could download \\asteam\builds\downloads\dschaffe\wmremote\wmremote.exe
and export CERUNNER to the exe
2) other prereqs are DeviceEmulator and V614 images

Comment 3

[Dan Schaffer]

the patch should include everything to build the wmrunner.exe tool and run acceptance/runtests.py like against ordinary tests.

also once an emulator is launched (by runtests.py) you can keep it running or close it and runtests.py will only relaunch if closed.

Comment 4

[Dan Schaffer]

Attachment: patch

update runtestBase.py to use 'file avmshell.exe' to determine the config is arm-winmobile.  also minor update to allow distributing a test among multiple hosts.  I found 2 emulators (50m) to run faster than 4 emulators (52m) on a host.  Although having 4 emulators running simultaneously looks cooler.  The only way for dramatic speedup seems to be multiple hosts.

Comment 5

[Rick Reitmaier]

Comment on attachment 370026 [details] [diff] [review]
patch

reviewed up to runtest() so far, and had these comments...

- shutil does this mean that requirements have been raised to 2.6.1?
- setupCEEmulators()  on non-win platform what happens?  silently ignore errors when processing EMULATORTHREADS; not good.
- why is a special call to 'asc version.as' needed?
- a comment at top of method describing how it works would be good.

Comment 6

[Dan Schaffer]

- 2.5.2 has shutil.  Does it not work < 2.6.1?  I can try to run on our 2.6.1.
- setupCEEmulators() only runs if the executable is windows arm.  I could exit if platform is not windows.
- yeah I should reuse the asc code in runtestBase.py.  The purpose is to get the version string out of the executable.
- comment? what's that?
thanks for the feedback.  I'm looking at some issues.  I don't think the 614 emulator image works on 5.0 shell builds.  Works when I build my own 614 shell executables.  Also if you do not have file installed in cygwin the auto-detection does not work.  I will fix.

Comment 7

[Dan Schaffer]

Attachment: patch

Comment 8

[Edwin Smith]

generally: 

- runtestBase.py in need of factoring; could more of the winmo emulator manager   code be factored out, and the testrunner be packaged so that it just looks to the test runner like a vm?  (litmus test... did this file just get easier to maintain or harder to maintain.)

- new files need copyright headers, and modelines if your feeling generous

- a howto or readme here in the code will be much better
  than a wiki.  or, good comments spelling out exactly what software
  you need installed, and exactly what commands to run, to run the
  acceptance tests locally on N cores.

Comment 9

[Brent Baker]

- generation of version.as|abc should be done in the util/emulator_files directory and that directory should be added to .hgignore

- should update solution file and default path to wmrunner.exe to just "Release" and not the full "Windows Mobile 6 Professional SDK (ARMV4I)/Release"

- should have a call to sleep() in the while len(versions)>0:[line 1241] in runtestBase.py when looking for the version.log files, so that python doesn't suck up the CPU waiting for the emulator to start up

- is it possible to capture ctrl+c in th setupCEEmulators() so that if there is a message about there being a lock/running file that needs to be removed there is no stacktrace if I stop the script

- should the while loop in wmemulatorshell.py that is trying to obtain a lock have a sleep at the end. 99.9% of the time #emulator == #threads so this should never be a problem, but you can set the number of each independently so it is possible that you can have more threads than emulators, so might be a good safe guard to not spin the CPU here if you can't instantly get a lock.

Comment 10

[Dan Schaffer]

- version.as|abc is deleted after it is used
- update wmrunner.exe to build to the Release
- added sleep when waiting for emulator to stop [1241]
- added small sleep in wmemulatorshell.py
- did not address the Ctrl-C, any locks are removed the next time the emulator runs,  the only open issue is when the emulator is killed and the running.txt in the shared directory exists.  prints a warning when detects a running emulator from runtests.py

Comment 11

[Dan Schaffer]

from comment #8:
- the setup is to set avmplus_arm.exe as the AVM.  When runtests.py detects the filetype matches arm (uses file command) the ../utils/wmemulatorshell.py replaces AVM internally.  The wmemulatorshell.py acts a vm.  The setupCEEmulators() function starts the emulators and tests they are working (one-time setup) before entering the main runtests.py test loop.  
- added MPL headers
- README, Brent suggested adding to utils/wmremote/readme.txt,  could add instructions to MDC.  The setup is really just export AVM=avmplus_arm.exe,  If the emulator or image is not in the default location runtests.py will tell environment variable to set.  Otherwise works same as ./runtests.py

Comment 12

[Dan Schaffer]

Attachment: patch

Comment 13

[Dan Schaffer]

Attachment: patch for buildbot script

add a buildbot script for running acceptance tests on winmo emulator.

added a test_emulator_threads variable since test_threads will still be used for ceremoteshell running on the device.

Comment 14

[Brent Baker]

Comment on attachment 370931 [details] [diff] [review]
patch

This has been pushed as 1724:c321a53ff488

Comment 15

[Brent Baker]

Comment on attachment 378603 [details] [diff] [review]
patch for buildbot script

+ line 57 of run-acceptance-release-arm-emulator.sh should be looking for $shell_release_arm instead of $shell_release
+ wmrunner.exe is not in build/buildbot/slaves/all/tools
+ please make sure that test/util/emulator_files/* is added to the .hgignore

Comment 16

[Dan Schaffer]

pushed into tr-redux 1944 : 9738e723ccf4

Comment 17

[Brent Baker]

Attachment: Remove static config name

Remove the setting of the config in run(), should just rely on determineConfig() to calculate the config string if it is not passed in.

Current problem is that without this patch the config never accounts for vmargs that are passed in, so there is no way to differentiate between hybrid, interp or jit runs.

Comment 18

[Brent Baker]

Without patch #6 the config is always:

-> current configuration: arm-winmobile-emulator-tvm

even when passing --vmargs=-Ojit, with the patch the config is correct:

-> current configuration: arm-winmobile-emulator-tvm-release-Ojit

Comment 19

[Dan Schaffer]

Comment on attachment 385081 [details] [diff] [review]
Remove static config name

r+, it would be nice to be able to auto-detect the arm executable and still add the -vmargs to the end of the self.config string.

Comment 20

[Dan Schaffer]

also please update the utils/wmremote/readme.txt with a note to require passing  --config=arm-winmobile-emulator-tvm-release to runtests.py when running with the emulator.

Comment 21

[Brent Baker]

There is no need to add the comment since the config is properly determined in determineConfig(). Running acceptance via the emulator:

./runtests.py --nohtml --vmargs=-Ojit

will produce the correct config string of "arm-winmobile-emulator-tvm-release-Ojit"

Comment 22

[Dan Schaffer]

yeah it looks like the 2 lines you removed were unnecessary since the determineConfig() is working properly.  Thanks.

Comment 23

[Brent Baker]

(attachment 385081 [details] [diff] [review]) pushed 2050:7d669d4ed415

Comment 24

[Dan Schaffer]

closing bug by marking verified/fixed.