490900 - create a smoke test script for buildbot

Status: VERIFIED FIXED

(Tamarin Graveyard :: Build Config, defect, P3)

Description

[Dan Schaffer]

goal is to create a buildstep and blocking buildbot stage to quickly run some tests.  the test list can be easily changed and may include any testsuite, performance test.  if any smokes fail the testing step is not run and the build is considered failed.

- the smoketest script reads an input file containing commands to call testsuites with a specific test or group of tests
- each line of the input file is run until a time limit has been reached (e.g. 2 min)
- environment variables avmr, avmd, avmrd, avmdd represent shell builds, all builds should be tested
- the test list can be changed to include any recent failures to regression test

Comment 1

[Dan Schaffer]

Attachment: patch

Comment 2

[Dan Schaffer]

Attachment: patch2

new attachment changes input file format to: cd ./acceptance; ./runtests.py ...

using, subprocess.Popen shell=True, setting cwd is not necessary, cd sets the directory and wildcards for test filenames works.  This format is better since failures can be easily reproduced by pasting the commands.

I will add another patch for a separate winmobile smoketest buildbot script and input file,  the script will deploy the release shell on the device and set AVM to the ceremoteshell.

Comment 3

[Brent Baker]

Comment on attachment 375338 [details] [diff] [review]
patch2

suggest breaking apart some of the groups, such as running all of mmgc as "one" test. This caused my machine to actually run for 3:03 instead of 2:00 since it run the mmgc tests with a debug VM for over a minute.

Comment 4

[Dan Schaffer]

broke up the groups further to avoid large runs.  Made a few other minor fixes.   submitted in 1821 : 7d814d0d98cc

Comment 5

[Dan Schaffer]

Attachment: patch to make 1 pass or 1 fail per test

patch to produce 1 pass or 1 fail per smoke test line in runsmokes.txt instead of aggregating passes and fails.

Comment 6

[Brent Baker]

Comment on attachment 375613 [details] [diff] [review]
patch to make 1 pass or 1 fail per test

This seems like you could increment the pass and the fail at the same time, they need to be exclusive, if there is a single fail, then it is a fail, even if there is a pass.

Comment 7

[Dan Schaffer]

Attachment: patch to make 1 pass or 1 fail per test #2

update to patch.  We already made sure if passes==0 increment fails.  changed to else should work so each test in smoketest is 1 pass or 1 fail.

Comment 8

[Dan Schaffer]

patched pushed.

Comment 9

[Dan Schaffer]

closing bug by marking verified/fixed.