401921 - Need a mechanism for running tests that handles known failures.

Status: VERIFIED FIXED

(Tamarin Graveyard :: Build Config, enhancement)

Description

[Brent Baker]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.8) Gecko/20071008 Firefox/2.0.0.8
Build Identifier: 

Need to have a test harness for running the testsuite that will handle known failures that are currently being tracked via bugzilla. 

This will allow for easy identification of newly introduced failures since it will not rely on you knowning if a specific testcase was failing prior to a change.

This will give us the concept of "expected failures" and "failures". "Excepted Failures" must have a backing bugzilla number and are allowed to fail. These will be tracked via a file that will be added to the repository. While "failures" are testcases that fail without prior reason and must be investigated.

Reproducible: Always

Comment 1

[Brent Baker]

Attachment: First pass at testharness

Comment 2

[Dan Schaffer]

Comment on attachment 286861 [details] [diff] [review]
First pass at testharness

patch looks good.  we have been running the python runtests mechanism for a while.  we'll clean up the script in a subsequent checkin.

Comment 3

[Brent Baker]

Attachment: Second Pass at testrunner

Work on getting the configuration automatically parsed if it is not passed. Currently will pick out:
cputype: [x86, ppc]
ostype: [win,mac,lnx,sol]
vmtype: [release,releasedebugger]

NOTE: There currently is no way to determine if the vm is a debug version. I will look into how to figure this out.

Comment 4

[Brent Baker]

Attachment: Update testconfig.txt

Need to update the testconfig.txt now that the vmswitch section includes the raw args, making the current "-interp" into "-Dinterp"

Comment 5

[Brent Baker]

Attachment: runtest.py patch when running on PPC

Minor spelling mistake in setting the CPUTYPE for PPC