Need a mechanism for running tests that handles known failures.

VERIFIED FIXED

Status

Tamarin
Build Config
--
enhancement
VERIFIED FIXED
10 years ago
8 years ago

People

(Reporter: Brent Baker, Unassigned)

Tracking

Details

Attachments

(3 attachments, 1 obsolete attachment)

(Reporter)

Description

10 years ago
User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.8) Gecko/20071008 Firefox/2.0.0.8
Build Identifier: 

Need to have a test harness for running the testsuite that will handle known failures that are currently being tracked via bugzilla. 

This will allow for easy identification of newly introduced failures since it will not rely on you knowning if a specific testcase was failing prior to a change.

This will give us the concept of "expected failures" and "failures". "Excepted Failures" must have a backing bugzilla number and are allowed to fail. These will be tracked via a file that will be added to the repository. While "failures" are testcases that fail without prior reason and must be investigated.

Reproducible: Always
(Reporter)

Comment 1

10 years ago
Created attachment 286861 [details] [diff] [review]
First pass at testharness
Attachment #286861 - Flags: review?(dschaffe)

Comment 2

10 years ago
Comment on attachment 286861 [details] [diff] [review]
First pass at testharness

patch looks good.  we have been running the python runtests mechanism for a while.  we'll clean up the script in a subsequent checkin.
Attachment #286861 - Flags: review?(dschaffe) → review+
(Reporter)

Comment 3

10 years ago
Created attachment 292600 [details] [diff] [review]
Second Pass at testrunner

Work on getting the configuration automatically parsed if it is not passed. Currently will pick out:
cputype: [x86, ppc]
ostype: [win,mac,lnx,sol]
vmtype: [release,releasedebugger]

NOTE: There currently is no way to determine if the vm is a debug version. I will look into how to figure this out.
Attachment #286861 - Attachment is obsolete: true
Attachment #292600 - Flags: review?(dschaffe)
(Reporter)

Comment 4

10 years ago
Created attachment 292601 [details] [diff] [review]
Update testconfig.txt

Need to update the testconfig.txt now that the vmswitch section includes the raw args, making the current "-interp" into "-Dinterp"
Attachment #292601 - Flags: review?(dschaffe)

Updated

10 years ago
Attachment #292600 - Flags: review?(dschaffe) → review+

Updated

10 years ago
Attachment #292601 - Flags: review?(dschaffe) → review+
(Reporter)

Comment 5

10 years ago
Created attachment 292767 [details] [diff] [review]
runtest.py patch when running on PPC

Minor spelling mistake in setting the CPUTYPE for PPC
(Reporter)

Updated

10 years ago
Attachment #292767 - Flags: review?(dschaffe)

Updated

10 years ago
Attachment #292767 - Flags: review?(dschaffe) → review+
(Reporter)

Updated

10 years ago
Status: NEW → RESOLVED
Last Resolved: 10 years ago
Resolution: --- → FIXED

Updated

8 years ago
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.