600661 - acceptance runtests.py should be able to distinguish failure modes

Status: RESOLVED WONTFIX

(Tamarin Graveyard :: Tools, enhancement)

Description

[Chris Peyer]

If a test is marked as expectedfail there is no way to distinguish as to what type of failure is expected.  We should be able to specify the type of failure (assert/testcase failure/etc).

Comment 1

[Chris Peyer]

Proposed expectedfail modes:

expectedfail : this covers any deviation in stdout
expectedfail-stderr : stderr output that is expected due to bug
expectedfail-exitcode : exitcode that is expected due to bug

In addition to the type of expectedfail, should we also be able to specify the exact type of failure in testconfig?  That way any other type of failure will still be flagged by runtests.  

e.g.:
path/testname, config, expectedfail-exitcode = 1, description & bugnumber

The testcase would then fail for any non-1 exitcode.

This would essentially be a parallel system to our specialized files (.err, .exitcode ...) that are used in the test system, except these can be defined for specific configurations.  I would think that the files are used when that is the expected correct behavior of the tests, and the testconfig string is only used when it is a failure related to a bug.

Comment 2

[Edwin Smith]

it would be helpful if the comments about testconfig.txt were included in a simple user guide page, and linked to from here:

https://developer.mozilla.org/En/Tamarin/Tamarin_Build_System_Documentation

Comment 3

[Sylvestre Ledru [:Sylvestre]]

Tamarin isn't maintained anymore. WONTFIX remaining bugs.