558126 - Runtests doesn't have a nice way to integrate tests that have expected verifyerrors

Status: VERIFIED FIXED

(Tamarin Graveyard :: Tools, enhancement, P3)

Description

[Chris Peyer]

(Forking this from Bug 555006 and email discussion)

We currently have no standard way of incorporating tests that produce expected verifyerrors.  The ways that have been used so far are:

1. Load the abc bytes (either from a checked-in abc or using a ByteArray), then wrap that load in a try/catch and catch the expected VerifyError.

2. Run an abcasm file or checked-in .abc and set it as expectedfail since we get a verifyError - problem with this is that we don't actually make sure that the error is the error we were expecting.

I think the best course of action is to add support for a .err file that resides next to the .abc_ or the abcasm source with the expected error in the err file (harness would adjust for release vs. debugger output).  I think that the majority of abc fuzzed testcases will be reproducible with abcasm.  This way we eliminate a lot of extra support files that are required for #1 above.

Comment 1

[Brent Baker]

Option #1 makes the running of the testcase easy without having to run it via runtests.py, the testcase self validates the results producing a PASS or a FAIL

Option #2 just adds to much into the testconfig.txt and does NOT actually validate the error that we are getting from running the abc file

Option "NEW" would require running of the testcase via runtests.py since it would be the piece of code that would validate the results.

I would choose Chris' NEW option of having .err files since running the abc file would actually show you the error (#1 hides the error) and we already have a precedent of having the "expected" results in a a separate file, exitcodes, so this is not something completely different.

Comment 2

[Edwin Smith]

if the .err approach avoids the need to write self-validating tests for verify errors, then its a lot easier to manage.  just generate a .abc any which way, capture the proper error, thats it.   (this would work for more than just verify errors, too)

Comment 3

[Chris Peyer]

Attachment: add .err support to runtests

Patch to add .err support

Here's a sample .err file:
# .err file defines an expected error message (should only be used for non-catchable errors)
# first line is the error
# second line is the expected exitcode (if it is non-zero)
# Note that a .err file takes precedence over a .exitcode file
VerifyError: Error #1024
exitcode: 1

as can be seen from the sample .err file, .exitcode files are ignored if a .err file is defined since the avm will never have a zero exitcode when exiting with an error

The exitcode: prefix is actually optional, an exitcode can be specified as just an integer.

Comment 4

[Dan Schaffer]

Comment on attachment 437927 [details] [diff] [review]
add .err support to runtests

I think it is cleaner to use loadBytes() with try/catch to for errors.  the .err is useful for cases that cannot use loadBytes()

Comment 5

[Chris Peyer]

pushed to tr and argo:
changeset 4372	d19eba3dd8bc