535100 - Test harness unable to catch catastrophic errors

Status: VERIFIED FIXED

(Tamarin Graveyard :: Build Config, defect, P1)

Description

[Lars T Hansen]

The test harness is unable to catch bus errors, leading to situations where virtually all tests fail but the test suite run is judged a success.

To reproduce:
- check out TR 3329:d7162814f1f5
- apply the patch (attachment #417700 [details] [diff] [review]) from bug #528977
- build a DD build using Xcode
- run the test harness on all tests in test/acceptance

Observed result:
test run PASSED!
Tests complete at 2009-12-16 10:48:40.386523
Start Date: 2009-12-16 10:48:24.710717
End Date  : 2009-12-16 10:48:40.386523
Test Time : 0:00:15.675806
passes               : 1
failures             : 0
tests skipped        : 7

Expected result:
The test run should fail; all tests crash with bus errors due to a bug in that patch.

Comment 1

[Lars T Hansen]

Attachment: The attachment in question, copied from the other bug

Comment 2

[Nicholas Nethercote [inactive]]

AFAICT it also fails to realise that assertion failures are bad.

Comment 3

[Lars T Hansen]

(In reply to comment #2)
> AFAICT it also fails to realise that assertion failures are bad.

If so that's a major regression; it's been able to recognize those in the past.

Comment 4

[Lars T Hansen]

Same as bug #539460, but a little more info here.

Comment 5

[Lars T Hansen]

Escalating.

Comment 6

[Trevor Baker]

status?

Comment 7

[Chris Peyer]

Fixed with Bug 544997.  Verified that Bug 544997 patch fixes the issue when applying patch for this bug:

executing: /Users/cpeyer/asc/shell   as3/DescribeType/describeType.abc
1 running as3/DescribeType/describeType.as  
   unexpected exit code expected:0 actual:-10 Signal Name: SIGBUS FAILED!
   captured output:
   FAILED passes:0 fails:1 unexpected passes: 0 expected failures: 0

Comment 8

[Brent Baker]

Chris has already verified this bug