Closed Bug 475055 Opened 15 years ago Closed 15 years ago

runtests.py should capture the exitcode and stderr

Categories

(Tamarin Graveyard :: Virtual Machine, defect)

defect
Not set
normal

Tracking

(Not tracked)

VERIFIED FIXED
flash10.1

People

(Reporter: brbaker, Assigned: dschaffe)

Details

Attachments

(1 file, 2 obsolete files)

The current implementation of run_pipe() does not track the exitcode or the stderr, both of these should be returned along with the stdout.

The acceptance testsuite should fail any test that does not return a zero exitcode, unless a "testname.exitcode" file exists and the exitcode in the file matches what was returned.
Flags: flashplayer-triage+
Flags: flashplayer-qrb?
Assignee: nobody → dschaffe
Status: NEW → ASSIGNED
Flags: flashplayer-qrb? → flashplayer-qrb+
Target Milestone: --- → flash10.x
the patch reads stderr and exitcode from test result
exit code is expected to be 0 unless .exitcode file exists
stderr is expected to be [] empty
open issues:
- what should we do about expected failures for exit code? add entry for testconfig.txt
- is there a case where stderr is not empty?  we could create a .stderr file
Attachment #363890 - Flags: review?(brbaker)
hmm I'm seeing the performance tests stop working with the patch applied.  investigating...
Attachment #363890 - Flags: review?(brbaker) → review-
Comment on attachment 363890 [details] [diff] [review]
patch to check exit codes and stderr of runtests

need to update all calls into run_pipe() in the performance suite and any other suite that may make direct calls to it.
- checks the exit codes for performance tests
- performance/runtests.py now exits with 1 if any perf tests fail
Attachment #363890 - Attachment is obsolete: true
Attachment #363948 - Attachment is patch: true
Attachment #363948 - Attachment mime type: application/octet-stream → text/plain
Attachment #363948 - Flags: review?(brbaker)
added new patch fixes the abcasm problems and typo in performance/runtests.py
Attachment #363948 - Attachment is obsolete: true
Attachment #364109 - Flags: review?(brbaker)
Attachment #363948 - Flags: review?(brbaker)
Attachment #364109 - Flags: review?(brbaker) → review+
submitted to tamarin-redux in 1514	81bab74661e0
Status: ASSIGNED → RESOLVED
Closed: 15 years ago
Resolution: --- → FIXED
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: