If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

runtests.py should capture the exitcode and stderr

VERIFIED FIXED in flash10.1

Status

Tamarin
Virtual Machine
VERIFIED FIXED
9 years ago
9 years ago

People

(Reporter: Brent Baker, Assigned: Dan Schaffer)

Tracking

unspecified
flash10.1
Bug Flags:
flashplayer-qrb +
flashplayer-triage +

Details

Attachments

(1 attachment, 2 obsolete attachments)

(Reporter)

Description

9 years ago
The current implementation of run_pipe() does not track the exitcode or the stderr, both of these should be returned along with the stdout.

The acceptance testsuite should fail any test that does not return a zero exitcode, unless a "testname.exitcode" file exists and the exitcode in the file matches what was returned.
Flags: flashplayer-triage+
Flags: flashplayer-qrb?

Updated

9 years ago
Assignee: nobody → dschaffe
Status: NEW → ASSIGNED
Flags: flashplayer-qrb? → flashplayer-qrb+
Target Milestone: --- → flash10.x
(Assignee)

Comment 1

9 years ago
Created attachment 363890 [details] [diff] [review]
patch to check exit codes and stderr of runtests

the patch reads stderr and exitcode from test result
exit code is expected to be 0 unless .exitcode file exists
stderr is expected to be [] empty
open issues:
- what should we do about expected failures for exit code? add entry for testconfig.txt
- is there a case where stderr is not empty?  we could create a .stderr file
Attachment #363890 - Flags: review?(brbaker)
(Assignee)

Comment 2

9 years ago
hmm I'm seeing the performance tests stop working with the patch applied.  investigating...
(Reporter)

Updated

9 years ago
Attachment #363890 - Flags: review?(brbaker) → review-
(Reporter)

Comment 3

9 years ago
Comment on attachment 363890 [details] [diff] [review]
patch to check exit codes and stderr of runtests

need to update all calls into run_pipe() in the performance suite and any other suite that may make direct calls to it.
(Assignee)

Comment 4

9 years ago
Created attachment 363948 [details] [diff] [review]
patch works with performance tests now

- checks the exit codes for performance tests
- performance/runtests.py now exits with 1 if any perf tests fail
Attachment #363890 - Attachment is obsolete: true
(Assignee)

Updated

9 years ago
Attachment #363948 - Attachment is patch: true
Attachment #363948 - Attachment mime type: application/octet-stream → text/plain
Attachment #363948 - Flags: review?(brbaker)
(Assignee)

Comment 5

9 years ago
Created attachment 364109 [details] [diff] [review]
new patch fixes the abcasm problems and typo in performance/runtests.py

added new patch fixes the abcasm problems and typo in performance/runtests.py
Attachment #363948 - Attachment is obsolete: true
Attachment #364109 - Flags: review?(brbaker)
Attachment #363948 - Flags: review?(brbaker)
(Reporter)

Updated

9 years ago
Attachment #364109 - Flags: review?(brbaker) → review+
(Assignee)

Comment 6

9 years ago
submitted to tamarin-redux in 1514	81bab74661e0
Status: ASSIGNED → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → FIXED
(Reporter)

Updated

9 years ago
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.