buildbot: smoke test results are not granular enough

VERIFIED FIXED

Status

Tamarin
Build Config
VERIFIED FIXED
9 years ago
9 years ago

People

(Reporter: Chris Peyer, Assigned: Dan Schaffer)

Tracking

Details

Attachments

(1 attachment, 1 obsolete attachment)

3.83 KB, patch
Brent Baker
: review+
Details | Diff | Splinter Review
(Reporter)

Description

9 years ago
When smoke tests fail, the script only indicates that a testcase failed, but does not provide the Description, Expected and Actual values from the testcase.

Comment 1

9 years ago
The script should also be updated so that the list of failed calls are shown with the proper tokens in place so that it can simply be pasted into the runsmokes.txt 

http://10.171.22.12:8760/builders/mac64-ppc-smoke/builds/16/steps/SmokeTest/logs/stdio

list of test failures:
cd ./acceptance;./runtests.py --nohtml --avm=/Users/build/buildbot/tamarin-redux/mac64-ppc/tamarin-redux/../builds/1851-8bbcef0c50f4/mac/avmshell_64_ppc abcasm

It would be more useful if this was:
list of test failures:
cd ./acceptance;./runtests.py --nohtml --avm=${avmr} abcasm
(Assignee)

Comment 2

9 years ago
Created attachment 376400 [details] [diff] [review]
patch

patch displays output with tokens still in strings,  failures captures stdout+stderr and prints when showing the failure list at the bottom of the report.  I'm wondering if displaying the failure output will change the buildsteps ability to count the passes and failures.  we'll have to investigate.
Assignee: brbaker → dschaffe
Status: NEW → ASSIGNED
Attachment #376400 - Flags: review?(brbaker)
(Assignee)

Comment 3

9 years ago
Created attachment 376487 [details] [diff] [review]
patch

this patch has been getting pretty thoroughly tested by my attempts to fix 491724.
- prints test command without env var substitutions
- at end, prints values of all env var referenced
- if failures:
  - print test command with and without env vars substituted
  - print stdout and stderr
  - replace "passes" with " passes" to avoid buildstep counting passes and failures so does not match beginning of line, same for failures, unexp pass, ...
- bug fixes
  expected failure, unexpected pass, assertions
Attachment #376400 - Attachment is obsolete: true
Attachment #376487 - Flags: review?(brbaker)
Attachment #376400 - Flags: review?(brbaker)

Updated

9 years ago
Attachment #376487 - Flags: review?(brbaker) → review+
(Assignee)

Comment 4

9 years ago
submitted in tamarin-redux 1865	83405cbf9278
(Assignee)

Updated

9 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → FIXED
(Assignee)

Updated

9 years ago
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.