Each test harness puts out a different set of numbers, parts of which appear to be basically useless. The 3rd number in the jstests suite is skipped tests: a number that nobody cares about. The last number in the jittests suite appears to be the total number of tests run: another number nobody cares about.
IMHO, the only numbers that should show up are "tests failed that should not have failed" and "tests passed that should not have passed". This would make it very easy and intuitive to check the status of a test run: not zero == not good.
We should also print a legend for these numbers to the help text.
jit-tests also includes information about timeouts. I find this useful when running with gczeal. Lots of tests time out, but I'm really only interested in whether actual failures have taken place. It would be nice if we could preserve that.
Ah, I did not know that jit-tests did that. I think we should definitely make jstests do that as well.
Links to source files, please?
Given that I've marked this as a [good first bug] I should also probably say what we need here :-). What we would like to have is:
[ nun-unexpected-fail-or-pass | num-timeout ]
This is where we update the counters that get displayed:
You can update the help text by calling methods on OptionParser:
Created attachment 642814 [details] [diff] [review]
I think all we want here is to add a fourth field (in the 3rd position) for timeouts. I'm going to leave the SKIP count in the last position, because this is more use than the total count that jittest shows.
Comment on attachment 642814 [details] [diff] [review]
Review of attachment 642814 [details] [diff] [review]:
@@ +87,5 @@
> self.n = 0
> self.pb = None
> if not options.hide_progress:
> + self.pb = ProgressBar('', testcount, 21)
I know this is picky, but any chance you could replace this magic number with something based on $COLUMNS (if set, else 80 or something)? It's fine if you just want to land it. (And no, I won't ask you to handle SIGWINCH or whatever.)
Wait! No! Never mind, this is the label width. The thing I'm talking about is currently a hardcoded constant 64 in progressbar.py. Ok, so let me change this request to use len('[%4d|%4d|%4d|%4d]' % (0,0,0,0)) instead. I just don't like the magic number.
@@ +123,2 @@
> + self.counts['SKIP'] += 1
Much, much nicer