726403 - test/performance runtests.py : set-difference( timing-fails, mem-fails ) != NULL.

Status: RESOLVED WORKSFORME

(Tamarin Graveyard :: Tools, defect)

Description

[Felix S. Klock II [:pnkfelix, :fklock]]

Forked off of Bug 725955, comment 15:

"the [output from runtests.py] data for some reason will only contain *just* time results for a few benchmarks and *just* memory results for a few others"

This is because some of the benchmarks produce an error when run without -m but not with -m (or vice versa), and so the final set of data has differing sets of "keys" via which to look up the result measurements.

Its a bug that any of the benchmarks are stil producing errors in any context.  But the fact that the use of the -m flag causes a divergence in the set is even worse in some sense, IMO.

Comment 1

[Felix S. Klock II [:pnkfelix, :fklock]]

(further poking indicates this may be pilot error, not-a-bug.  waiting until i get a chance to double-check properly before I close it though.)