572861 - enable csv output for performance runtests

Status: VERIFIED FIXED

(Tamarin Graveyard :: Tools, defect)

Description

[Chris Peyer]

need to have csv output of all raw test result data

Comment 1

[Chris Peyer]

Attachment: initial csv output patch

Patch adds ability to output perf data to a csv file (while also displaying the standard output to screen).  Each line in the csv file is a single run for a single avm and metric.  This data format works very nicely with excel when using a pivottable to analyze the data.

Also added a dictionary that stores info about each metric encountered.  This data is currently not used, but I envision that in addition to storing whether the metric has the largerIsFaster bit set, also being able to store the metric unit which could be output to the csv file.

The current setup allows for multiple metrics per avm/test, however the harness is not currently setup to allow that to be scraped from the test output - a new bug will be filed for that.

Comment 2

[Edwin Smith]

Comment on attachment 454163 [details] [diff] [review]
initial csv output patch

I'd like columns that provide as much detail as possible; at least the invocation string (path+options) would be a starting point.  build options would be a plus too.

Is there any way to generate a CSV with >1 metric per vm invocation yet?  Looking forward to experimenting with analysis.  A typical scenario would be charting performance and heap size together in an X-Y plot, for two vm configurations. (two lines, or two series of x-y points).  the critical part is that each (mem,perf) pair be associated together as outputs from a single run.

Comment 3

[Edwin Smith]

Comment on attachment 454163 [details] [diff] [review]
initial csv output patch

on the code, it looks basically right.  Do you gain anything by keeping csvData1 and csvData2 separate?  CSV data by its very nature should be self describing (one  or more of the fields should indicate which vm it was), and there's only one output file.

Comment 4

[Edwin Smith]

The hardcoded column names and order is going to get messy pretty fast.  Python's "csv" module supports reading/writing csv where each row is represented as an associative object in memory; each invocation of the VM would produce an object:  { avm:"/path/avmplus blah", metric: "v8", ... } and the write code just needs a sequence of objects.  DictWriter has logic to write out a default value for rows that are missing a particular field.

Comment 5

[Chris Peyer]

Pushed functionality via patch for 572860.  Leaving bug open as there is probably more work to be done on csv output.

Comment 6

[Chris Peyer]

Closing bug out.  Any additional csv issues should be filed in a new bug.