833525 - talos summary numbers are not easily connectable to the numbers displayed post-test

Status: RESOLVED FIXED

(Testing :: Talos, defect)

Description

[Nathan Froyd [:froydnj]]

I was running some Talos tests locally for a compilers comparison and am getting numbers that are difficult to interpret.  For dromaeo dom/css, I get per-page numbers in the results file, which is good, but looking at the raw output, I see things like:

NOISE: |0;attr.html;1307;1311;1312;1316;1317;672;672;1027;1028;1028;409;427.57242757242756;427.57242757242756;429.14171656686625;429.14171656686625;666.3336663336663;668;670;672.3276723276723;672.3276723276723;238.28514456630108;242.54473161033798;256.7432567432567;264.73526473526476;267;190;203.77733598409543;259.7402597402597;264;266.4015904572565
NOISE: |1;modify.html;374.6253746253746;381.6183816183816;382.61738261738265;385;388;76.70182166826463;83.33333333333333;84.55882352941177;85.6353591160221;85.82834331337325;130.60817547357925;141.71656686626747;143.14115308151094;143.42629482071712;146.70658682634732;151.54536390827516;151.84815184815184;152.54237288135593;153.23383084577114;153.38645418326692;746;746;747;747;747.2527472527472;567;569;586.4135864135864;589;590
NOISE: |2;query.html;258.22532402791626;261.2163509471585;341;351.59362549800795;401.39442231075697;1733;1733;1740;1741;1745;67060;68602;69052;69919;70459;69871;70567;70578;71628;71948;71131;72209;72437;73395;74812;72495;72891;74327;75375;77046;123332;126298;126365;126697;127514;598;655;680.3196803196803;686;690;1756;1758;1758;1759;1759

and I have no idea:

- what those numbers mean;
- how they relate to the single per-page number I get in the results file;

and the extreme range of values looks--at least at first sight--quite worrisome: 2x-5x for attr.html and modify.html, orders of magnitude (!) for query.html.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

For graph server we cannot upload a series of numbers per page.  Because of that and the post processing that is done on the numbers after uploading it becomes confusing.  After looking into modifying graph server we found it to be just as much work to create datazilla.

I think to answer you question we need the test authors to chime in.  Every test should have an owner and that owner should know what the numbers mean and what we are testing.  For dromaeo* tests this is the js team.

Comment 2

[David Mandelin [:dmandelin]]

Nathan, are you wondering what those numbers mean in the sense of "what code is being measured" or in the sense of "are they individual test runs, averages, etc."?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Currently on the wiki page we have documented all the tests, the data they collect (including examples), and how it is summarized.  There are links to explain the summarization and to find it in the code.

While this could be more discoverable- I think for purposes of self discovery and investigation there is enough information to resolve questions.  Please reopen if this doesn't work for you and we can figure out what else needs to be done.