1469280 - Fix raptor speedometer score calculation

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement)

Description

[Robert Wood [:rwood]]

The subtest results order is incorrect, therefore the wrong summary values are being used in the final score calculation.

Comment 1

[Robert Wood [:rwood]]

Also set the pagecycles back to 5.

Comment 2

[Robert Wood [:rwood]]

Attachment: Bug 1469280 - Fix raptor speedometer score calculation;

Review commit: https://reviewboard.mozilla.org/r/251928/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/251928/

Comment 3

[Robert Wood [:rwood]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=166e7c3462c5e885d2e2cbf12839551585410055

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8986622 [details]
Bug 1469280 - Fix raptor speedometer score calculation;

https://reviewboard.mozilla.org/r/251928/#review258426

r- as this seems overly complicated- glad to see we have a solution in the works though.

::: testing/raptor/raptor/output.py:108
(Diff revision 1)
> +                overall_suite_results = []
> +
> +                for page_cycle in test.measurements['speedometer']:
> +                    # use an OrderedDict so the order entries are added is persisted
> +                    # we need the speedometer subsuites to be in the order they're reported
> +                    from collections import OrderedDict

can we import at the top of the source file?

::: third_party/webkit/PerformanceTests/Speedometer/resources/benchmark-report.js:89
(Diff revision 1)
>                      for (var count=0; count < this.iterationCount; count++) {
>                          fullNames.push(fullName);
>                      }
>                  }
>                  if (location.search == '?raptor') {
> -                    _data = ['raptor-benchmark', 'speedometer', measuredValuesByFullName];
> +                    _data = ['raptor-benchmark', 'speedometer', values, fullNames];

if we are doing values/fullNames instead of measuredValuesByFullName, can we not make a much simpler solution on the raptor python harness side?

I think we are transforming it 3 times now in python where 1 would be ideal and 2 would be ok.

Comment 5

[Robert Wood [:rwood]]

Comment on attachment 8986622 [details]
Bug 1469280 - Fix raptor speedometer score calculation;

https://reviewboard.mozilla.org/r/251928/#review258426

> can we import at the top of the source file?

Yep meant to move that thanks

> if we are doing values/fullNames instead of measuredValuesByFullName, can we not make a much simpler solution on the raptor python harness side?
> 
> I think we are transforming it 3 times now in python where 1 would be ideal and 2 would be ok.

Values and fullnanes are two separate lists, subsuite names (160) and replicate values (800) for a single page cycle.

I really don't know how else to simplify this.Do you want to take it and hack on it? That'd be great if you can improve it, but I am stuck sorry.

Comment 6

[Robert Wood [:rwood]]

Using measuredValuesByFullName is easier in that it's already one set of data with each subsuite and corresponding values together. However that's what I was using first, and that is somehow in the wrong order.

Talos uses values and fullnames and processes it in a similar way. It's def over-complicated but I need help to improve it please, I can't seem to figure out a better way.

Comment 7

[Robert Wood [:rwood]]

Maybe the solution is to hack on speedometer benchmark-report.js and have it report in a better way? Although I don't want to break talos... but maybe that would be better?

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

We could easily hack in the benchmark, or make something more streamlined in raptor.  My concern with the current approach is we have data to parse which is very complicated and hard to explain- this will need to be done for every benchmark :(  I would like to find a cleaner method for speedometer as it will be used for stylebench as well (either shared code or similar duplicated code).

Let me play around a bit.

Comment 9

[Phabricator Automation]

Attachment: Bug 1469280 - Fix raptor speedometer score calculation. r=rwood

ensure that raptor summarizes data properly from all benchmarks

Comment 10

[Phabricator Automation]

Comment on attachment 8987602 [details]
Bug 1469280 - Fix raptor speedometer score calculation. r=rwood

Robert Wood [:rwood] has approved the revision.

https://phabricator.services.mozilla.com/D1815

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7b09b3aa7293
Fix raptor speedometer score calculation. r=rwood

Comment 12

[Raul Gurzau (:RaulG)]

https://hg.mozilla.org/mozilla-central/rev/7b09b3aa7293