1488422 - Possibly wrong "higher is better" setting for Raptor benchmarks

Status: RESOLVED FIXED

(Testing :: Raptor, enhancement, P1)

Description

[Ionuț Goldan [:igoldan]]

Perfherder benchmarks, such as Sunspider or Assorted DOM are configured as "higher is better" while both benchmarks are reporting the execution time in "ms".

We need to investigate this issue and bring a fix if case.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

assorted dom is 'ms' and lower_is_better=true:
https://searchfox.org/mozilla-central/source/testing/raptor/raptor/tests/raptor-assorted-dom.ini

and here is what we get as output from the job:
PERFHERDER_DATA: {"framework": {"name": "raptor"}, "suites": [{"extraOptions": [], "name": "raptor-assorted-dom-firefox", "lowerIsBetter": false, "alertThreshold": 2.0, "value": 20.4, "subtests": [{"name": "worker-getprop-performance-getter", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [4.3, 4.8, 4.7, 4.2, 4.5, 3.9, 4.7, 5, 4.6, 4.5], "value": 4.55, "unit": "ms"}, {"name": "worker-getname-performance-getter", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [4.4, 4.4, 4.9, 4.4, 3.5, 3.9, 4.6, 4.3, 4.5, 4], "value": 4.4, "unit": "ms"}, {"name": "window-getprop-performance-getter", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [4.6, 4.8, 4.5, 4.4, 4.6, 4.5, 5.3, 4.5, 4.8, 4.2], "value": 4.55, "unit": "ms"}, {"name": "window-getname-performance-getter", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [4.6, 4.1, 4, 3.8, 4.3, 4, 5.1, 3.7, 4.5, 4], "value": 4.05, "unit": "ms"}, {"name": "total", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [20.6, 21.3, 20.9, 19.7, 19.5, 19.1, 22.9, 20.2, 21.3, 19.7], "value": 20.4, "unit": "ms"}, {"name": "test", "lowerIsBetter": false, "alertThreshold": 2.0, "replicates": [2.7, 3.2, 2.9, 2.9, 2.6, 2.8, 3.3, 2.6, 2.9, 2.9], "value": 2.9, "unit": "ms"}], "type": "benchmark", "unit": "ms"}]}

I suspect this is breakage from my work on bug 1486789 to allow subtests to have a different value than the summarized result

Comment 2

[Ionuț Goldan [:igoldan]]

Attachment: Bug 1488422 - Use boolean literal instead of string

Comment 3

[Ionuț Goldan [:igoldan]]

I tested this on Try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=5c3a465379d2f773064e7eae6b9fef96aaced442&filter-tier=1&filter-tier=2&filter-tier=3&selectedJob=197563993

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 9006482 [details]
Bug 1488422 - Use boolean literal instead of string

Joel Maher ( :jmaher ) (UTC-4) has approved the revision.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3c5a69821efd
Use boolean literal instead of string r=jmaher

Comment 6

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/3c5a69821efd