1502073 - Update metadata for Speedometer and Web-tooling subtests

Status: RESOLVED INVALID

(Tree Management :: Perfherder, enhancement)

Description

[Armen [:armenzg]]

From bug 1502036:
#################
(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #1)
For speedometer we specifically set subtests as ms (lower is better) and
report a score (higher is better) as of bug 1486789.

(In reply to Armen [:armenzg] from comment #2)
It seems that Perfherder has *not* noticed the change and still reports the
subtests with the old values:
https://treeherder.mozilla.org/api/project/mozilla-central/performance/
signatures/?parent_signature=bf4fcd6df34988c8aea87ccda51b0f70b4141643
#################

The subtests are marked as "Higher is better" (Score) "lower_is_better: false",
however, jmaher indicates that they should be reporting as "Lower is better (ms)" [1]

Can we either change the logic for Perfherder's submissions can update the initial stored value? or fix the Speedometer subtests metadata? (We would need to fix it for every platform).

This bug is causing issues for arewefastyet.com. I will patch the UI to mitigate this issue.

Many thanks!

[1] From https://treeherder.mozilla.org/api/project/mozilla-central/performance/
signatures/?parent_signature=bf4fcd6df34988c8aea87ccda51b0f70b4141643
> "0013bc4c95e7b6a91b972d555851049919cf6fc8": {
>        "suite": "raptor-speedometer-chrome",
>        "lower_is_better": false,
>        "option_collection_hash": "102210fe594ee9b33d82058545b1ed14f4c8206e",
>        "framework_id": 10,
>        "id": 1714973,
>        "machine_platform": "linux64",
>        "test": "Preact-TodoMVC/CompletingAllItems",
>        "parent_signature": "bf4fcd6df34988c8aea87ccda51b0f70b4141643"
>    },

Comment 1

[Armen [:armenzg]]

Speedometer subtests should be lower is better (do not defined "lower_is_better": false).
Web-tooling subtests should be higher is better (define "lower_is_better": false).

Thanks!

Comment 2

[William Lachance (:wlach)]

Perfherder should be updating the subtest information as it comes in:

https://github.com/mozilla/treeherder/blob/bfd784e0fbb68d3a1fba44f287dedeaa95db1ace/treeherder/etl/perf.py#L161

Can you take a look at a log from one of these jobs and verify that the PERFHERDER_DATA blob is as you expect it to be?

Comment 3

[Armen [:armenzg]]

Thanks Will!

I see this for Speedometer:
>	"suites": [{
...
>		"name": "raptor-speedometer-firefox",
>		"lowerIsBetter": false,
...
>		"subtests": [{
>			"name": "jQuery-TodoMVC/DeletingAllItems/Sync",
>			"lowerIsBetter": false,
which means that Speedometer is reporting higher is better for both.
IIUC this means that we need to revisit bug 1486789.

Web tooling only defines the parent test with `"lowerIsBetter": false` which means "higher is better" and subtests as "lower is better".

IIUC this bug is invalid.