1232270 - ignore dromaeo* alerts unless they are 10%

Status: RESOLVED FIXED

(Testing :: Talos, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

we get a lot of alerts in the perfherder alert view related to dromaeo, this is a noisy set of tests and for graph server we have only alerted on 10% or greater regressions, not 2% like the rest of the tests.

take for example:
https://treeherder.mozilla.org/perf.html#/alerts?id=1647

and the corresponding graph:
https://treeherder.mozilla.org/perf.html#/graphs?timerange=1209600&series=[mozilla-inbound,5d92d35aeb763b5852f6e260466d6b9b8fc17a56,1]&highlightedRevisions=d68d7fa71614

this is just a noisy test and we happened to have a set of data in the noise which triggered a regression alert.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

it appears the next step here is to modify the PERFHERDER_DATA structure to support the new alertThreshold, example of existing data:
PERFHERDER_DATA: {"framework": {"name": "talos"}, "suites": [{"lowerIsBetter": false, "subtests": [{"lowerIsBetter": false, "replicates": [73757.26], "unit": "score", "name": "attr.html", "value": 2836.04}, {"lowerIsBetter": false, "replicates": [377.48], "unit": "score", "name": "modify.html", "value": 218.94}, {"lowerIsBetter": false, "replicates": [18807.90], "unit": "score", "name": "query.html", "value": 25158.98}, {"lowerIsBetter": false, "replicates": [236.46], "unit": "score", "name": "traverse.html", "value": 385.78}], "name": "dromaeo_dom", "value": 1568.79}]}

to something like this:
PERFHERDER_DATA: {"framework": {"name": "talos"}, "suites": [{"lowerIsBetter": false, "alertThreshold": 10.0, "subtests": [{"lowerIsBetter": false, "alertThreshold": 10.0, "replicates": [73757.26], "unit": "score", "name": "attr.html", "value": 2836.04}, {"lowerIsBetter": false, "alertThreshold": 10.0, "replicates": [377.48], "unit": "score", "name": "modify.html", "value": 218.94}, {"lowerIsBetter": false, "alertThreshold": 10.0, "replicates": [18807.90], "unit": "score", "name": "query.html", "value": 25158.98}, {"lowerIsBetter": false, "alertThreshold": 10.0, "replicates": [236.46], "unit": "score", "name": "traverse.html", "value": 385.78}], "name": "dromaeo_dom", "value": 1568.79}]}

this is adding the "alertThreshold" field to the suite and the subtests.

:wlach, is this accurate?

Comment 2

[William Lachance (:wlach)]

Yup, exactly -- this is now a Talos bug. Ideally we would make it possible to override the default alerting threshold per test/suite, so this isn't a "just for dromaeo" special case.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: MozReview Request: Bug 1232270 - set alert_threshold definitions for talos. r?wlach

Review commit: https://reviewboard.mozilla.org/r/44695/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/44695/

Comment 4

[William Lachance (:wlach)]

Comment on attachment 8738805 [details]
MozReview Request: Bug 1232270 - set alert_threshold definitions for talos. r?wlach

https://reviewboard.mozilla.org/r/44695/#review41447

One question, but this looks fine. Please do a quick try run for talos to make sure all is well before landing.

::: testing/talos/talos/test.py:569
(Diff revision 1)
>  
>  class dromaeo(PageloaderTest):
>      """abstract base class for dramaeo tests"""
>      filters = filter.dromaeo.prepare()
>      lower_is_better = False
> +    alert_threshold = 5.0

I thought we wanted this to be 10%?

(if you just want to try 5% at first to see how it goes, that's ok too) :)

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

I thought 5% would be a good start, and I snuck it in for ally as well.

here is my try run which I verified in the logs that we have the proper 2% and 5% thresholds:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=edc5aef4c6ce70a4f19bb2f5ee1eae1b68e0f1bd

Comment 6

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/ab03da27fcb7

Comment 7

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/ab03da27fcb7