Closed Bug 1232270 Opened 4 years ago Closed 4 years ago

ignore dromaeo* alerts unless they are 10%


(Testing :: Talos, defect)

Not set


(firefox48 fixed)

Tracking Status
firefox48 --- fixed


(Reporter: jmaher, Unassigned)




(1 file)

we get a lot of alerts in the perfherder alert view related to dromaeo, this is a noisy set of tests and for graph server we have only alerted on 10% or greater regressions, not 2% like the rest of the tests.

take for example:

and the corresponding graph:[mozilla-inbound,5d92d35aeb763b5852f6e260466d6b9b8fc17a56,1]&highlightedRevisions=d68d7fa71614

this is just a noisy test and we happened to have a set of data in the noise which triggered a regression alert.
it appears the next step here is to modify the PERFHERDER_DATA structure to support the new alertThreshold, example of existing data:
PERFHERDER_DATA: {"framework": {"name": "talos"}, "suites": [{"lowerIsBetter": false, "subtests": [{"lowerIsBetter": false, "replicates": [73757.26], "unit": "score", "name": "attr.html", "value": 2836.04}, {"lowerIsBetter": false, "replicates": [377.48], "unit": "score", "name": "modify.html", "value": 218.94}, {"lowerIsBetter": false, "replicates": [18807.90], "unit": "score", "name": "query.html", "value": 25158.98}, {"lowerIsBetter": false, "replicates": [236.46], "unit": "score", "name": "traverse.html", "value": 385.78}], "name": "dromaeo_dom", "value": 1568.79}]}

to something like this:
PERFHERDER_DATA: {"framework": {"name": "talos"}, "suites": [{"lowerIsBetter": false, "alertThreshold": 10.0, "subtests": [{"lowerIsBetter": false, "alertThreshold": 10.0, "replicates": [73757.26], "unit": "score", "name": "attr.html", "value": 2836.04}, {"lowerIsBetter": false, "alertThreshold": 10.0, "replicates": [377.48], "unit": "score", "name": "modify.html", "value": 218.94}, {"lowerIsBetter": false, "alertThreshold": 10.0, "replicates": [18807.90], "unit": "score", "name": "query.html", "value": 25158.98}, {"lowerIsBetter": false, "alertThreshold": 10.0, "replicates": [236.46], "unit": "score", "name": "traverse.html", "value": 385.78}], "name": "dromaeo_dom", "value": 1568.79}]}

this is adding the "alertThreshold" field to the suite and the subtests.

:wlach, is this accurate?
Flags: needinfo?(wlachance)
Yup, exactly -- this is now a Talos bug. Ideally we would make it possible to override the default alerting threshold per test/suite, so this isn't a "just for dromaeo" special case.
Component: Perfherder → Talos
Flags: needinfo?(wlachance)
Product: Tree Management → Testing
Version: --- → unspecified
Comment on attachment 8738805 [details]
MozReview Request: Bug 1232270 - set alert_threshold definitions for talos. r?wlach

One question, but this looks fine. Please do a quick try run for talos to make sure all is well before landing.

::: testing/talos/talos/
(Diff revision 1)
>  class dromaeo(PageloaderTest):
>      """abstract base class for dramaeo tests"""
>      filters = filter.dromaeo.prepare()
>      lower_is_better = False
> +    alert_threshold = 5.0

I thought we wanted this to be 10%?

(if you just want to try 5% at first to see how it goes, that's ok too) :)
Attachment #8738805 - Flags: review?(wlachance) → review+
I thought 5% would be a good start, and I snuck it in for ally as well.

here is my try run which I verified in the logs that we have the proper 2% and 5% thresholds:
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla48
You need to log in before you can comment on or make changes to this bug.