1506593 - Allow setting different alert thresholds for subtests

Status: NEW

(Testing :: Talos, enhancement, P3)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

currently right now we have a way to set a threshold for all tests:
https://searchfox.org/mozilla-central/source/testing/talos/talos/output.py#126

but in the case of damp we would like to look at dynamically changing this as some tests are noisier than others.  In that case we would either need data coming from the test itself, or a metadata file that supports this.

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

:igoldan are we seeing a lot of invalid alerts for the damp subtests? This seems like a reasonable way to reduce that for any subtests that have a large amount of noise.

Comment 2

[Ionuț Goldan [:igoldan]]

(In reply to Dave Hunt [:davehunt] [he/him] ⌚️UTC from comment #1)

:igoldan are we seeing a lot of invalid alerts for the damp subtests? This seems like a reasonable way to reduce that for any subtests that have a large amount of noise.

I need to do a database query to figure this out.

Comment 3

[Ionuț Goldan [:igoldan]]

Finished the query. Seems like we had 30 invalid alerts on devtools framework.

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Ionuț Goldan [:igoldan], Performance Sheriff from comment #3)

Finished the query. Seems like we had 30 invalid alerts on devtools framework.

Over how much time?

Comment 5

[Ionuț Goldan [:igoldan]]

(In reply to Dave Hunt [:davehunt] [he/him] ⌚️UTC from comment #4)

(In reply to Ionuț Goldan [:igoldan], Performance Sheriff from comment #3)

Finished the query. Seems like we had 30 invalid alerts on devtools framework.

Over how much time?

From 2018 Aug 21 up to 2018 Oct 13. Then no more invalid damp alerts.

Comment 6

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Ionuț Goldan [:igoldan], Performance Sheriff from comment #5)

From 2018 Aug 21 up to 2018 Oct 13. Then no more invalid damp alerts.

It sounds like this is not a current concern then. Let's keep this open as having thresholds for subtests sounds like a reasonable enhancement.

Comment 7

[Alexandre Poirot [:ochameau]]

Note that, since we took over the manual sheriffing of DAMP, we probably don't do it as well as the other test suites.

I know, that, myself, I tend to rather use our custom dashboard, available here:
https://firefox-dev.tools/performance-dashboard/
Which allows to quickly scan all the subtests rather than opening each subtest graph one by one.

The result of that is that I'm not sure we ever flagged any alert as INVALID, even if I imagine that a signifant part of them are.
Note that, as we aren't able to be fully effective with the alerts dashboard:
https://treeherder.mozilla.org/perf.html#/alerts?status=0&framework=12&hideDwnToInv=1&page=1
We are not flagging "valid" alerts correctly either...