1870640 - Consider adding alert for Jetstream2 sub-tests for >10% change (Was: 10% regression on AWFY-Jetstream2-base64-SP-Average around 9December2023 was not reported)

Status: NEW

(Testing :: Performance, defect, P2)

Description

[Mayank Bansal]

See bug 1869464 for details.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:afinder, could you have a look please?

For more information, please visit BugBot documentation.

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

We probably don't have alerts enabled for JetStream2 subtests, and the overall score may not have been impacted by this change. @kshampur could you confirm? @bas should we consider enabling subtest alerts for this benchmark?

Comment 3

[Bas Schouten (:bas.schouten)]

(In reply to Dave Hunt [:davehunt] [he/him] ⌚BST from comment #2)

We probably don't have alerts enabled for JetStream2 subtests, and the overall score may not have been impacted by this change. @kshampur could you confirm? @bas should we consider enabling subtest alerts for this benchmark?

In my opinion, not really unless it comes at almost 0 cost. We don't really care that much about individual jetstream subtests from a general performance perspective. But it'd be good for someone from JS to weigh in, as they are in the best position to derive value.

Comment 4

[Kash Shampur [:kshampur] ⌚EST]

as far as I can tell It doesn't look like we alert on the subtests

Comment 5

[Jan de Mooij [:jandem]]

I think alerts for sub tests would be nice if it's easy to do, but I'm a little worried about false positives or noise for some of these tests... Ideally we'd watch for large regressions only, say > 10%, but I don't know if that's possible.

Comment 6

[Greg Mierzwinski [:sparky]]

:jandem, it's possible to enable alerting on these tests with custom alert thresholds on them so we can make them alert only if they're greater than 10% (for example). We're wondering about the impact of having these subtests alerting for you? Alternatively, what do you think about the severity of missing alerts/changes from these subtests?

Comment 7

[Jan de Mooij [:jandem]]

(In reply to Greg Mierzwinski [:sparky] from comment #6)

:jandem, it's possible to enable alerting on these tests with custom alert thresholds on them so we can make them alert only if they're greater than 10% (for example). We're wondering about the impact of having these subtests alerting for you? Alternatively, what do you think about the severity of missing alerts/changes from these subtests?

If the threshold is sufficiently high (for example >= 10%), that should reduce the number of alerts and false positives, right? If it's easy to enable I think it's worth trying.

Comment 8

[Mayank Bansal]

Florin shared a list of alerts generated for Jetstream2 : https://treeherder.mozilla.org/perfherder/alerts?hideDwnToInv=0&page=1&filterText=jetstream2&framework=13&status=-1