1481728 - Improvement alerts aren't generated automatically

Status: RESOLVED INVALID

(Tree Management :: Perfherder, defect, P3)

Description

[Ionuț Goldan [:igoldan]]

I noticed this happens on the build_metrics framework, for at least the compiler warnings and installer size tests.

Recent examples, were I had to manually create the alerts:

https://bugzilla.mozilla.org/show_bug.cgi?id=1474860#c10

https://bugzilla.mozilla.org/show_bug.cgi?id=1476194#c8

https://bugzilla.mozilla.org/show_bug.cgi?id=1476194#c9

https://bugzilla.mozilla.org/show_bug.cgi?id=1476194#c10

Comment 1

[Joel Maher ( :jmaher ) (UTC -0800)]

we are getting improvement alerts for the other frameworks still, so this seems like an edge case. Would we expect to see improvement alerts for <2%, that is our default threshold for an alert unless otherwise specified.

The 13% improvement should have been alerted on.  Maybe there are messages in the log file?  Were data points backfilled here and we got the notification >2 weeks later because we were backfilling 2 weeks later?

Comment 2

[Sarah Clements [:sclements]]

This is marked as a P1 but isn't assigned to anyone. Can this be marked down to P2 or P3?

Comment 3

[Ionuț Goldan [:igoldan]]

(In reply to Sarah Clements [:sclements] from comment #2)

This is marked as a P1 but isn't assigned to anyone. Can this be marked down to P2 or P3?

Yes, I'd mark it as P3.

Comment 4

[Ionuț Goldan [:igoldan]]

Reason for this issue is that these tests have an absolute threshold, not a percentage based one.
Because of this, there's a visual confusion that makes us think an alert should have been generated.