863061 - No alerts for PGO libxul linker memory regressions under 2%

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Matt Brubeck (:mbrubeck)]

Bug 710840 added libxul link memory data to the graph server.  However, we have not received email alerts for the regressions since then because bug 822249 suppresses email for regressions of less than 2%.  There have been at least 6 since those changes were deployed, but all of theme were between 0.45% and 1.05% and so no one was notified.

Some suggestions:

1) For specific whitelisted datasets where we care about small changes (like libxul memory), ignore the threshold and always send emails.

2) For changes below the 2% threshold, do not send email to individual authors but do send it to the dev-tree-management list.

3) For changes below the 2% threshold, send email only if the t-test score is above an extra-high threshold like 20 or 30, instead of the regular threshold of 9.

4) Get rid of the 2% threshold and try reducing false alarms in other ways, e.g. bug 627860.

Comment 1

[(no longer active)]

The next doomsday is close.  Bumping this to critical.

Comment 2

[Ed Morley [:emorley]]

I'd say we should go with #1.

Comment 3

[Ed Morley [:emorley]]

Note also that Datazilla means analyze_talos.py will be EOLed soon, so not worth coming up with an elaborate solution for this.

Comment 4

[Matt Brubeck (:mbrubeck)]

Attachment: ignore percentage for "LibXUL Memory during link" regressions

This implements option 1 from comment 0.  (Note that this will require a change to analysis.cfg during deployment.)

Comment 5

[Matt Brubeck (:mbrubeck)]

Comment on attachment 739160 [details] [diff] [review]
ignore percentage for "LibXUL Memory during link" regressions

http://hg.mozilla.org/graphs/rev/640b38d873e0

Leaving open until deployed (including analysis.cfg edit).

Comment 6

[Chris AtLee [:catlee]]

Deployed, with analysis.cfg updated.

Comment 7

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/d1a8bcf8a4d832b99764214c7da13b46b1186a84
Bug 863061 - Ignore percentage_threshold for libxul linker memory [r=catlee]