1809729 - [meta] work to revamp intermittent failure view and bz comments

Status: NEW

(Tree Management :: Treeherder, task)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

currently intermittent failure view does a great job of displaying counts of failures, but we often overlook perma fails when we only run on mozilla-central (i.e. low counts). It is unclear if an issue has a chance to reproduce, or if it is so infrequent we shouldn't worry about it.

I would like this to be 2 parts:

1. treeherder IFV UI to be more actionable- show more data and bucket failures into categories for developers to focus more on what is needed.
   1a) treeherder IFV UI summary to show bugs that have actionable items (i.e. need disabling)
   1b) account for cases where it is unclear how often the test ran, or edge cases (like an infrastructure failure)
2. bugzilla commenter - this is misleading, noisy, and not the most effective communication tool.
   2a) if there is something actionable add a comment every day or week, otherwise 1 comment/month
   2b) for daily/weekly comments, highlight real failure rate, what can be done and what to watch out for, specifically by something more actionable (like test variant)
   2c) watch out for edge cases (like infrastructure, crashes, etc.)