1428857 - create a rotten oranges dashboard

Status: RESOLVED FIXED

(Testing :: General, enhancement)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

currently we have neglected oranges:
https://charts.mozilla.org/NeglectedOranges/index.html

this is used as the initial point for triage.  We want to highlight oranges that are on trend to be disabled.  Our current policy for disabling tests is 200 failures in 30 days (although this might be changed to something like 100 failures in 21 days).

Given this, we could look at the average failure rate over the last 7 days for stockwell needswork bugs, and extrapolate that to 30 days.  When we are within 7 days of disabling (currently 23 days, possibly 14 days in the future), this would result in a smaller list of bugs and require a different workflow for sheriffs.  We would then use a human eye to determine:
* how did this fail over the last 30 days?
* did the failure rate change recently causing a spike?
* is there a real history here of developers working on this?  
* recommend disabling or prioritizing this.

I imagine this will be a new dashboard that will be triaged before neglected oranges.  Also this might serve a dual purpose on highlighting new failures (filed in the last 7 days) which are on track for being high frequency/disabling and allow sheriffs to retrigger tests and find the root cause which could help result in a fix faster.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

we are doing this with a bugzilla whiteboard tag and the process is documented here:
https://wiki.mozilla.org/Auto-tools/Projects/Stockwell/disable-recommended