1503188 - adjust rules for [stockwell disable-recommended] to account for "fixes"

Status: NEW

(Tree Management :: Treeherder, enhancement, P3)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

many times a test will be high frequency and switch to low frequency, in this case something was fixed, but the bug is not marked as fixed.

Unfortunately our rules will say 150 failures in 21 days, no matter what.

I think we should make it:
* 150 failures in 21 days
AND
* >15 failures in the last 5 days (recent 3+/day)

this is not going to be perfect, but I want to avoid long weekends or long weekends where we run on lower volume and might not see as many failures.

Comment 1

[Sarah Clements [:sclements]]

I think we'd need to either add a new query with the 5-day time range or else parse the original results and put any 5-day results into a dict for lookup. Since I won't be able to work on this until early next year, here's where to make changes if you're so inclined :)

https://github.com/mozilla/treeherder/blob/master/treeherder/intermittents_commenter/commenter.py#L255