1421356 - Allow automatic classification to classify mass failures that resulted in a backout

Status: NEW

(Tree Management :: Treeherder, enhancement, P3)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

• Because of different durations for builds and test runs on different platforms and that several pushes can land before the need for a backout has been identified, test failures show up on Treeherder over time. The sheriffs’ dilemma is that they can either check new failures to search for other new failures among the failures fixed by the backout or wait for most of the failing tests to finish and then do a big failure starring run and use the filters to select multiple tests or patterns which they identified, e.g. always failing in the same chunk. For those reasons, the frequent classification is more time consuming.
    • The ability to generate temporary auto-classification rules to star failures with content X in the failure log starting with push A and before push B (backout of A) would load that off to automation.
    • Don’t rely on the commit message of the backout for detecting the failing tests. A patch can be backed out for ESLint failures and later it turns out that it also fails many browser-chrome failures.
    • There can be more than one failing test (in different test chunks), so keeping the parameters for the rule form will be helpful because the filter creation might get repeated.
    • Filters should be kept for some time (a month?) to prevent retriggers and backfilled job from needing manual classification.
    • Different new failures in the same test chunk won’t be noticed, but that’s an improvement compared to the status quo.

Comment 1

[Ed Morley [:emorley]]

So we chatted about this in the Treeherder meeting and we think manual rules might end up being pretty risky. As such, this seems like a feature that should be built into auto-classification (since it's already going to have to be able to figure out failures due to backouts, to ensure that they don't taint the suggestions for days later).

Also worth noting is that in a world where pushes are obsoleted (rather than backed out) on the autoland repo, then the whole push will be marked as obsoleted, so this problem will be lessened.