1591911 - Firefox Bug Husbandry Bot probably shouldn't close intermittent test-verify bugs so frequently

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Bulk Bug Edit Requests, task, P2)

Description

[Mark Banner (:standard8)]

If a test is permanently or intermittently failing test-verify, then this will only show on treeherder when the test gets touched.

For a lot of tests, this could be months or years before the failure gets seen again.

A sheriff may file an intermittent bug on this initially, which then gets closed after a month or two. Subsequently, someone touches the test again, and it shows up on treeherder again, at which points the bug gets touched again. And repeat...

For example, see bug 1515401.

I think for intermittent/perma fails that are test-verify only, we shouldn't be automatically closing the bugs or we should have some sort of extra verification that the bug has actually gone away.

Comment 1

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

Mark, we'll need a way to know what a test-verify fail is so we can exclude it from the search for intermittent test failure bugs to resolve.

Comment 2

[Mark Banner (:standard8)]

The only way I know of currently is that test-verify bugs tend to have TV or test-verify in the title.

Comment 3

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

We need to make that a keyword then. We need Treeherder to match the behavior.

Comment 4

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

We have a test-verify-fail keyword. I can update AutoNag such that it doesn't close bugs with that keyword. Shall I go ahead and make that change?

Comment 5

[Mark Banner (:standard8)]

Yes, I think that would be useful. Thank you.

Comment 6

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

Attachment: GitHub Pull Request

Comment 7

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

PR merged to master, closing.