1580287 - disabling auto-P5ing of intermittents/permafails

Status: RESOLVED FIXED

(Tree Management :: Treeherder, task)

Description

[Dan Mosedale (:dmosedale, :dmose)]

The bug auto-classifier automagically sets intermittents/permafails to P5.

Our team (User Journey), and possibly the New Tab team as well, is trying to keep our number of failing tests near or close to zero so that our developers continue to trust the tests.

Is there any way to set them to autoclassify bugs in (at least) the Messaging System component to not have a priority, which will make them show up in our untriaged bug list, so that they don't fall through the cracks?

If not, would that be straightforward to implement?

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Emma, could the information if a component shall have priorityP5 automatically be stored in bugzilla.mozilla.org and be public (to be queried by Treeherder browser clients)?

Or shall no new intermittent bugs have the priority set to P5? There are 9317 intermittent bugs since 2019-01-01, up to ~800 per component. That's ~52 bugs/working day.

Implementing it with a list of components where P5 shouldn't be set in Treeherder without dependency on bugzilla.mozilla.org is pretty straightforward, deployment can take a few days.

Comment 2

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

Thanks, :aryx, if I understand correctly you'd want to be able to query Bugzilla and get a list of components for whom we'd not set the priority when Treeherder files an intermittent.

Bugzilla doesn't have the notion of labels at the component level, though it could be added. The label could be auto added at

The alternative would be something a bit hacky like embedding something in the component's descriptions like [ leave intermittent test failure bugs untriaged ], and Treeherder would search components using the API using that string.

If we stop classifying new intermittent bugs as P5s, that's a lot of new bugs in the triage pipeline, and we'd need agreement from Firefox engineering.

I'm going to propose an experiment where Treeherder sets up a list of components to not set P5 and fills that with the User Journey components for now.

Then I can ask engineering if other teams want this and we can decide what to do in the long run.

In the short term, please move forward with excluding Dan's components.

Comment 3

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://github.com/Archaeopteryx/treeherder/commit/e2f14500cbabf087894b941caff061f7d843c537 doesn't set a priority for Firefox's 'Messaging System' and 'New Tab Page'. Dan, shall it land with these two components excluded?

Comment 4

[Dan Mosedale (:dmosedale, :dmose)]

After some out-of-band discussion, I think we probably want to just start with Messaging System, and work with the New Tab team to decide how to manage that component. New Tab folks, feel free to reach out to me in Slack if you want to chat about this morning...

Thanks so much for the quick implementation!

Comment 5

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/5374

Comment 6

[Cameron Dawson [:camd]]

Comment on attachment 9093074 [details] [review]
Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/5374

Aryx: marking this as r- till the linting errors are fixed on the PR.  It's very close!  :)

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

This landed two months ago as https://github.com/mozilla/treeherder/commit/9326906d9de49cca4e6efd94c575f1dfc7d95797