Open Bug 1809731 Opened 2 years ago Updated 2 years ago

consider new comments for bugzilla from IFV that are more actionable.

Categories

(Tree Management :: Treeherder, enhancement)

Product:
Tree Management
Component:
Treeherder
Type:
enhancement

Tracking

(Not tracked)

People

(Reporter: jmaher, Unassigned)

References

(Blocks 1 open bug)

Details

I would like to make the comments that IFV adds to bugzilla more useful. as per bug 1809729, we should not be posting weekly, only monthly if there is nothing actionable- first we need to rework IFV and commenters to post more actionable content.

The first path I would like to pursue is assuming this rule:

  • =10% failure rate == good chance of reproducing, should be disabled

  • -5% failure rate == something to keep an eye on and consider in debugging, or disabling

With that, I would like to propose something like this as a comment (for bug 1775902):

test toolkit/components/passwordmgr/test/browser/browser_autocomplete_import.js ran 4812 times on 19 configurations in the last 30 days

Top failing configurations to disable (>10%):

job name num failures num runs % failure
test-macosx1015-64-qr/debug-mochitest-browser-chrome 32 272 11.76%
test-macosx1015-64-qr/debug-mochitest-browser-chrome-spi-nw 28 269 10.41%

Top failing configurations to keep an eye on (>5%):

job name num failures num runs % failure
test-macosx1015-64-shippable-qr/opt-mochitest-browser-chrome 3 56 5.36%
test-macosx1015-64-qr/opt-mochitest-browser-chrome 10 200 5.00%

(In reply to Joel Maher ( :jmaher ) (UTC -8) from comment #0)

we should not be posting weekly, only monthly

  1. Intermittent bugs get closed after 3 weeks without activity. This will need an adjustment.
  2. Tasks for Try classifications might be gone from Treeherder at the same time the comment gets posted (4 week retention).

"Top failing configurations to keep an eye on" - the wording doesn't call for action. Developers would likely wait until these platforms hit the 10% threshold. Is it worth posting this section?

the section to keep an eye on will highlight similar errors in some cases like the above example. It would be realistic to consider skipping / debugging on all OSX. I can see how this would be less interesting- removing it would actually make the comment simpler.

Good point about the monthly issues. Maybe 2 weeks or adjust the bug closing rules- If we were to adjust the bug closing rules, I would prefer to get data about how often we "reopen" a closed bug now with the 3 weeks rule.

On the topic of Try, we really shouldn't be including those in the reports and data by default. try can be misleading, more often there is an older base revision, or a failure turned perma by a typo in a patch.

2222 bugs got reopened by Treeherder in 2022H2 - expect a significant share to have gotten closed and reopened multiple times.

in 1 year we have 23000 bugs in the treeherder bugscache and 17237 in the last 6 months. 2222/17237 is ~13% - the % of bugs reopened.

that seems like we might be closing bugs too soon, especially if many of these have frequent (not random) failures.

You need to log in before you can comment on or make changes to this bug.