1032643 - Bug suggestions: Tweak algorithm for determining how many to show & which to hide under "show more"

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P1)

Description

[Phil Ringnalda (:philor)]

For the Android 2.3 opt gl job in https://treeherder.mozilla.org/ui/#/jobs?repo=mozilla-aurora&revision=4851cd40200f the correct suggestion would be "Bug 777574 - Intermittent test_webgl_conformance_test_suite.html | [conformance/more/conformance/quickCheckAPI-B2.html,quickCheckAPI-B3.html,quickCheckAPI-B4.html] Timeout in this test page" but instead we get 10 others.

Wait, *10* others? There are 28 open bugs in https://bugzilla.mozilla.org/buglist.cgi?quicksearch=test_webgl_conformance_test_suite.html%20kw%3Aintermittent, is treeherder limiting to a maximum of 10 open bugs? That flat out will not work, there are far too many nasty tests which use a single filename for what is in effect an entire suite of tests, and require 20 or 30 or who knows how many suggestions.

Comment 1

[Ed Morley [:emorley]]

This overlaps with bug 1032647 - morphing summary to something that covers both.

Current TBPL behaviour:
* Search for all bugs (both open and closed) that have been modified in the last 3 months (https://hg.mozilla.org/webtools/tbpl/file/479901292056/php/inc/AnnotatedSummaryGenerator.php#l202)
* If more than 50 bugs returned (total - ie open+closed) - then give up and don't show any of them (https://hg.mozilla.org/webtools/tbpl/file/479901292056/php/inc/AnnotatedSummaryGenerator.php#l125)

Ideal treeherder behaviour:
* Search for all bugs that match, regardless of open/closed state, or when they were modified
* Split the results into two buckets:
  1) "Open recent bugs" (ie bug is not resolved & was modified in last 3 months)
  2) "All other bugs" (ie all closed bugs + open bugs that were not modified in the last 3 months)
* The second bucket is collapsed by default in the UI (under a "show more" or something).
* If the first bucket is empty, then don't collapse the second bucket by default
* If either bucket exceeds N bugs in size then give up for that bucket and don't display any bugs for that bucket (just a "Exceeded max bug suggestions" or similar). Perhaps go with N of 50 to start?

Does that sound reasonable? :-)

Comment 3

[Ed Morley [:emorley]]

Some extra context, if it helps...

Whilst 90%+ of the jobs that fail intermittently will only 1-3 bug suggestions, there are the odd test runners (like the one mentioned in comment 0) that output failures in a way that is hard to avoid false positives. Now the ideal longer term solution is to fix the output of those test runners, but for now we need to try and strike the fine balance between too many and too few suggestions shown. (The 50 limit in TBPL was to stop UI freezes)

Another factor that the comment 1 proposal is trying to solve, is that bugs that have not been recently modified (even if open) are sometimes cause for mis-classification + if they are ancient, the chances are a new bug should be filed since the underlying intermittent failure might be unrelated. Which is why we don't show bugs > 3 months in TBPL - but is sometimes annoying for bugs that are really infrequent (and don't need a new bug filing). However with treeherder we may be able to compromise by hiding "stale bugs" in the "show more" bucket. Comment 1 doesn't put any time limit on these, but we may still need to set a 6 months cap after all - let's see how we go.

Another edge case is on release branches (eg aurora/beta). An intermittent failure might be fixed on mozilla-central (and so the bug marked as fixed) - but the fix not be present on the release repo, and so the failure still need classifying on aurora/beta. Therefore on aurora/beta approx 20-50% of the time, the bug picked will be resolved - hence needing the "show more". With the comment 1 proposal, this will work fine so long as there isn't a new open intermittent failure bug that causes the "all other bugs" bucket to be collapsed by default. At a later date we may have to tweak this further to avoid constantly having to click "show more" on release branches - but I think for now, let's see how we go.

Comment 4

[Mauro Doglio [:mdoglio]]

I started implementing what's described in comment 1

Comment 5

[Mauro Doglio [:mdoglio]]

I put my changes on http://treeherder-dev.allizom.org, please wait a couple of hours for new bug suggestions to come in. I had to change the format of the "Bug suggestions" artifact to support the ui development. Also, the search algorithm used is now "all words" as requested. I set the maximum number of suggestions per failure  to 20, but I can quickly change it up to 50. 
I'll needinfo :edmorley for feedback on this. I'm happy to open a PR once he gives the green light

Comment 6

[Ed Morley [:emorley]]

treeherder-dev seems to have regressed the display or error lines on the "failure summary" tab.

Compare the failures shown in the log viewer:
http://treeherder-dev.allizom.org/ui/logviewer.html#?job_id=155977&repo=mozilla-aurora

With those that appear for the job ("Linux debug - M( oth )"):
http://treeherder-dev.allizom.org/ui/#/jobs?repo=mozilla-aurora&revision=b5e67236050a

And also with prod:
https://treeherder.mozilla.org/ui/#/jobs?repo=mozilla-aurora&revision=b5e67236050a

Note that prod also doesn't suggest bug 981638 for this failure either.

Comment 7

[Ed Morley [:emorley]]

Sorry s/display or/display of/

Comment 8

[Mauro Doglio [:mdoglio]]

I just fixed an issue on dev, the bug suggestions should be available in the next jobs

Comment 9

[Ed Morley [:emorley]]

It generally looks much better thank you. I see that we don't have any suggestions for:
http://treeherder-dev.allizom.org/ui/logviewer.html#?job_id=231072&repo=mozilla-central

I'm guessing we're missing some of the logic from:
https://hg.mozilla.org/webtools/tbpl/file/tip/php/inc/AnnotatedSummaryGenerator.php#l88

Comment 10

[Mauro Doglio [:mdoglio]]

Attachment: Github PR #123 on treeherder-ui

Comment 11

[Mauro Doglio [:mdoglio]]

Attachment: Github PR #1817 on treeherder-service

Comment 12

[Mauro Doglio [:mdoglio]]

(In reply to Ed Morley [:edmorley] from comment #9)
> It generally looks much better thank you. I see that we don't have any
> suggestions for:
> http://treeherder-dev.allizom.org/ui/logviewer.
> html#?job_id=231072&repo=mozilla-central
> 
> I'm guessing we're missing some of the logic from:
> https://hg.mozilla.org/webtools/tbpl/file/tip/php/inc/
> AnnotatedSummaryGenerator.php#l88

I fixed a bug that prevented from retrieving suggestions for crash and leaks lines

Comment 13

[Ed Morley [:emorley]]

On production, I'm seeing another summary with no failures listed, even though they appear in the log viewer:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=33891e5cb4e9
https://treeherder.mozilla.org/ui/logviewer.html#?job_id=1105618&repo=try

Comment 14

[Mauro Doglio [:mdoglio]]

New failing jobs are now showing bug suggestions. Bug 1054955 was preventing the population of the bugscache table, so jobs completed > 1h ago have no suggestions.

Comment 15

[Ed Morley [:emorley]]

To keep things simpler, I've broken out the "bug suggestions are incorrect/missing" parts to bug 1057359 (since they're more about the extraction of a search term and fall-back logic), whereas this bug is more about determining how many bugs to display from the results returned from Bugzilla & what to do with closed bugs etc.

For the failure in bug 1057359 comment 2, I can't see a "show more" link to show the closed bugs. Is this bug in production? Or is that feature not implemented? :-)

Comment 16

[Ed Morley [:emorley]]

Though I see the feature when clicking on  b2g_emulator_vm mozilla-inbound opt test mochitest-7 in:
https://treeherder.mozilla.org/ui/#/jobs?repo=mozilla-inbound&revision=daa75e9e4f30

Strange...

Comment 17

[Ed Morley [:emorley]]

I haven't seen the issue in comment 15 again, closing for now.

Comment 18

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/ca795a03b1a61717b37e77dfdea52a5240187bb2
(Bug 1032643) Tweak bug suggestions

https://github.com/mozilla/treeherder/commit/0e512b1de67e6ad36ad59e8289bfed9f44ea195c
Merge pull request #123 from mozilla/bug-1032643-tweak-bug-suggestions

Bug 1032643 tweak bug suggestions