Closed Bug 1315275 Opened 6 years ago Closed 6 years ago

Add rank to OrangeFactor Robot bug comments

Categories

(Tree Management Graveyard :: OrangeFactor, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: gbrown, Assigned: gbrown)

References

(Blocks 1 open bug)

Details

Attachments

(1 file, 1 obsolete file)

The total number of failures (in a time period) of one OF bug in relation to the total number of failures of other OF bugs is important in that we want people to work preferentially on bugs that cause more failures.

The War on Orange email/newsgroup post and the default view on brasstacks order bugs by total number of failures in the time period, but this order information is not apparent in OrangeFactor Robot bug comments. I'm thinking of adding a comment like:

  "This bug is ranked #8 in this week's War on Orange."

The primary purpose of this comment would be to provide an easy comparison of the importance of this OF bug in relation to other OF bugs.


To be decided:
 - apply to daily comments, weekly comments, or both?
 - apply to all OF bugs, or just the top N? N=?
See Also: → 1315281
ideally this should apply to the top 50 bugs- and the weekly comments.  As for daily comments- that seems a bit iffy since things can change so drastically daily- I think daily we care about "did this issue have a massive change/increase vs the last day"?

How would this work if we set a general rule of "all bugs that have >20 instances/week need to be fixed"- it wouldn't matter what rank it is- but it would matter if it crossed a threshold (ideally a threshold we can configure).  Assuming we do get the high frequency failures resolved, I see a great need for the top 50 bugs in general- would we ensure those get emailed out daily/weekly?
Top 50 bugs ranked in weekly comments seems reasonable - thanks.

I think we can treat ranking as a separate issue from your >20 instances/week idea, although I do want to avoid information overload.

In today's environment, I prefer the message "fix the top N most frequent failures" rather than "fix all bugs with >20 instances/week". Sometimes - as today - bugs with >20 instances/week seems overwhelming, whereas fixing the #3 failure is potentially thrilling.

I suspect the current 5 failures/week thresholds would ensure that the top 50 bugs get weekly comments (you said email, but I assume we're only talking about bugzilla comments here), and the 15/day threshold would trigger daily comments for most top 50 bugs. I expect it would be easy to add a rule to ensure daily/weekly comments for the top 50...but that likely won't make much of a difference until failure rates are substantially lower.
Comment on attachment 8807774 [details] [diff] [review]
add rank to weekly comments, for top 50 bugs

Review of attachment 8807774 [details] [diff] [review]:
-----------------------------------------------------------------

::: woo_commenter.py
@@ +107,5 @@
>              continue
> +        rank_message = None
> +        if options.weekly_mode and (bug_id, counts['total']) in top[:RANK_THRESHOLD]:
> +            rank = top.index((bug_id, counts['total']))+1
> +            rank_message = "\n\nThis bug is ranked #%d in this week's War on Orange." % rank

to keep the text simple, could we leave out War on Orange?

Maybe:
This bug is ranked #%d for the past week
For example:

# Bug 1313136:
39 failures in 606 pushes (0.064 failures/push) were associated with this bug in the last 7 days. 

This is the #42 most frequent failure this week.

Repository breakdown:
* autoland: 18
* mozilla-inbound: 12
* mozilla-central: 6
* try: 3

Platform breakdown:
* linux64: 29
* linux32: 9
* android-4-3-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1313136&startday=2016-10-31&endday=2016-11-06&tree=all
Attachment #8807774 - Attachment is obsolete: true
Attachment #8808421 - Flags: review?(emorley)
Comment on attachment 8808421 [details] [diff] [review]
add rank to weekly comments, for top 50 bugs

Review of attachment 8808421 [details] [diff] [review]:
-----------------------------------------------------------------

::: woo_commenter.py
@@ +99,4 @@
>      tb = TopBugs(local_server_url, start_date, end_date, tree='all')
>      stats = tb.stats()
>      bug_stats = tb.stats_by_bug()
> +    top = tb.top_bugs()

What do you think about moving the `weekly_mode` conditional and array slicing to here instead? This would reduce the time taken for the script to run in the most common case (daily mode).
Attachment #8808421 - Flags: review?(emorley) → review+
For this and bug 1315110, I haven't tested locally. Happy for you to land and either deploy yourself if you have access to brasstacks, or else I can deploy for you.
(In reply to Ed Morley [:emorley] from comment #6)
> What do you think about moving the `weekly_mode` conditional and array
> slicing to here instead? This would reduce the time taken for the script to
> run in the most common case (daily mode).

Good idea - will do.
(In reply to Ed Morley [:emorley] from comment #7)
> For this and bug 1315110, I haven't tested locally. Happy for you to land
> and either deploy yourself if you have access to brasstacks, or else I can
> deploy for you.

I have tested locally with --test in both daily and weekly modes -- seems fine.

I will wait a few days to land, in case additional OF ideas develop. Will ping you for deployment - thanks!
https://hg.mozilla.org/automation/orangefactor/rev/4a6e98d39a8e1e331d17fe87ce3e29ea6998b9dd
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Blocks: 1317303
Product: Tree Management → Tree Management Graveyard
You need to log in before you can comment on or make changes to this bug.