1315275 - Add rank to OrangeFactor Robot bug comments

Status: RESOLVED FIXED

(Tree Management Graveyard :: OrangeFactor, defect)

Description

[Geoff Brown [:gbrown]]

The total number of failures (in a time period) of one OF bug in relation to the total number of failures of other OF bugs is important in that we want people to work preferentially on bugs that cause more failures.

The War on Orange email/newsgroup post and the default view on brasstacks order bugs by total number of failures in the time period, but this order information is not apparent in OrangeFactor Robot bug comments. I'm thinking of adding a comment like:

  "This bug is ranked #8 in this week's War on Orange."

The primary purpose of this comment would be to provide an easy comparison of the importance of this OF bug in relation to other OF bugs.


To be decided:
 - apply to daily comments, weekly comments, or both?
 - apply to all OF bugs, or just the top N? N=?

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

ideally this should apply to the top 50 bugs- and the weekly comments.  As for daily comments- that seems a bit iffy since things can change so drastically daily- I think daily we care about "did this issue have a massive change/increase vs the last day"?

How would this work if we set a general rule of "all bugs that have >20 instances/week need to be fixed"- it wouldn't matter what rank it is- but it would matter if it crossed a threshold (ideally a threshold we can configure).  Assuming we do get the high frequency failures resolved, I see a great need for the top 50 bugs in general- would we ensure those get emailed out daily/weekly?

Comment 2

[Geoff Brown [:gbrown]]

Top 50 bugs ranked in weekly comments seems reasonable - thanks.

I think we can treat ranking as a separate issue from your >20 instances/week idea, although I do want to avoid information overload.

In today's environment, I prefer the message "fix the top N most frequent failures" rather than "fix all bugs with >20 instances/week". Sometimes - as today - bugs with >20 instances/week seems overwhelming, whereas fixing the #3 failure is potentially thrilling.

I suspect the current 5 failures/week thresholds would ensure that the top 50 bugs get weekly comments (you said email, but I assume we're only talking about bugzilla comments here), and the 15/day threshold would trigger daily comments for most top 50 bugs. I expect it would be easy to add a rule to ensure daily/weekly comments for the top 50...but that likely won't make much of a difference until failure rates are substantially lower.

Comment 3

[Geoff Brown [:gbrown]]

Attachment: add rank to weekly comments, for top 50 bugs

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8807774 [details] [diff] [review]
add rank to weekly comments, for top 50 bugs

Review of attachment 8807774 [details] [diff] [review]:
-----------------------------------------------------------------

::: woo_commenter.py
@@ +107,5 @@
>              continue
> +        rank_message = None
> +        if options.weekly_mode and (bug_id, counts['total']) in top[:RANK_THRESHOLD]:
> +            rank = top.index((bug_id, counts['total']))+1
> +            rank_message = "\n\nThis bug is ranked #%d in this week's War on Orange." % rank

to keep the text simple, could we leave out War on Orange?

Maybe:
This bug is ranked #%d for the past week

Comment 5

[Geoff Brown [:gbrown]]

Attachment: add rank to weekly comments, for top 50 bugs

For example:

# Bug 1313136:
39 failures in 606 pushes (0.064 failures/push) were associated with this bug in the last 7 days. 

This is the #42 most frequent failure this week.

Repository breakdown:
* autoland: 18
* mozilla-inbound: 12
* mozilla-central: 6
* try: 3

Platform breakdown:
* linux64: 29
* linux32: 9
* android-4-3-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1313136&startday=2016-10-31&endday=2016-11-06&tree=all

Comment 6

[Ed Morley [:emorley]]

Comment on attachment 8808421 [details] [diff] [review]
add rank to weekly comments, for top 50 bugs

Review of attachment 8808421 [details] [diff] [review]:
-----------------------------------------------------------------

::: woo_commenter.py
@@ +99,4 @@
>      tb = TopBugs(local_server_url, start_date, end_date, tree='all')
>      stats = tb.stats()
>      bug_stats = tb.stats_by_bug()
> +    top = tb.top_bugs()

What do you think about moving the `weekly_mode` conditional and array slicing to here instead? This would reduce the time taken for the script to run in the most common case (daily mode).

Comment 7

[Ed Morley [:emorley]]

For this and bug 1315110, I haven't tested locally. Happy for you to land and either deploy yourself if you have access to brasstacks, or else I can deploy for you.

Comment 8

[Geoff Brown [:gbrown]]

(In reply to Ed Morley [:emorley] from comment #6)
> What do you think about moving the `weekly_mode` conditional and array
> slicing to here instead? This would reduce the time taken for the script to
> run in the most common case (daily mode).

Good idea - will do.

Comment 9

[Geoff Brown [:gbrown]]

(In reply to Ed Morley [:emorley] from comment #7)
> For this and bug 1315110, I haven't tested locally. Happy for you to land
> and either deploy yourself if you have access to brasstacks, or else I can
> deploy for you.

I have tested locally with --test in both daily and weekly modes -- seems fine.

I will wait a few days to land, in case additional OF ideas develop. Will ping you for deployment - thanks!

Comment 10

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/automation/orangefactor/rev/4a6e98d39a8e1e331d17fe87ce3e29ea6998b9dd