1315281 - Add a special call-to-action message to OrangeFactor Robot comments for frequent failures

Status: RESOLVED FIXED

(Tree Management Graveyard :: OrangeFactor, defect)

Description

[Geoff Brown [:gbrown]]

When an OF bug is in the top 10 of the weekly War on Orange report, let's comment to that effect in the bug. I'm thinking of something like:

"This bug is one of the 10 most frequent test failures this week. Fixing this bug is a priority in the War on Orange."

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

this is simple to do and will be useful:)

What if a bug is already fixed?  Do we still post?  I would really like to be notified of a top N (or count > threshold) bugs which do not have recent activity (outside of woo commenter issues).  Secondarily ones that are not assigned to a developer should be considered in need of attention:)

Comment 2

[Geoff Brown [:gbrown]]

This would be easy, but in light of some other comments from jmaher, I am wondering about the best criteria for the call to action message. The top 10 are certainly important, but also quite restrictive. Using a weekly failure count threshold may be more clear-cut and meaningful, and reach a wider audience:

"This failure happened more than 20 times this week. Fixing this bug is a priority."

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

of course that type of threshold would need to be communicated and agreed upon- "by Feb 28th all tests over 30 instances/week must be fixed unless it occurred in the last few days", then "By March 31st, all tests over 20 instances/week must be fixed ..."

We will find the right metric for development teams to understand.

Comment 4

[Kyle Lahnakoski [:ekyle]]

We could put the NeglectedOranges page onto the big screens in the offices. Instead of comments, we can display all oranges that are open/unassigned. The good thing about the big screens is people may be more likely to assign the bug to themselves to get it off the screen before they fix it.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

I think the neglected oranges might be a good dashboard.  I have been rolling an idea around my head that we could include component/module in the list- that helps point out which team might be responsible- more of a shared responsibility.  Possibly just using the whiteboard tags to point out team names, etc.)

If we do propose a dashboard for the screens, would it make sense to have a dashboard with recent bugs fixed/resolved to give props to people for fixing stuff :)

Comment 6

[Geoff Brown [:gbrown]]

Attachment: add special message to weekly comment when >50 failures

Let's try this: In weekly comments, if we encounter 50 or more failures in the week, add a special message encouraging quick resolution. We currently have about 20 such bugs. Over time, I hope to reduce the threshold to perhaps 20 failures/week.

Example:

# Bug 1315235:
101 failures in 674 pushes (0.15 failures/push) were associated with this bug in the last 7 days. 

This is the #14 most frequent failure this week. 

** This failure happened more than 50 times this week! Resolving this bug is a high priority. **

Repository breakdown:
* mozilla-central: 101

Platform breakdown:
* windows8-64: 17
* windows7-64: 17
* windowsxp: 9
* windows8-32: 9
* windows7-32: 9
* osx-10-9: 8
* osx-10-11: 8
* osx-10-10: 8
* linux64: 8
* linux32: 8

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1315235&startday=2016-11-02&endday=2016-11-08&tree=all

Comment 7

[Ed Morley [:emorley]]

Comment on attachment 8809203 [details] [diff] [review]
add special message to weekly comment when >50 failures

Review of attachment 8809203 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with:

priority = options.weekly_mode and counts['total'] >= PRIORITY_THRESHOLD

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8809203 [details] [diff] [review]
add special message to weekly comment when >50 failures

Review of attachment 8809203 [details] [diff] [review]:
-----------------------------------------------------------------

the only concern here is this message is once/week, not when the bug crosses the threshold.  While mentioning it once/week will help, I see many instances flare up on Monday or Tuesday and if someone is watching we address them, otherwise we move on- in this case we would not know until the following Monday morning when the weekly emails hit our inboxes.

If we were to print this in any comment, my initial thoughts are that it would be overkill to print it in every comment- then again, if the bug continues to go unattended a daily reminder might be enough to nudge people.  For now, I would be fine posting weekly and when we institute a more official change to policy on intermittents we can consider updating the verbiage or frequency.

Comment 9

[Geoff Brown [:gbrown]]

With change from comment 7:

https://hg.mozilla.org/automation/orangefactor/rev/49e68815735598347b7f4e38bafa92a5f36c6aab