1382273 - Mark high-frequency DOM bugs for triage based on daily threshold also

Status: RESOLVED FIXED

(Tree Management Graveyard :: OrangeFactor, defect)

Description

[Geoff Brown [:gbrown]]

With bug 1380439, the OF commenter marks high frequency bugs (restricted to certain DOM components for a trial period) for triage, based on the weekly orange count surpassing a threshold.

For very high frequency failures, it may be important to act sooner than once a week. Let's consider making the decision on the daily commenter runs also.

Is this a good idea?

What should the daily failure threshold be? (The current threshold for daily comments is 15 failures.)

Comment 1

[Geoff Brown [:gbrown]]

Attachment: update priority/needswork for DOM trial based on daily threshold

I think it makes sense to go ahead with a daily threshold of 15 failures: 

If it's in a DOM component and the OF robot is making a daily comment (15+ failures yesterday) or a high priority weekly comment (30+ failures in the last 7 days), then priority and needswork:DOM are adjusted if necessary.

In a test run today, only a couple of DOM changes were found on the daily run:
  bug 1338339 - priority + needswork:DOM
  bug 1356014 - needswork:DOM only


Since I'll be away for a few days, I won't deploy this before Tuesday.

Comment 2

[Ed Morley [:emorley]]

Comment on attachment 8888356 [details] [diff] [review]
update priority/needswork for DOM trial based on daily threshold

Review of attachment 8888356 [details] [diff] [review]:
-----------------------------------------------------------------

::: woo_commenter.py
@@ +171,5 @@
>                                 platforms=dict_to_sorted_list(counts['per_platform']))
>          params = {'comment': {'body': text} }
>          # DOM triage updates to priority and whiteboard
> +        if ((options.weekly_mode and (counts['total'] >= PRIORITY2_THRESHOLD)) or
> +            (not options.weekly_mode)):

This is equivalent to:
```
        if (counts['total'] >= PRIORITY2_THRESHOLD) or not options.weekly_mode:
```

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8888356 [details] [diff] [review]
update priority/needswork for DOM trial based on daily threshold

Review of attachment 8888356 [details] [diff] [review]:
-----------------------------------------------------------------

::: woo_commenter.py
@@ +171,5 @@
>                                 platforms=dict_to_sorted_list(counts['per_platform']))
>          params = {'comment': {'body': text} }
>          # DOM triage updates to priority and whiteboard
> +        if ((options.weekly_mode and (counts['total'] >= PRIORITY2_THRESHOLD)) or
> +            (not options.weekly_mode)):

I think this logic is right if you look at the parenthesis.

Comment 4

[Geoff Brown [:gbrown]]

(In reply to Ed Morley [:emorley] from comment #2)
> This is equivalent to:
> ```
>         if (counts['total'] >= PRIORITY2_THRESHOLD) or not options.weekly_mode:
> ```

That is more concise -- will make that change on landing.

Comment 5

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/automation/orangefactor/rev/ba322694a641041ef23aef8dbd3dc030499dca75

Comment 6

[Geoff Brown [:gbrown]]

Deployed -- will run tonight.

Comment 7

[Geoff Brown [:gbrown]]

Verified: bug 1383512 marked for triage tonight.