1850785 - consider tracking marionette intermittent failure bugs as single tracking bugs

Status: RESOLVED FIXED

(Testing :: General, task, P2)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

in any given week we have upwards of 20 unique intermittent failure bugs related to marionette test cases.

For other test harnesses we assume that any failure of a given test case (filename) falls into a bucket and we track them all together. So an example would be:
test_foo.html | timed out
test_foo.html | failed to set widget bar
test_foo.html | assertion count 1 is great than 0

so we would then have:
test_foo.html | single tracking bug

and then collect all the failures together. This is beneficial as a developer will be editing test_foo.html to work on one failure, and could determine if it is worth the time to fix a secondary error. i.e. we don't lose track of the other stuff, but instead we can see it all in one place.

For Marionette tests, I see a batch of bug summaries that looks like this:

Intermittent layout/base/tests/marionette/test_accessiblecaret_selection_mode.py AccessibleCaretSelectionModeTestCase.test_minimum_select_one_character2_textarea2 | AssertionError: u'F' != u'\r\n'
Intermittent layout/base/tests/marionette/test_accessiblecaret_selection_mode.py AccessibleCaretSelectionModeTestCase.test_minimum_select_one_character_textarea | AssertionError: 'A' != ''
Intermittent layout/base/tests/marionette/test_accessiblecaret_selection_mode.py AccessibleCaretSelectionModeTestCase.test_minimum_select_one_character_textarea_disabled | AssertionError: 'A' != 'GHI JKL MNO PQR'
Intermittent layout/base/tests/marionette/test_accessiblecaret_selection_mode.py AccessibleCaretSelectionModeTestCase.test_minimum_select_one_character_textarea_rtl | marionette_driver.errors.JavascriptException: [Exception... "Unexpected error"  nsresult
Intermittent layout/base/tests/marionette/test_accessiblecaret_selection_mode.py AccessibleCaretSelectionModeTestCase.test_minimum_select_one_character_textarea_disabled | AssertionError: 'A' != ''
Intermittent layout/base/tests/marionette/test_accessiblecaret_selection_mode.py AccessibleCaretSelectionModeTestCase.test_minimum_select_one_character_textarea_disabled | marionette_driver.errors.JavascriptException: [Exception... "Unexpected error" nsr
Intermittent layout/base/tests/marionette/test_accessiblecaret_selection_mode.py AccessibleCaretSelectionModeTestCase.test_carets_not_show_after_key_scroll_down_and_up | AssertionError: 'AAAAA\nBBBBB' == 'AAAAA\nBBBBB'

would it make sense to have a single bug:
Intermittent layout/base/tests/marionette/test_accessiblecaret_selection_mode.py | single tracking bug

or is there a different layer of granularity that makes sense for most of the marionette tests?

:whimboo, I would really appreciate your input on what would make sense here

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

If the failures are happening all within the same file I think that we can apply the same rules as for other test failures that cause single tracking bugs to be created. Then the triage owner will have to check if parts of the failures or all can be fixed. Usually we file new bugs for fixes that block the single failure bugs which means that we can keep the latter open way longer and no new such bug has to be filed.

So go ahead and apply these rules and we will adapt our work process. It's better to have general rules as test harness specific ones.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Actually let us discuss this in the next triage meeting on Monday. And we can give a team-wide feedback on that.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

sure; I will stay tuned!

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Joel, this proposal sounds fine to us. So lets go ahead and allow sheriffs to file single tracking bugs.

Regarding the handling of those bugs do we have some guidelines for developers and triage owners? If not I wonder if it might make sense to add something to eg. firefox-source-docs. Something like I mentioned in my comment 2 works actually pretty fine for us and allow for re-using the same bug multiple times. I'm personally not a fan of having multiple bugs with the same bug summary filed. What do you think?

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

there are no documents written up- I think making something would be good. I just got my PR to make the new bugs from sheriffs use single tracking bugs added to treeherder staging; next week it will be live and I can add existing bugs to there.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

:whimboo, where would the docs be stored ideally?

I am thinking something like this:

Single tracking bugs are used to track test failures seen in CI.  These are tracked at the test case level (typically path/filename) instead of the error message.  We have found that many times we have >1 bug tracking failures on a test case, but none of the bugs are frequent enough to get attention of the test owners.  In addition, when a developer is looking into fixing an intermittent failure, they are debugging the file, and it is great to see ALL the related failures in one place.

There are 2 ways to get detailed information about test failures:
1) https://treeherder.mozilla.org/intermittent-failures, when viewing a specific issue, there is a table, on the far right side of the table is the column titled `Log`.  If you click the text box underneath that, a drop down of all failure types will be populated, selecting a failure will filter on that failure to see logs, etc.
2) from `mach test-info`, one can view a breakdown of all failures and where they occur (example: `./mach test-info failure-report --bugid 1781668`)

The workflow of single tracking bugs is as follows:
 * Sheriff's find new failures in CI and create new bugs.  If Treeherder can find the path, it will recommend a `single tracking bug` and remove the error message.
 * Sheriff's will annotate existing bugs if the failure to annotate has a test path and in the list of bug suggestions there is a `single tracking bug`.  Treeherder will offer that up as the choice as long as the test paths match (and other criteria like crashes, assertions,leaks, etc. are met)
 * Sheriff's will needinfo triage owner if enough failures occur (currently 30 failures a week, etc.)
 * Developer's will be able to investigate the set of failures, any specific bugs they are fixing (for some or all of the conditions).  It is best practice to use the `single tracking bug` as a META bug and file a new bug blocking the META bug with the specific fix.

NOTE, to make ^ work fully the plan is to make single tracking bugs have the META keyword, and communicate out and adjust any tooling to encourage new bugs for fixes.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

That's a great description Joel and it would be very helpful to have it located at a good location. But I'm not sure what would be the best place for it. Maybe somewhere on https://firefox-source-docs.mozilla.org/ under the Testing & Test Infrastructure category?

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1850785 - add firefox source docs for single tracking bugs. r=aryx!

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c2997b11204b
add firefox source docs for single tracking bugs. r=aryx

Comment 10

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/c2997b11204b