1349516 - [mozlog] Add support to pytest-mozlog for tests rerun using pytest-rerunfailures

Status: RESOLVED FIXED

(Testing :: Mozbase, enhancement)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Tests that are automatically rerun using pytest-rerunfailures [1] do not currently show up in the logs generated by pytest-mozlog, which makes it difficult to identify flaky tests that are frequently rerun. Fixing this may involve patching pytest-rerunfailures, but will also likely involve some work to mozlog. For each rerun we would expect additional test start and end log message, ideally with an indication that the test is a rerun. We should also discuss if we want to allow a test outcome of RERUN, or if we'll expect that to be inferred by the multiple start/end message for a test.

[1] https://github.com/pytest-dev/pytest-rerunfailures

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Bug 1349516 - Add support to pytest-mozlog for tests rerun using pytest-rerunfailures;

Review commit: https://reviewboard.mozilla.org/r/159336/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/159336/

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Comment on attachment 8888386 [details]
Bug 1349516 - Add support to pytest-mozlog for tests rerun using pytest-rerunfailures;

https://reviewboard.mozilla.org/r/159336/#review164740

::: testing/mozbase/mozlog/mozlog/pytest_mozlog/plugin.py:85
(Diff revision 1)
>          status = expected = 'PASS'
>          message = stack = None
>          if hasattr(report, 'wasxfail'):
>              expected = 'FAIL'
> -        if report.failed:
> +        if report.failed or report.outcome == 'rerun':
>              status = 'FAIL' if report.when == 'call' else 'ERROR'

I decided to set the status of rerun tests to FAIL instead of a new status of RERUN. This is accurate to the result of the test, and means we didn't need to add a new status. Reruns can be identified by tests occurring multiple times in the log.

Comment 3

[Andrew Halberstadt [:ahal]]

Comment on attachment 8888386 [details]
Bug 1349516 - Add support to pytest-mozlog for tests rerun using pytest-rerunfailures;

https://reviewboard.mozilla.org/r/159336/#review164746

I can't say I understand how this fix works, but it looks good to me!

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d6eb564eef59
Add support to pytest-mozlog for tests rerun using pytest-rerunfailures; r=ahal

Comment 5

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/d6eb564eef59