Closed Bug 1349516 Opened 2 years ago Closed 2 years ago

[mozlog] Add support to pytest-mozlog for tests rerun using pytest-rerunfailures

Categories

(Testing :: Mozbase, enhancement)

Version 3
enhancement
Not set

Tracking

(firefox56 fixed)

RESOLVED FIXED
mozilla56
Tracking Status
firefox56 --- fixed

People

(Reporter: davehunt, Assigned: davehunt, Mentored)

Details

Attachments

(1 file)

Tests that are automatically rerun using pytest-rerunfailures [1] do not currently show up in the logs generated by pytest-mozlog, which makes it difficult to identify flaky tests that are frequently rerun. Fixing this may involve patching pytest-rerunfailures, but will also likely involve some work to mozlog. For each rerun we would expect additional test start and end log message, ideally with an indication that the test is a rerun. We should also discuss if we want to allow a test outcome of RERUN, or if we'll expect that to be inferred by the multiple start/end message for a test.

[1] https://github.com/pytest-dev/pytest-rerunfailures
Assignee: nobody → dave.hunt
Comment on attachment 8888386 [details]
Bug 1349516 - Add support to pytest-mozlog for tests rerun using pytest-rerunfailures;

https://reviewboard.mozilla.org/r/159336/#review164740

::: testing/mozbase/mozlog/mozlog/pytest_mozlog/plugin.py:85
(Diff revision 1)
>          status = expected = 'PASS'
>          message = stack = None
>          if hasattr(report, 'wasxfail'):
>              expected = 'FAIL'
> -        if report.failed:
> +        if report.failed or report.outcome == 'rerun':
>              status = 'FAIL' if report.when == 'call' else 'ERROR'

I decided to set the status of rerun tests to FAIL instead of a new status of RERUN. This is accurate to the result of the test, and means we didn't need to add a new status. Reruns can be identified by tests occurring multiple times in the log.
Comment on attachment 8888386 [details]
Bug 1349516 - Add support to pytest-mozlog for tests rerun using pytest-rerunfailures;

https://reviewboard.mozilla.org/r/159336/#review164746

I can't say I understand how this fix works, but it looks good to me!
Attachment #8888386 - Flags: review?(ahalberstadt) → review+
Pushed by dhunt@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/d6eb564eef59
Add support to pytest-mozlog for tests rerun using pytest-rerunfailures; r=ahal
https://hg.mozilla.org/mozilla-central/rev/d6eb564eef59
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla56
You need to log in before you can comment on or make changes to this bug.