Closed Bug 1349516 Opened 7 years ago Closed 6 years ago
[mozlog] Add support to pytest-mozlog for tests rerun using pytest-rerunfailures
59 bytes, text/x-review-board-request
Tests that are automatically rerun using pytest-rerunfailures  do not currently show up in the logs generated by pytest-mozlog, which makes it difficult to identify flaky tests that are frequently rerun. Fixing this may involve patching pytest-rerunfailures, but will also likely involve some work to mozlog. For each rerun we would expect additional test start and end log message, ideally with an indication that the test is a rerun. We should also discuss if we want to allow a test outcome of RERUN, or if we'll expect that to be inferred by the multiple start/end message for a test.  https://github.com/pytest-dev/pytest-rerunfailures
Comment on attachment 8888386 [details] Bug 1349516 - Add support to pytest-mozlog for tests rerun using pytest-rerunfailures; https://reviewboard.mozilla.org/r/159336/#review164740 ::: testing/mozbase/mozlog/mozlog/pytest_mozlog/plugin.py:85 (Diff revision 1) > status = expected = 'PASS' > message = stack = None > if hasattr(report, 'wasxfail'): > expected = 'FAIL' > - if report.failed: > + if report.failed or report.outcome == 'rerun': > status = 'FAIL' if report.when == 'call' else 'ERROR' I decided to set the status of rerun tests to FAIL instead of a new status of RERUN. This is accurate to the result of the test, and means we didn't need to add a new status. Reruns can be identified by tests occurring multiple times in the log.
Comment on attachment 8888386 [details] Bug 1349516 - Add support to pytest-mozlog for tests rerun using pytest-rerunfailures; https://reviewboard.mozilla.org/r/159336/#review164746 I can't say I understand how this fix works, but it looks good to me!
Attachment #8888386 - Flags: review?(ahalberstadt) → review+
Pushed by email@example.com: https://hg.mozilla.org/integration/autoland/rev/d6eb564eef59 Add support to pytest-mozlog for tests rerun using pytest-rerunfailures; r=ahal
You need to log in before you can comment on or make changes to this bug.