Closed Bug 1605731 Opened 2 years ago Closed 2 years ago

[mozlog] Unclear what the contract is for multiple runs of the same test

Categories

(Testing :: Mozbase, defect, P3)

Version 3
defect

Tracking

(firefox73 fixed)

RESOLVED FIXED
mozilla73
Tracking Status
firefox73 --- fixed

People

(Reporter: jgraham, Assigned: jgraham)

Details

Attachments

(1 file)

In wpt we have a pattern like

for iter in num_iterations:
    logger.suite_start()
    run_tests()
    logger.suite_end()

With --log-mach and num_iterations > 1 this causes an assert:

mozlog.structuredlog: Failure calling log handler:
Traceback (most recent call last):
  File "/home/jgraham/develop/web-platform-tests/_venv/lib/python2.7/site-packages/mozlog/structuredlog.py", line 274, in _handle_log
    handler(data)
  File "/home/jgraham/develop/web-platform-tests/_venv/lib/python2.7/site-packages/mozlog/handlers/base.py", line 74, in __call__
    formatted = self.formatter(data)
  File "/home/jgraham/develop/web-platform-tests/_venv/lib/python2.7/site-packages/mozlog/handlers/base.py", line 42, in __call__
    return self.inner(item)
  File "/home/jgraham/develop/web-platform-tests/_venv/lib/python2.7/site-packages/mozlog/formatters/machformatter.py", line 88, in __call__
    s = super(MachFormatter, self).__call__(data)
  File "/home/jgraham/develop/web-platform-tests/_venv/lib/python2.7/site-packages/mozlog/reader.py", line 74, in __call__
    return handler(data)
  File "/home/jgraham/develop/web-platform-tests/_venv/lib/python2.7/site-packages/mozlog/formatters/machformatter.py", line 137, in suite_end
    self.summary.current))
  File "/home/jgraham/develop/web-platform-tests/_venv/lib/python2.7/site-packages/mozlog/formatters/machformatter.py", line 255, in _format_suite_summary
    assert len(results) == 1
AssertionError

So there are a few possible options here:

  • The mach formatter should look at only the latest result for each test (i.e. results[-1] rather than results[0]
  • The mach formatter should iterate over all results in the summary
  • The mach formatter should reset the summary handler after each suite_end call.
  • wpt should move the suite_start and suite_end parts out of the loop

I think the last option is wrong, because generally we want tests to be unique per suite. The third option seems somewhat confusing since we lose the useful behaviour of summarising the complete set of runs when trying to verify a result. So I think one of the first two options is probably best?

When we run a suite more than once we can get more than one result for
a test. Iterate over all results when writing the summary rather than
erroring out in this case.

Probably a better solution would be to only write the result from the
current run, but we don't necessarily know what that is, since it's
not tracked by the SummaryHandler.

Assignee: nobody → james
Status: NEW → ASSIGNED
Priority: -- → P3
Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/autoland/rev/0c6130ddadd5
Fix assertion in mach formatter with multiple repetitions of suite, r=ahal
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla73
You need to log in before you can comment on or make changes to this bug.