1605731 - [mozlog] Unclear what the contract is for multiple runs of the same test

Status: RESOLVED FIXED

(Testing :: Mozbase, defect, P3)

Description

[James Graham [:jgraham]]

In wpt we have a pattern like

for iter in num_iterations:
    logger.suite_start()
    run_tests()
    logger.suite_end()

With --log-mach and num_iterations > 1 this causes an assert:

mozlog.structuredlog: Failure calling log handler:
Traceback (most recent call last):
  File "/home/jgraham/develop/web-platform-tests/_venv/lib/python2.7/site-packages/mozlog/structuredlog.py", line 274, in _handle_log
    handler(data)
  File "/home/jgraham/develop/web-platform-tests/_venv/lib/python2.7/site-packages/mozlog/handlers/base.py", line 74, in __call__
    formatted = self.formatter(data)
  File "/home/jgraham/develop/web-platform-tests/_venv/lib/python2.7/site-packages/mozlog/handlers/base.py", line 42, in __call__
    return self.inner(item)
  File "/home/jgraham/develop/web-platform-tests/_venv/lib/python2.7/site-packages/mozlog/formatters/machformatter.py", line 88, in __call__
    s = super(MachFormatter, self).__call__(data)
  File "/home/jgraham/develop/web-platform-tests/_venv/lib/python2.7/site-packages/mozlog/reader.py", line 74, in __call__
    return handler(data)
  File "/home/jgraham/develop/web-platform-tests/_venv/lib/python2.7/site-packages/mozlog/formatters/machformatter.py", line 137, in suite_end
    self.summary.current))
  File "/home/jgraham/develop/web-platform-tests/_venv/lib/python2.7/site-packages/mozlog/formatters/machformatter.py", line 255, in _format_suite_summary
    assert len(results) == 1
AssertionError

So there are a few possible options here:

• The mach formatter should look at only the latest result for each test (i.e. results[-1] rather than results[0]
• The mach formatter should iterate over all results in the summary
• The mach formatter should reset the summary handler after each suite_end call.
• wpt should move the suite_start and suite_end parts out of the loop

I think the last option is wrong, because generally we want tests to be unique per suite. The third option seems somewhat confusing since we lose the useful behaviour of summarising the complete set of runs when trying to verify a result. So I think one of the first two options is probably best?

Comment 1

[James Graham [:jgraham]]

Attachment: Bug 1605731 - Fix assertion in mach formatter with multiple repetitions of suite,

When we run a suite more than once we can get more than one result for
a test. Iterate over all results when writing the summary rather than
erroring out in this case.

Probably a better solution would be to only write the result from the
current run, but we don't necessarily know what that is, since it's
not tracked by the SummaryHandler.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0c6130ddadd5
Fix assertion in mach formatter with multiple repetitions of suite, r=ahal

Comment 3

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/0c6130ddadd5