1301182 - Add test result summary to Marionette harness test logs

Status: NEW

(Testing :: Marionette Client and Harness, defect, P5)

Description

[Anjana Vakil (:vakila)]

Attachment: Example summary from Mozlog mach-formatted log

When the Marionette harness tests logs are output in Mozlog's TBPL format (see Bug 1285299 and Bug 1301177), no test result summary is recorded.  

We should include a result summary similar to the summaries given in the Marionette logs (written by `BaseMarionetteTestRunner._print_summary` [1]) and Mozlog's mach-formatted logs (written by `MachFormatter.suite_end` [2]). An example summary (in mozlog-mach style) is attached.

Some ideas for how this could be approached (just spitballing):
- Write a custom plugin (possibly extending the `MozLog` class of the pytest_mozlog plugin - see Bug 1285299) which outputs a summary to the same stream as that used by the TBPL logger in the harness test logs 
- Create a new Mozlog formatter extending `TbplFormatter` [3] that adds a summary report at suite end in the same way `MachFormatter` [2] does


[1] https://dxr.mozilla.org/mozilla-central/rev/91c2b9d5c1354ca79e5b174591dbb03b32b15bbf/testing/marionette/harness/marionette/runner/base.py#933

[2] https://dxr.mozilla.org/mozilla-central/rev/91c2b9d5c1354ca79e5b174591dbb03b32b15bbf/testing/mozbase/mozlog/mozlog/formatters/machformatter.py#115

[3] https://dxr.mozilla.org/mozilla-central/rev/91c2b9d5c1354ca79e5b174591dbb03b32b15bbf/testing/mozbase/mozlog/mozlog/formatters/tbplformatter.py#21

Comment 1

[Anjana Vakil (:vakila)]

Example summary: (Copying here instead of in attachment for easier reference)


Summary
=======

Ran 68 tests
Expected results: 59
Unexpected results: 4 (ERROR: 1, FAIL: 2, PASS: 1)
Skipped: 5

Unexpected Results
==================

FAIL test_logging.py test_sample_test_fail
FAIL test_logging.py test_sample_test_exception
PASS expected FAIL test_logging.py test_sample_unexpected_pass
ERROR test_logging.py test_setup_error

Comment 2

[David Burns :automatedtester]

[mass update] Setting priority