1117110 - Loop Marionette frontend tests should use structured output and have clearer test failures

Status: RESOLVED INCOMPLETE

(Hello (Loop) :: Client, defect, P4)

Description

[Mark Banner (:standard8)]

From cmanchester on bug 1058038 comment 3:

> If this is providing useful diagnostics, looks good to me. Ongoing
> (under-publicized, I'm realizing) efforts by myself and others mean logging
> "TEST-UNEXPECTED-FAIL" is no longer the primary means for a test harness to
> communicate failure. Test harnesses are being converted to log a raw data
> format for test results that is serialized as JSON and can be made useful to
> a range of consumers. The formatter used by automation translates a failed
> test or error into a "TEST-UNEXPECTED" prefixed line for compatibility with
> tools like tbpl, but the default formatter used by mach does not. You can
> pass "--log-tbpl -" to mach to see how failure output would be formatted in
> this context.

Comment 1

[Mark Banner (:standard8)]

Update. I've found out a bit of background info to this. Marionette already has capability for structured logging. We need to change our code to use that. For example, this is where it logs js tests:

https://hg.mozilla.org/mozilla-central/file/7f343964210b/testing/marionette/client/marionette/marionette_test.py#l570

Whilst we're here we also want to improve our test failure messages. They currently look like:

23:30:11 ERROR - TEST-UNEXPECTED-FAIL | test_desktop_all.py TestDesktopUnits.test_units | AssertionError: build/tests/marionette/tests/browser/components/loop/test/desktop-local/index.html: 1 failure(s) encountered:
23:30:11 ERROR - TEST-UNEXPECTED-FAIL | index.html | should show the gravatars promo box - timeout of 2000ms exceeded. Ensure the done() callback is being called in this test.
23:30:11 INFO - timeout of 2000ms exceeded. Ensure the done() callback is being called in this test. 

They should look more like:

23:30:11 ERROR - TEST-UNEXPECTED-FAIL | test_desktop_all.py TestDesktopUnits.test_units | should show the gravatars promo box - timeout of 2000ms exceeded
<additional failure information>

Given comment 0, the TEST-UNEXPECTED may be obsolete now. The important thing here is the file which failed and the precise test are on the first line. Anything else can follow that.

Comment 2

[Mark Banner (:standard8)]

Support for Hello/Loop has been discontinued.

https://support.mozilla.org/kb/hello-status

Hence closing the old bugs. Thank you for your support.