370289 - meta: test output in log needs to be matchable to test identity

Status: RESOLVED DUPLICATE of bug 443090

(Testing :: Mochitest, defect)

Description

[Ray Kiddy]

This meta-bug can be used to track bugs against specific tests that need to change.

When tests run, their log output needs to make it clear what output comes from which test. Tests should log output that makes it clear whether the test passed or failed. These results need to be both visually obvious and we need to be able to parse the results automatically as well.

Note: This is specific to testing and not Testopia. I am parking this in the Testopia component right now.

Comment 1

[Ray Kiddy]

For now, I am just flattening the results of mochitest runs. In other words, the log will show the following:

*** 149 INFO Running /tests/content/base/test/test_bug308484.html...
*** 150 INFO PASS | Setting Host header in unprivileged context
*** 151 INFO PASS | Setting Content-Length header in unprivileged context
*** 152 INFO PASS | Setting Transfer-Encoding header in unprivileged context
*** 153 INFO PASS | Setting Via header in unprivileged context
*** 154 INFO PASS | Setting Upgrade header in unprivileged context
*** 155 INFO PASS | Setting Host header in privileged context
*** 156 INFO PASS | Setting Content-Length header in privileged context
*** 157 INFO PASS | Setting Transfer-Encoding header in privileged context
*** 158 INFO PASS | Setting Via header in privileged context
*** 159 INFO PASS | Setting Upgrade header in privileged context

I turn this into one result:

Test: M content/base/test/test_bug308484.html PASS

If any of the "sub-results" are "FAIL", then the whole test would be marked as having failed.

Comment 2

[Ray Kiddy]

Note: Bug # 379327 logs out individual test names, but on failure only. Writing out all test names seems pretty basic to me, but that's just me.

Comment 3

[(not currently active) Ted Mielczarek]

Picking a random component here.