1421978 - Hide first xpcshell failure line for test (xpcshell return code: 0) if there are more failure messages for that test (= more detailed)

Status: NEW

(Testing :: XPCShell Harness, enhancement, P3)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

XPCshell failures often have a first failure line like this:

TEST-UNEXPECTED-FAIL | netwerk/test/unit/test_race_cache_with_network.js | xpcshell return code: 0

This indicates that there was a failure. If there is no bug suggested for that, less experienced people might (sheriffs and developers) might file a new bug for this because most of the time, there are further error messages for the test which are more detailed and these should be used for filing bugs, e.g.

TEST-UNEXPECTED-FAIL | netwerk/test/unit/test_race_cache_with_network.js | testSteps - [testSteps : 225] Some of the responses should be from the cache - 0 > 0

For this reason, failure lines with "xpcshell return code: 0" should be hidden if there are more failures lines for the same test.

Example in Treeherder:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=ba283baf4e98aa3a5f45a17981077b98697aa73a&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-searchStr=xpcshell&selectedJob=148757164

Comment 1

[Ed Morley [:emorley]]

I know chasing test harness fixes for these kind of things is tedious (I spent multiple years driving bug 778688 on and off), however in most cases it's still best to fix suboptimal log output directly, rather than wallpaper over it in Treeherder - since:
* hardcoding workarounds in Treeherder is fragile - upstream mozilla-central changes can break them at any time
* fixing the log output helps developers locally, unlike workarounds embedded in Treeherder
* it otherwise puts the burden on the under-staffed Treeherder team rather than the bigger teams that deal with test harnesses/other parts of automation

Comment 2

[Geoff Brown [:gbrown]]

I am disappointed to see so many current xpcshell failures reported against "xpcshell return code", which is rarely informative.

The xpcshell harness does track failure messages with self.has_failure_output:

https://searchfox.org/mozilla-central/rev/da855d65d1fbdd714190cab2c46130f7422f3699/testing/xpcshell/runxpcshelltests.py#173

which might be suitable for determining whether to output an additional error. However, "xpcshell return code" is output via test_end(),

https://searchfox.org/mozilla-central/rev/da855d65d1fbdd714190cab2c46130f7422f3699/testing/xpcshell/runxpcshelltests.py#769.

It is important that the test_end() is not omitted, and it would be inappropriate to force test_end() pass if errors are known...so I'm not sure what can be done in the harness.