942910 - Improve the message in the xml output from an unexpected pass when using runtests.py

Status: RESOLVED INVALID

(Remote Protocol :: Marionette, defect)

Description

[Bob Silverberg [:bsilverberg]]

Currently when a test unexpectedly passes, the following is written to the xml file (this is an example):

<testcase classname="test_cards_view_with_three_apps.TestCardsViewThreeApps" name="test_cards_view" time="0">
	<failure message="test failure">N/A</failure>
</testcase>

It would be preferable to record the failure as:

	<failure message="test failure">Unexpected Pass</failure>

This would make reviewing the failures on Jenkins more straighforward.

Dave Hunt pointed at two different components which come into play:

http://hg.mozilla.org/mozilla-central/file/default/testing/marionette/client/marionette/runtests.py#l726
https://github.com/mozilla/mozbase/blob/57a1c6dd00cc945e3274b0b17aba93fb3000cc43/moztest/moztest/results.py#L316

Comment 1

[Zac C (:zac)]

I think the recent change to structured logging will have resolved this.

Ahal, is that true? If so can you close this bug?

Comment 2

[Andrew Halberstadt [:ahal]]

Yes, I believe the built-in marionette formatters have been replaced by the mozlog equivalents, in this case:
http://mozbase.readthedocs.org/en/latest/mozlog_structured.html#mozlog.structured.formatters.xunit.XUnitFormatter

That being said, I've never used that formatter, so maybe it has the same problem. I think if it does we can open a new bug for it.