Improve the message in the xml output from an unexpected pass when using runtests.py

RESOLVED INVALID

Status

Testing
Marionette
RESOLVED INVALID
4 years ago
3 years ago

People

(Reporter: bsilverberg, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [runner])

(Reporter)

Description

4 years ago
Currently when a test unexpectedly passes, the following is written to the xml file (this is an example):

<testcase classname="test_cards_view_with_three_apps.TestCardsViewThreeApps" name="test_cards_view" time="0">
	<failure message="test failure">N/A</failure>
</testcase>

It would be preferable to record the failure as:

	<failure message="test failure">Unexpected Pass</failure>

This would make reviewing the failures on Jenkins more straighforward.

Dave Hunt pointed at two different components which come into play:

http://hg.mozilla.org/mozilla-central/file/default/testing/marionette/client/marionette/runtests.py#l726
https://github.com/mozilla/mozbase/blob/57a1c6dd00cc945e3274b0b17aba93fb3000cc43/moztest/moztest/results.py#L316
Whiteboard: [runner]

Comment 1

3 years ago
I think the recent change to structured logging will have resolved this.

Ahal, is that true? If so can you close this bug?
Flags: needinfo?(ahalberstadt)
Yes, I believe the built-in marionette formatters have been replaced by the mozlog equivalents, in this case:
http://mozbase.readthedocs.org/en/latest/mozlog_structured.html#mozlog.structured.formatters.xunit.XUnitFormatter

That being said, I've never used that formatter, so maybe it has the same problem. I think if it does we can open a new bug for it.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Flags: needinfo?(ahalberstadt)
Resolution: --- → INVALID
You need to log in before you can comment on or make changes to this bug.