Closed Bug 758452 Opened 8 years ago Closed 7 years ago

Add xUnit reporting to Marionette

Categories

(Testing :: Marionette, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED
mozilla16

People

(Reporter: jgriffin, Assigned: mihneadb)

References

Details

Attachments

(1 file, 3 obsolete files)

We should add xUnit test reporting to Marionette.  See http://xunit.codeplex.com/wikipage?title=XmlFormat.

If we report results in xUnit format, Jenkins can display the results prettily; otherwise, we need to rely on autolog or looking at the console output.

This shouldn't be too hard; we already collect the results to report to Autolog; we just need to dump them in a different format as well.
There is http://hg.mozilla.org/qa/mozmill-automation/file/tip/libs/report.py that is used with mozmill. If we don't have to worry about adding things ourselves there are python testing frameworks that can report this for us.
Mihnea, can you take a look at this bug while we're waiting for the emulator problems to be resolved?

Marionette currently uses a unittest TextTestRunner (see http://mxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/runtests.py#78); you could probably plug in xUnit support here.
Assignee: nobody → mbalaur
Attached patch xml output per test suite (obsolete) — Splinter Review
Attachment #639863 - Flags: feedback?(jgriffin)
I had to add both --xml and --xml-file because optparse does not support a way in which an option can have a default value that will not be used if the option is not present.

As in (considering we only had --xml with a default value of output.xml):
./py --xml # would work, creating output.xml
./py --xml asd.xml # would work, creating asd.xml
./py # this would also create output.xml although we don't want it to

This is why there is one option (--xml) with a flag role.
Attachment #639863 - Attachment is obsolete: true
Attachment #639863 - Flags: feedback?(jgriffin)
Comment on attachment 639886 [details] [diff] [review]
xUnit XML output for whole runs (including tests from more than one file)

Review of attachment 639886 [details] [diff] [review]:
-----------------------------------------------------------------

Looking very good.  I think we should make a couple of small changes:

When looking for failures, we should include results.unexpectedSuccesses; when looking for skipped tests, we should include results.expectedFailures.

Also, I think it's a little awkward to have both --xml and --xml-file; let's just collapse it into one option, --xml-output, which doesn't have a default value (so that the user is forced to specify one).
Attachment #639886 - Flags: review-
One nit: the import xml.dom.minidom should be at the top of the file for readability.
Attached patch update according to the reviews (obsolete) — Splinter Review
Attachment #639886 - Attachment is obsolete: true
Attachment #640333 - Flags: review?(jgriffin)
Comment on attachment 640333 [details] [diff] [review]
update according to the reviews

Review of attachment 640333 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good; one last change:  the total number of failed and skipped tests should take the unexpectedFailures and unexpectedSuccesses into account.  r+ with that change.
Attachment #640333 - Flags: review?(jgriffin) → review+
Attachment #640354 - Flags: review?(jgilbert)
Attachment #640354 - Flags: review?(jgilbert) → review?(jgriffin)
Attachment #640333 - Attachment is obsolete: true
Comment on attachment 640354 [details] [diff] [review]
add all tests to the counts

Review of attachment 640354 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good, I'll land it.
Attachment #640354 - Flags: review?(jgriffin) → review+
https://hg.mozilla.org/mozilla-central/rev/2c74d07ba892
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Blocks: 773394
You need to log in before you can comment on or make changes to this bug.