758452 - Add xUnit reporting to Marionette

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect)

Description

[Jonathan Griffin (:jgriffin)]

We should add xUnit test reporting to Marionette.  See http://xunit.codeplex.com/wikipage?title=XmlFormat.

If we report results in xUnit format, Jenkins can display the results prettily; otherwise, we need to rely on autolog or looking at the console output.

This shouldn't be too hard; we already collect the results to report to Autolog; we just need to dump them in a different format as well.

Comment 1

[David Burns :automatedtester]

There is http://hg.mozilla.org/qa/mozmill-automation/file/tip/libs/report.py that is used with mozmill. If we don't have to worry about adding things ourselves there are python testing frameworks that can report this for us.

Comment 2

[Jonathan Griffin (:jgriffin)]

Mihnea, can you take a look at this bug while we're waiting for the emulator problems to be resolved?

Marionette currently uses a unittest TextTestRunner (see http://mxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/runtests.py#78); you could probably plug in xUnit support here.

Comment 3

[Mihnea Dobrescu-Balaur (:mihneadb)]

Attachment: xml output per test suite

Comment 4

[Mihnea Dobrescu-Balaur (:mihneadb)]

Attachment: xUnit XML output for whole runs (including tests from more than one file)

I had to add both --xml and --xml-file because optparse does not support a way in which an option can have a default value that will not be used if the option is not present.

As in (considering we only had --xml with a default value of output.xml):
./py --xml # would work, creating output.xml
./py --xml asd.xml # would work, creating asd.xml
./py # this would also create output.xml although we don't want it to

This is why there is one option (--xml) with a flag role.

Comment 5

[Jonathan Griffin (:jgriffin)]

Comment on attachment 639886 [details] [diff] [review]
xUnit XML output for whole runs (including tests from more than one file)

Review of attachment 639886 [details] [diff] [review]:
-----------------------------------------------------------------

Looking very good.  I think we should make a couple of small changes:

When looking for failures, we should include results.unexpectedSuccesses; when looking for skipped tests, we should include results.expectedFailures.

Also, I think it's a little awkward to have both --xml and --xml-file; let's just collapse it into one option, --xml-output, which doesn't have a default value (so that the user is forced to specify one).

Comment 6

[Andrew Halberstadt [:ahal]]

One nit: the import xml.dom.minidom should be at the top of the file for readability.

Comment 7

[Mihnea Dobrescu-Balaur (:mihneadb)]

Attachment: update according to the reviews

Comment 8

[Jonathan Griffin (:jgriffin)]

Comment on attachment 640333 [details] [diff] [review]
update according to the reviews

Review of attachment 640333 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good; one last change:  the total number of failed and skipped tests should take the unexpectedFailures and unexpectedSuccesses into account.  r+ with that change.

Comment 9

[Mihnea Dobrescu-Balaur (:mihneadb)]

Attachment: add all tests to the counts

Comment 10

[Jonathan Griffin (:jgriffin)]

Comment on attachment 640354 [details] [diff] [review]
add all tests to the counts

Review of attachment 640354 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good, I'll land it.

Comment 11

[Jonathan Griffin (:jgriffin)]

http://hg.mozilla.org/integration/mozilla-inbound/rev/2c74d07ba892

Comment 12

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/2c74d07ba892