Closed Bug 758452 Opened 8 years ago Closed 7 years ago

Add xUnit reporting to Marionette


(Testing :: Marionette, defect)

Not set


(Not tracked)



(Reporter: jgriffin, Assigned: mihneadb)




(1 file, 3 obsolete files)

We should add xUnit test reporting to Marionette.  See

If we report results in xUnit format, Jenkins can display the results prettily; otherwise, we need to rely on autolog or looking at the console output.

This shouldn't be too hard; we already collect the results to report to Autolog; we just need to dump them in a different format as well.
There is that is used with mozmill. If we don't have to worry about adding things ourselves there are python testing frameworks that can report this for us.
Mihnea, can you take a look at this bug while we're waiting for the emulator problems to be resolved?

Marionette currently uses a unittest TextTestRunner (see; you could probably plug in xUnit support here.
Assignee: nobody → mbalaur
Attached patch xml output per test suite (obsolete) — Splinter Review
Attachment #639863 - Flags: feedback?(jgriffin)
I had to add both --xml and --xml-file because optparse does not support a way in which an option can have a default value that will not be used if the option is not present.

As in (considering we only had --xml with a default value of output.xml):
./py --xml # would work, creating output.xml
./py --xml asd.xml # would work, creating asd.xml
./py # this would also create output.xml although we don't want it to

This is why there is one option (--xml) with a flag role.
Attachment #639863 - Attachment is obsolete: true
Attachment #639863 - Flags: feedback?(jgriffin)
Comment on attachment 639886 [details] [diff] [review]
xUnit XML output for whole runs (including tests from more than one file)

Review of attachment 639886 [details] [diff] [review]:

Looking very good.  I think we should make a couple of small changes:

When looking for failures, we should include results.unexpectedSuccesses; when looking for skipped tests, we should include results.expectedFailures.

Also, I think it's a little awkward to have both --xml and --xml-file; let's just collapse it into one option, --xml-output, which doesn't have a default value (so that the user is forced to specify one).
Attachment #639886 - Flags: review-
One nit: the import xml.dom.minidom should be at the top of the file for readability.
Attached patch update according to the reviews (obsolete) — Splinter Review
Attachment #639886 - Attachment is obsolete: true
Attachment #640333 - Flags: review?(jgriffin)
Comment on attachment 640333 [details] [diff] [review]
update according to the reviews

Review of attachment 640333 [details] [diff] [review]:

Looks good; one last change:  the total number of failed and skipped tests should take the unexpectedFailures and unexpectedSuccesses into account.  r+ with that change.
Attachment #640333 - Flags: review?(jgriffin) → review+
Attachment #640354 - Flags: review?(jgilbert)
Attachment #640354 - Flags: review?(jgilbert) → review?(jgriffin)
Attachment #640333 - Attachment is obsolete: true
Comment on attachment 640354 [details] [diff] [review]
add all tests to the counts

Review of attachment 640354 [details] [diff] [review]:

Looks good, I'll land it.
Attachment #640354 - Flags: review?(jgriffin) → review+
Closed: 7 years ago
Resolution: --- → FIXED
Blocks: 773394
You need to log in before you can comment on or make changes to this bug.