If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Add xUnit reporting to Marionette

RESOLVED FIXED in mozilla16

Status

Testing
Marionette
RESOLVED FIXED
5 years ago
5 years ago

People

(Reporter: jgriffin, Assigned: mihneadb)

Tracking

(Blocks: 1 bug)

unspecified
mozilla16
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 3 obsolete attachments)

(Reporter)

Description

5 years ago
We should add xUnit test reporting to Marionette.  See http://xunit.codeplex.com/wikipage?title=XmlFormat.

If we report results in xUnit format, Jenkins can display the results prettily; otherwise, we need to rely on autolog or looking at the console output.

This shouldn't be too hard; we already collect the results to report to Autolog; we just need to dump them in a different format as well.
There is http://hg.mozilla.org/qa/mozmill-automation/file/tip/libs/report.py that is used with mozmill. If we don't have to worry about adding things ourselves there are python testing frameworks that can report this for us.
(Reporter)

Comment 2

5 years ago
Mihnea, can you take a look at this bug while we're waiting for the emulator problems to be resolved?

Marionette currently uses a unittest TextTestRunner (see http://mxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/runtests.py#78); you could probably plug in xUnit support here.
Assignee: nobody → mbalaur
(Assignee)

Comment 3

5 years ago
Created attachment 639863 [details] [diff] [review]
xml output per test suite
Attachment #639863 - Flags: feedback?(jgriffin)
(Assignee)

Comment 4

5 years ago
Created attachment 639886 [details] [diff] [review]
xUnit XML output for whole runs (including tests from more than one file)

I had to add both --xml and --xml-file because optparse does not support a way in which an option can have a default value that will not be used if the option is not present.

As in (considering we only had --xml with a default value of output.xml):
./py --xml # would work, creating output.xml
./py --xml asd.xml # would work, creating asd.xml
./py # this would also create output.xml although we don't want it to

This is why there is one option (--xml) with a flag role.
Attachment #639863 - Attachment is obsolete: true
Attachment #639863 - Flags: feedback?(jgriffin)
(Reporter)

Comment 5

5 years ago
Comment on attachment 639886 [details] [diff] [review]
xUnit XML output for whole runs (including tests from more than one file)

Review of attachment 639886 [details] [diff] [review]:
-----------------------------------------------------------------

Looking very good.  I think we should make a couple of small changes:

When looking for failures, we should include results.unexpectedSuccesses; when looking for skipped tests, we should include results.expectedFailures.

Also, I think it's a little awkward to have both --xml and --xml-file; let's just collapse it into one option, --xml-output, which doesn't have a default value (so that the user is forced to specify one).
Attachment #639886 - Flags: review-
One nit: the import xml.dom.minidom should be at the top of the file for readability.
(Assignee)

Comment 7

5 years ago
Created attachment 640333 [details] [diff] [review]
update according to the reviews
Attachment #639886 - Attachment is obsolete: true
Attachment #640333 - Flags: review?(jgriffin)
(Reporter)

Comment 8

5 years ago
Comment on attachment 640333 [details] [diff] [review]
update according to the reviews

Review of attachment 640333 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good; one last change:  the total number of failed and skipped tests should take the unexpectedFailures and unexpectedSuccesses into account.  r+ with that change.
Attachment #640333 - Flags: review?(jgriffin) → review+
(Assignee)

Comment 9

5 years ago
Created attachment 640354 [details] [diff] [review]
add all tests to the counts
Attachment #640354 - Flags: review?(jgilbert)
(Assignee)

Updated

5 years ago
Attachment #640354 - Flags: review?(jgilbert) → review?(jgriffin)
(Assignee)

Updated

5 years ago
Attachment #640333 - Attachment is obsolete: true
(Reporter)

Comment 10

5 years ago
Comment on attachment 640354 [details] [diff] [review]
add all tests to the counts

Review of attachment 640354 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good, I'll land it.
Attachment #640354 - Flags: review?(jgriffin) → review+
(Reporter)

Comment 11

5 years ago
http://hg.mozilla.org/integration/mozilla-inbound/rev/2c74d07ba892
Target Milestone: --- → mozilla16
https://hg.mozilla.org/mozilla-central/rev/2c74d07ba892
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED

Updated

5 years ago
Blocks: 773394
You need to log in before you can comment on or make changes to this bug.