Closed Bug 773394 Opened 13 years ago Closed 7 years ago

Consolidate xUnit Python code

Categories

(Testing :: General, defect)

Product:
Testing
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: gps, Unassigned)

References

Details

Bug 725478 added xUnit file writing to the xpcshell harness. Bug 758452 recently added it to Marionette, duplicating the logic of writing the XML in the process. This is a DRY violation and the code should be consolidated. In bug 729098, I realized I was being silly by putting the xUnit writing code in the xpcshell test runner itself, constraining its use to there, so I refactored it into build/automationutils.py. Unfortunately, that patch never landed. The current patch on bug 729098 (the one that touches make files in case you read this in the distant future) will probably be abandoned in favor of a solution that integrates with bug 751795. Specifically, the Python would live in build/pylib, not some random .py file in build/. So, options for this bug. If done in the near future, you could probably take the xUnit Python refactoring from bug 729098 and just submit it as a patch here. If bug 751795 lands, you should probably move the xUnit code to build/pylib/mozbuild or perhaps create a build/pylib/moztesting or something. Long term, I think it would be worthwhile to have a few generic Python classes that represent test suites and test case executions. Each test runner could e.g. instantiate a new TestCaseResult instance and populate test output, metadata, etc. These could be collected in a TestSuiteResult, which would have a toXUnit() method or something. I'm not sure if we have something like that today (I know somewhere we have a somewhat unified framework for running tests, but am not sure how the results are modeled). Regardless, that isn't strictly required for this bug. But, if you feel like writing something, I think it would be a welcome addition.
Assignee: nobody → mbalaur
I will create a method that lives in mozlog/xUnit.py that takes as input some test results and returns the XML document generated with those results.
I don't think mozlog is semantically the proper place for this. Where is all the Python code for driving unit tests? I think it should go there.
It could possibly go in automationutils.py. Jeff, do you have an opinion on where common xUnit-generating code should live?
Mozbase (basically) is our python code for driving unit tests, and mozlog is a part of that. We are trying to phase out automation.py so I would prefer not to see it land in automationutils.
I am going to advocate for the creation of moztesting (a mozbase component). I think it would be great to have an abstract representation of a test suite, test case, and test results. Even if it's only xUnit code today, that's a good start. And, it provides an obvious place for us to land code related to testing. AFAIK all this code today is scattered across the various test runners. Surely there is some overlap and some unification wins that can result from running on the same code base.
(In reply to Gregory Szorc [:gps] from comment #5) > I am going to advocate for the creation of moztesting (a mozbase component). > I think it would be great to have an abstract representation of a test > suite, test case, and test results. Even if it's only xUnit code today, > that's a good start. And, it provides an obvious place for us to land code > related to testing. AFAIK all this code today is scattered across the > various test runners. Surely there is some overlap and some unification wins > that can result from running on the same code base. I agree. Posting test results to autolog could also live in this component, as that logic is duplicated in N places right now.
(In reply to Jonathan Griffin (:jgriffin) from comment #6) > (In reply to Gregory Szorc [:gps] from comment #5) > > I am going to advocate for the creation of moztesting (a mozbase component). > > I think it would be great to have an abstract representation of a test > > suite, test case, and test results. Even if it's only xUnit code today, > > that's a good start. And, it provides an obvious place for us to land code > > related to testing. AFAIK all this code today is scattered across the > > various test runners. Surely there is some overlap and some unification wins > > that can result from running on the same code base. > > I agree. Posting test results to autolog could also live in this component, > as that logic is duplicated in N places right now. +1 from me too.
So then I'll make a moztesting component to mozbase that contains something like [1]. Is that ok? [1] http://mxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/runtests.py#34
+1 Though if it's just subclassing unittest.py I'd advocate calling it "mozunit" to be consistent with other unit testing frameworks (i.e pyunit, junit, etc).
FWIW... A. I'm not sure if what we're talking about is just subclassing unittest.py. Even if it starts that way, is that what it will be in a month? B. ... I'm also fairly unconvinced in naming things after "unit" tests that aren't (necessarily) unit tests. Unit testing has a fairly strict definition: http://en.wikipedia.org/wiki/Unit_testing ; and just because everyone abuses the name doesn't mean we should. <bikeshed> My preferences are, in order, MozResults MozTestResults MozTest
+1 for MozTest. We don't want to limit ourselves to just results. I think there are execution aspects of tests we may want to eventually add to this package.
(In reply to Gregory Szorc [:gps] from comment #11) > +1 for MozTest. We don't want to limit ourselves to just results. I think > there are execution aspects of tests we may want to eventually add to this > package. I'd be inclined to think of execution as yet a separate package, but am open to possibilities here. I'm going to ticket the creation of this package separately as I think we've exceeded the scope of this bug. I also posted to the tools group about this: https://groups.google.com/forum/#!topic/mozilla.tools/FJYN4wg4erk
(In reply to Jeff Hammel [:jhammel] from comment #12) > (In reply to Gregory Szorc [:gps] from comment #11) > > +1 for MozTest. We don't want to limit ourselves to just results. I think > > there are execution aspects of tests we may want to eventually add to this > > package. > > I'd be inclined to think of execution as yet a separate package, but am open > to possibilities here. > > I'm going to ticket the creation of this package separately as I think we've > exceeded the scope of this bug. I also posted to the tools group about > this: https://groups.google.com/forum/#!topic/mozilla.tools/FJYN4wg4erk Filed: https://bugzilla.mozilla.org/show_bug.cgi?id=774419
See Also: → 774419
+1 for MozTest, a base class of a testing framework that uses all the mozbase modules. I think the time has come. :D
Moztest has xUnit reporting. https://github.com/mozilla/mozbase/tree/master/moztest
Assignee: mihneadb → nobody
Mass closing bugs with no activity in 2+ years. If this bug is important to you, please re-open.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.