595984 - Instantiate Mozmill Tests

Status: RESOLVED INCOMPLETE

(Release Engineering :: General, defect)

Description

[John Ford [:jhford] CET/CEST Berlin Time]

Split off from bug 516984.

This bug is to track the buildbotcustom/misc.py work required to get the factory, builders and schedulers set up for mozmill testing.

Comment 1

[Jeff Hammel]

Currently the way that the unittest frameworks (mochitest, xpcshell,reftest) are invoked is the following:- generateTalosBranchObjects is invoked from the buildbot-configs- generateTalosBranchObjects calls generateTestBuilder to create a  unit test suite factory- generateTestBuilder instantiates one (or more, for chunked tests)  instances of UnittestPackagedBuildFactoryThat's it.  I assumed, erroneously it seems, that the basic structuregiven in the prototype(https://bug516984.bugzilla.mozilla.org/attachment.cgi?id=433163) wasgood to build off of.  In light of the previous, this seems incorrect.So in order to add mozmill tests one of three things should be done:- a special switch could be added to generateTestBuilder just for  MozmillFactory [LEAST AMOUNT OF TIME]- MozmillFactory could be nixed entirely and instead have all of that  logic live in MozmillStep (or to conform to naming conventions,  MozillaPackagedMozmillTests) [MODERATE AMOUNT OF TIME]- if UnittestPackagedBuildFactory.addRunTestSteps serves (only) as a  switch on test type (that is to say, there is not a case where one  is running multiple types of tests with one factory) then the three  existing suites invoked there (mochitest, xpcshell tests, reftests)  could be each moved to their own subclass with arguments appropriate  to their class[*] and an abstract factory (function) could be added  to misc.py to instantiate the appropriate factory in  generateTestBuilder [MOST AMOUNT OF TIME]Other restructurings of the buildbot code to use an abstract factoryat various places to invoke certain types of tests are also possible.We should decide which of these methods to pursue considering thefactors of time to rollout as well as the path forward forbuildbotcustom and also the configs.Additionally, I am not sure what additional work is necessary to addthe invokation of the mozmill tests in buildbot-configs.  Is it assimple as adding another line to UNITTEST_SUITES in (one or more)config.py file(s)?--[*] currently, a superset of the needed arguments is passed into theUnittestPackagedBuildFactory constructor so that each test suite canhave its desired arguments.  If each suite became its own class, thenone will pass only those arguments appropriate to a certain test suite--[*] currently, a superset of the needed arguments is passed into theUnittestPackagedBuildFactory constructor so that each test suite canhave its desired arguments.  If each suite became its own class, thenone will pass only those arguments appropriate to a certain test suite

Comment 2

[cmtalbert]

Assuming this was never brought up in reviews (which if it had been, we wouldn't have continued going down the specialized factory path), I'm wondering why we are suddenly realizing that the entire structure is incorrect. 

We have landed everything necessary for mozmill to work like any other packaged unit test.  Does it need its own factory?  Does it need to be a set of Mozmill buildbot steps like the other packaged unit tests?

If Buildbot steps are the proper way to implement the new test framework then we should do that with all available haste.  What does that mean in terms of other buildbot support code?  Does that affect these branchObject classes?  I would think not, and therefore that might actually be the quickest way forward.

Comment 3

[Jeff Hammel]

It was decided to pursue putting the steps from MozmillFactory into UnittestPackagedBuildFactory

Comment 4

[Jeff Hammel]

(In reply to comment #3)
> It was decided to pursue putting the steps from MozmillFactory into
> UnittestPackagedBuildFactory

...so, do we need this bug anymore?

Comment 5

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to comment #4)
> (In reply to comment #3)
> > It was decided to pursue putting the steps from MozmillFactory into
> > UnittestPackagedBuildFactory
> 
> ...so, do we need this bug anymore?

nope