606512 - continuous integration for testing mozmill

Status: RESOLVED WONTFIX

(Testing Graveyard :: Mozmill, defect)

Description

[Jeff Hammel]

in order to ensure that mozmill is kept in working condition, a buildbot should be made to test mozmill.  This should support:

* testing when the master has changed
* testing when the release branch has changed
* testing nightly FF builds (and thunderbird, if we care about that)

This will tell us not only when we broke mozmill, but when upstream changes broke mozmill.

There should be a buildslave for each supported platform.  The list of these should be determined.

There should also be runs for 3.5 and 3.6 probably, at least while these are supported (I care less about these, but other people seem to).  

The mozautomation/mozmill github repository should be setup with a post-receive hook that will then (RESTfully) trigger a test run: http://help.github.com/post-receive-hooks/

On sendchanges, buildbot should
* run the appropriate binary and the tests from http://hg.mozilla.org/qa/mozmill-tests
* run tests internal to mozmill

Note that this is different from bug 516984.  The purpose of that bug is testing mozilla central.  The purpose of this bug is testing the mozmill test harness

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to comment #0)
> be made to test mozmill.  This should support:
> 
> * testing nightly FF builds (and thunderbird, if we care about that)

Do we really want that? With a good set of tests available we wouldn't have to do that. On the other side testing the latest trunk build wouldn't be that bad. But that should be everything which would be necessary here.

> There should be a buildslave for each supported platform.  The list of these
> should be determined.

Do we have any docs available which describe the work which has to be done here? I wonder if that's something we could use on our QA lab boxes to make testing stronger.

> There should also be runs for 3.5 and 3.6 probably, at least while these are
> supported (I care less about these, but other people seem to).  

I don't think that's necessary. 

> The mozautomation/mozmill github repository should be setup with a post-receive
> hook that will then (RESTfully) trigger a test run:
> http://help.github.com/post-receive-hooks/

That's a sweety! 

> On sendchanges, buildbot should
> * run the appropriate binary and the tests from
> http://hg.mozilla.org/qa/mozmill-tests
> * run tests internal to mozmill

Flip the ordering here. We should have more internal tests as external ones in the future to ensure the quality of Mozmill.

Comment 2

[Jeff Hammel]

(In reply to comment #1)
> (In reply to comment #0)
> > be made to test mozmill.  This should support:
> > 
> > * testing nightly FF builds (and thunderbird, if we care about that)
> 
> Do we really want that? With a good set of tests available we wouldn't have to
> do that. On the other side testing the latest trunk build wouldn't be that bad.
> But that should be everything which would be necessary here.

I'm not sure what is being objected to or proposed here.  The main goal is that we need to know when a change in Firefox (etc) causes (known good) Mozmill tests to fail.  I was thinking of a nightly run to solve this (e.g. run Mozmill + tests regardless of whether or not Mozmill is changed every day).  I don't really care about the details (nightly vs. hourly, etc) just as long as we have at least some daily way of knowing that Mozmill + tests are not broken.

> 
> > There should be a buildslave for each supported platform.  The list of these
> > should be determined.
> 
> Do we have any docs available which describe the work which has to be done
> here? I wonder if that's something we could use on our QA lab boxes to make
> testing stronger.

I don't have any docs, but I know most of the moving parts except slave maintainence (installing software, syncing with puppet, that sort of thing).  If this becomes a priority, then maybe :anodelman could help with that aspect.

> > There should also be runs for 3.5 and 3.6 probably, at least while these are
> > supported (I care less about these, but other people seem to).  
> 
> I don't think that's necessary. 

I think its necessary as long as we support these platforms.  If I make a change in mozmill and I am responsible for ensuring that it works on 3.5 + 3.6, then buildbot should tell me if I broke these.

> > The mozautomation/mozmill github repository should be setup with a post-receive
> > hook that will then (RESTfully) trigger a test run:
> > http://help.github.com/post-receive-hooks/
> 
> That's a sweety! 
> 
> > On sendchanges, buildbot should
> > * run the appropriate binary and the tests from
> > http://hg.mozilla.org/qa/mozmill-tests
> > * run tests internal to mozmill
> 
> Flip the ordering here. We should have more internal tests as external ones in
> the future to ensure the quality of Mozmill.

Not sure what is meant here.  It doesn't matter which order the tests are run in.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to comment #2)
> I'm not sure what is being objected to or proposed here.  The main goal is that
> we need to know when a change in Firefox (etc) causes (known good) Mozmill
> tests to fail.  I was thinking of a nightly run to solve this (e.g. run Mozmill
> + tests regardless of whether or not Mozmill is changed every day).  I don't

Jeff, that's what we are doing already on our qa-horus box. I don't see a reason to duplicate the efforts.

> > Do we have any docs available which describe the work which has to be done
> > here? I wonder if that's something we could use on our QA lab boxes to make
> > testing stronger.
> 
> I don't have any docs, but I know most of the moving parts except slave
> maintainence (installing software, syncing with puppet, that sort of thing). 
> If this becomes a priority, then maybe :anodelman could help with that aspect.

Alice, I would still be interested to know if we have docs which describe the setup process.

> I think its necessary as long as we support these platforms.  If I make a
> change in mozmill and I am responsible for ensuring that it works on 3.5 + 3.6,
> then buildbot should tell me if I broke these.

As long as those are test-runs for new check-ins it's fine. But we do not need another daily test-run.

Comment 4

[Jeff Hammel]

(In reply to comment #3)
> (In reply to comment #2)
> > I'm not sure what is being objected to or proposed here.  The main goal is that
> > we need to know when a change in Firefox (etc) causes (known good) Mozmill
> > tests to fail.  I was thinking of a nightly run to solve this (e.g. run Mozmill
> > + tests regardless of whether or not Mozmill is changed every day).  I don't
> 
> Jeff, that's what we are doing already on our qa-horus box. I don't see a
> reason to duplicate the efforts.

We do this with latest versions of mozmill + release branches?  It was my understanding that mozmill + the tests were only occassionally updated there.
 
> > > Do we have any docs available which describe the work which has to be done
> > > here? I wonder if that's something we could use on our QA lab boxes to make
> > > testing stronger.
> > 
> > I don't have any docs, but I know most of the moving parts except slave
> > maintainence (installing software, syncing with puppet, that sort of thing). 
> > If this becomes a priority, then maybe :anodelman could help with that aspect.
> 
> Alice, I would still be interested to know if we have docs which describe the
> setup process.
> 
> > I think its necessary as long as we support these platforms.  If I make a
> > change in mozmill and I am responsible for ensuring that it works on 3.5 + 3.6,
> > then buildbot should tell me if I broke these.
> 
> As long as those are test-runs for new check-ins it's fine. But we do not need
> another daily test-run.

Not, not another daily test run.  Just runs for each checkin.  The daily run would be for nightly/hourly/whatever I think

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to comment #4)
> > Jeff, that's what we are doing already on our qa-horus box. I don't see a
> > reason to duplicate the efforts.
> 
> We do this with latest versions of mozmill + release branches?  It was my
> understanding that mozmill + the tests were only occassionally updated there.

That's why we have our dashboard here:
http://mozmill-release.brasstacks.mozilla.com/

All those tests are run with the latest stable version of Mozmill and the tests in our repository.

> Not, not another daily test run.  Just runs for each checkin.  The daily run
> would be for nightly/hourly/whatever I think

Something I forgot to say before, please keep in mind that our tests will not be compatible to the latest dev version of Mozmill. So I major changes happen you cannot use them.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Mozmill will reach its end of life soon. We are currently working on getting all the tests for Firefox ported to Marionette. For status updates please see bug 1080766.