778140 - Mozbase needs C-I or at least more stringent commit policies

Status: RESOLVED DUPLICATE of bug 790765

(Testing :: Mozbase, defect)

Description

[Andrew Halberstadt [:ahal]]

== Problem ==

Historically what has happened is we run into a problem on test harness X. We realize that the problem would be best solved within mozbase. We would then fix the problem, test that it works and commit the working solution to mozbase. This was good when mozbase was still under heavy development and nothing really depended on it yet.

Now we have more and more harnesses that are using it: mozmill, peptest, eideticker, marionette, xpcshell, talos and soon mochitest and reftest. We have tools: mozregression, mozcommitbuilder, others. We have teams outside of our control using it: firebug, jetpack, thunderbird, others. We even have contributors stepping in to contribute patches for pain points in their own personal projects. Jeff has done some work trying to document it all here: http://k0s.org/mozilla/mozbase/dependencies.html

This is really good, it means mozbase is useful. But it also means that we need to be very careful. Last weekend two regressions were committed to mozbase which broke peptest. It wasn't a huge deal since peptest was hidden anyway, but it will be worse in the future.

== Possible solutions ==

At the very least I think we should start committing changes to m-i so we get backing out as a last line of defense. We should also push to try and run the mozbase tests too. TBH, I don't think the mozbase tests on their own will be enough. Every harness and tool will be using mozbase slightly differently, and it will be hard to predict what effect certain changes will have on each one.

But I think we should set up some sort of continuous integration for changes going into the github repo as well. I know everyone is busy and this probably won't have a very high priority.. but I think it's something we should keep on the back burner.

Comment 1

[Jeff Hammel]

First, let me echo the sentiment of this bug.  As mozbase grows and
our test harnesses become more integrated, our test harnesses will
need tests and continuous integration.

> Last weekend two regressions were committed to mozbase which broke
> peptest. It wasn't a huge deal since peptest was hidden anyway, but it
> will be worse in the future.

So part of the problem was that peptest was hidden.  The mozbase
changes were pushed to try, but since peptest was hidden, I didn't see
the results.  Yes, I should have used &noignore=1. But I didn't. My
mistake. I believe peptest is now unhidden:
https://bugzilla.mozilla.org/show_bug.cgi?id=774817#c6

> At the very least I think we should start committing changes to m-i

I fully support this.

> But I think we should set up some sort of continuous integration for
> changes going into the github repo as well.

We do have continuous integration for mozmill tests of some sort:
http://k0s.org:8010/waterfall . They are currently failing from
ctalbert's and ahal's changes, but other than that have been green
since May 8.

So there are two major problems with CI at k0s.org:8010 , one technical, one
conceptual:

1. k0s.org:8010 does not send out notifications.  Talking to Clint
-- well, at least half a year ago -- the notifications most desired
were to post to autolog.  This being an unofficial project I had very
little time to work on this nor did anyone else, and I have not been
able to get this to function in the small amounts of time I did devote
to this.  Talking to Clint yesterday, we've decided to add email
notification to the individual breaker and/or to the tools mailing
list. I intend to implement this next week.  It is not ideal, but it
should at leat help for the short-term.

2. The tests in the mozbase repo are only for mozbase.  So if you make
an API change or introduce a legitimate bug that breaks upstream
harnesses (etc), you will not detect this just through the mozbase
tests.

My solution to this is to have continuous integration not only for
mozbase, but also for the projects that depend on it.  Talos has tests
that are currently not run. Other harnesses could also have at least
smoke tests to ensure that they're not completely broken.

There are other big problems with k0s.org:8010 as well:

- I don't have X or Firefox installed, nor do I intend to, as while
  the hardware is perfectly adequate for a small webserver, having
  these running would completely kill performance.  Currently no
  mozbase tests require Firefox, nor in fact do we have a good way of
  having tests that run Firefox currently.  I have some ideas here,
  but this is an issue of its own right.

- k0s.org is of course one platform: linux. If we want more platforms,
  we'll need some real slaves.

If anyone wants to play with the code, it lives at
http://k0s.org/mozilla/hg/autobot . I am more than happy to move it to
hg.m.o/automation if there is sufficient interest.

Another thing that could conceivably help is mirroring only packages
as we release them.  Our current mirroring policy is to mirror over
the entirity: https://wiki.mozilla.org/Auto-tools/HowTo/MirrorRepo
You release a package (or packages) to pypi and then mirror the whole
repo. This has the unfortunate side-effect of getting changes to
packages other than the ones that are mirrored.  I would be open to
changing this, but this isn't entirely a trivial or risk-free change
either.

So this touches on several issues I've been intending to bring up with
Will and other interested parties in the next few weeks.  I'm happy to
discuss these issues in more detail here, on the tools mailing list,
or in #ateam.

Comment 2

[Jeff Hammel]

BTW, k0s.org:8010 now sends out notifications.  They're theoretically sent to the tools list...not sure if the one failure was received there or not. They are also sent to people that caused problems with the build.  However, k0s.org's postfix is configured very minimally and so may be flagged as spam by mail relayers. It does send correctly to my @mozilla.com email address.  It does not send to my gmail address.  I can put some time into fixing these, albeit configuring postfix isn't one of my favorite things to do or test in the world, but only if we feel that this is worthwhile.

A bigger gain would be to run autobot + mozbase tests (+ whatever else) on mozilla hardware with a real email server configured and managed by SAs that actually know what they're doing :) Testing mozbase has and continues to be a low priority.  Having a robust solution that will test not only mozbase but its dependencies, while absolutely critical if we want reliable software, will require actual time being thrown at it and not just free hours here and there.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Not sure if that has been discussed somewhere else yet, but what are the requirements for a CI solution in terms of platform coverage, dependencies (e.g. Python versions), supported applications under test? I'm sure it's a complicated matrix we should probably put together first to get a better overview what we really need here.

Comment 4

[Jeff Hammel]

Ideally, we would cover all supported platforms and all versions of python.

Realistically, I would want one version of each major OS (win 7, OS 10.7 [presumedly], and linux preferably ubuntu recent).  Two python versions would be nice (2.5 and 2.7) but not required, since we're gunning towards 2.7 anyway.  The system would need to have the ability to run firefox.

The interesting part of the matrix would be supporting the frameworks in http://k0s.org/mozilla/mozbase/dependencies.html .  So this would be 7 or so builders, in buildbot terms.

This would be a lot to setup.  So the real limiting factors here are:

- time: someone will actually need to do this
- boxes

Practically speaking we currently have one builder on one platform testing mozbase only. So anything better than this would be an improvement.  I am open to adding more frameworks under test here (talos, etc) but I don't want to add X or Firefox as the box is not that powerful

Comment 5

[Andrew Halberstadt [:ahal]]

At least in terms of building all the various projects, mozharness will go a long way to making that easier. In many cases the mozharness scripts can directly be re-used and if it doesn't exist, it would be fairly simple to write it.

Comment 6

[Jeff Hammel]

:ahal, not sure what the status is here.  Is bug 790765 enough to fix?  If not, could you spell out where this bug is?

Comment 7

[Andrew Halberstadt [:ahal]]

(In reply to Jeff Hammel [:jhammel] from comment #6)
> :ahal, not sure what the status is here.  Is bug 790765 enough to fix?  If
> not, could you spell out where this bug is?

I think bug 790765 is pretty much what I was looking for out of this bug, I'll resolve duplicate.