== Problem == Historically what has happened is we run into a problem on test harness X. We realize that the problem would be best solved within mozbase. We would then fix the problem, test that it works and commit the working solution to mozbase. This was good when mozbase was still under heavy development and nothing really depended on it yet. Now we have more and more harnesses that are using it: mozmill, peptest, eideticker, marionette, xpcshell, talos and soon mochitest and reftest. We have tools: mozregression, mozcommitbuilder, others. We have teams outside of our control using it: firebug, jetpack, thunderbird, others. We even have contributors stepping in to contribute patches for pain points in their own personal projects. Jeff has done some work trying to document it all here: http://k0s.org/mozilla/mozbase/dependencies.html This is really good, it means mozbase is useful. But it also means that we need to be very careful. Last weekend two regressions were committed to mozbase which broke peptest. It wasn't a huge deal since peptest was hidden anyway, but it will be worse in the future. == Possible solutions == At the very least I think we should start committing changes to m-i so we get backing out as a last line of defense. We should also push to try and run the mozbase tests too. TBH, I don't think the mozbase tests on their own will be enough. Every harness and tool will be using mozbase slightly differently, and it will be hard to predict what effect certain changes will have on each one. But I think we should set up some sort of continuous integration for changes going into the github repo as well. I know everyone is busy and this probably won't have a very high priority.. but I think it's something we should keep on the back burner.
First, let me echo the sentiment of this bug. As mozbase grows and our test harnesses become more integrated, our test harnesses will need tests and continuous integration. > Last weekend two regressions were committed to mozbase which broke > peptest. It wasn't a huge deal since peptest was hidden anyway, but it > will be worse in the future. So part of the problem was that peptest was hidden. The mozbase changes were pushed to try, but since peptest was hidden, I didn't see the results. Yes, I should have used &noignore=1. But I didn't. My mistake. I believe peptest is now unhidden: https://bugzilla.mozilla.org/show_bug.cgi?id=774817#c6 > At the very least I think we should start committing changes to m-i I fully support this. > But I think we should set up some sort of continuous integration for > changes going into the github repo as well. We do have continuous integration for mozmill tests of some sort: http://k0s.org:8010/waterfall . They are currently failing from ctalbert's and ahal's changes, but other than that have been green since May 8. So there are two major problems with CI at k0s.org:8010 , one technical, one conceptual: 1. k0s.org:8010 does not send out notifications. Talking to Clint -- well, at least half a year ago -- the notifications most desired were to post to autolog. This being an unofficial project I had very little time to work on this nor did anyone else, and I have not been able to get this to function in the small amounts of time I did devote to this. Talking to Clint yesterday, we've decided to add email notification to the individual breaker and/or to the tools mailing list. I intend to implement this next week. It is not ideal, but it should at leat help for the short-term. 2. The tests in the mozbase repo are only for mozbase. So if you make an API change or introduce a legitimate bug that breaks upstream harnesses (etc), you will not detect this just through the mozbase tests. My solution to this is to have continuous integration not only for mozbase, but also for the projects that depend on it. Talos has tests that are currently not run. Other harnesses could also have at least smoke tests to ensure that they're not completely broken. There are other big problems with k0s.org:8010 as well: - I don't have X or Firefox installed, nor do I intend to, as while the hardware is perfectly adequate for a small webserver, having these running would completely kill performance. Currently no mozbase tests require Firefox, nor in fact do we have a good way of having tests that run Firefox currently. I have some ideas here, but this is an issue of its own right. - k0s.org is of course one platform: linux. If we want more platforms, we'll need some real slaves. If anyone wants to play with the code, it lives at http://k0s.org/mozilla/hg/autobot . I am more than happy to move it to hg.m.o/automation if there is sufficient interest. Another thing that could conceivably help is mirroring only packages as we release them. Our current mirroring policy is to mirror over the entirity: https://wiki.mozilla.org/Auto-tools/HowTo/MirrorRepo You release a package (or packages) to pypi and then mirror the whole repo. This has the unfortunate side-effect of getting changes to packages other than the ones that are mirrored. I would be open to changing this, but this isn't entirely a trivial or risk-free change either. So this touches on several issues I've been intending to bring up with Will and other interested parties in the next few weeks. I'm happy to discuss these issues in more detail here, on the tools mailing list, or in #ateam.
BTW, k0s.org:8010 now sends out notifications. They're theoretically sent to the tools list...not sure if the one failure was received there or not. They are also sent to people that caused problems with the build. However, k0s.org's postfix is configured very minimally and so may be flagged as spam by mail relayers. It does send correctly to my @mozilla.com email address. It does not send to my gmail address. I can put some time into fixing these, albeit configuring postfix isn't one of my favorite things to do or test in the world, but only if we feel that this is worthwhile. A bigger gain would be to run autobot + mozbase tests (+ whatever else) on mozilla hardware with a real email server configured and managed by SAs that actually know what they're doing :) Testing mozbase has and continues to be a low priority. Having a robust solution that will test not only mozbase but its dependencies, while absolutely critical if we want reliable software, will require actual time being thrown at it and not just free hours here and there.
Not sure if that has been discussed somewhere else yet, but what are the requirements for a CI solution in terms of platform coverage, dependencies (e.g. Python versions), supported applications under test? I'm sure it's a complicated matrix we should probably put together first to get a better overview what we really need here.
Ideally, we would cover all supported platforms and all versions of python. Realistically, I would want one version of each major OS (win 7, OS 10.7 [presumedly], and linux preferably ubuntu recent). Two python versions would be nice (2.5 and 2.7) but not required, since we're gunning towards 2.7 anyway. The system would need to have the ability to run firefox. The interesting part of the matrix would be supporting the frameworks in http://k0s.org/mozilla/mozbase/dependencies.html . So this would be 7 or so builders, in buildbot terms. This would be a lot to setup. So the real limiting factors here are: - time: someone will actually need to do this - boxes Practically speaking we currently have one builder on one platform testing mozbase only. So anything better than this would be an improvement. I am open to adding more frameworks under test here (talos, etc) but I don't want to add X or Firefox as the box is not that powerful
At least in terms of building all the various projects, mozharness will go a long way to making that easier. In many cases the mozharness scripts can directly be re-used and if it doesn't exist, it would be fairly simple to write it.
:ahal, not sure what the status is here. Is bug 790765 enough to fix? If not, could you spell out where this bug is?
(In reply to Jeff Hammel [:jhammel] from comment #6) > :ahal, not sure what the status is here. Is bug 790765 enough to fix? If > not, could you spell out where this bug is? I think bug 790765 is pretty much what I was looking for out of this bug, I'll resolve duplicate.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 790765
You need to log in before you can comment on or make changes to this bug.