Closed Bug 369809 Opened 18 years ago Closed 17 years ago

need nightly builds with tests enabled

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: ray, Unassigned)

Details

We have a community of users who are willing to download nightly builds and do things with them. We should allow them to download debug builds with tests, in addition to regular builds. If they are willing to run tests, we should help them do so. FYI, right now the presence of tests is tied to debug-enabled. I am asking for tests to be available. The ability to debug is different than the ability to test. If debug-enabled and testability are kept together, I am requesting that we put out nightly builds configured for this. If there is some problem with releasing debug builds, I can make a separate argument for testability.
I'm not sure what you mean when you say "the presence of tests is tied to debug-enabled". Assuming "tests available" means "configured with --enable-tests", there's no requirement that I know of that test enabled builds must also be debug builds. That being said, ignoring the debug issue, I'm not sure what we'd really gain from distributing --enable-tests builds (at least not at this point). Which tests would nightly users actually be able to run? From what I know about --enable-tests builds (which admittedly isn't very much), the main difference is that a bunch of test programs are put into dist/bin, most of which are old and unmaintained. Having mochitests distributed with nightly builds could definitely be useful, but I don't think that we currently have any infrastructure set up to do that for packaged builds, and currently would require nightly build testers to have Perl installed, as far as I know. Is that what this bug is about?
This mozconfig gets me tests: mk_add_options MOZ_CO_PROJECT=browser ac_add_options --enable-application=browser ac_add_options --enable-debug ac_add_options --enable-accessibility=yes Perhaps the --enable-tests is a side-effect of --enable-debug. I will clarify that for myself. Gavin - Yes, this bug is about enabling people who download nightly tests to execute some tests. It will never be all the tests we have, but it could be a significant subset. I am used to a Mac. I am not used to a requirement like "must have perl" being a difficult thing to accomplish.... Gavin - You are right that the tests are mostly those binaries in dist/bin and they are not very useful right now. But I am doing a survey of tests and test tools to identify which are useful and whether ones that do not work now should be fixed. So, the situation may improve. Setting up nightly builds to use --enable-tests is not the only step in this process, or the first that must be done, but it is a task that is doable and independent from other tasks.
> This mozconfig gets me tests: What do you mean by "gets me tests"? In any case, --enable-tests is the default for browser builds. You have to explicitly specify --disable-tests (which we do in our release/production builds, as I recall) to not get things like mochitest built. I certainly have several trees that are --disable-debug, --enable-optimize, and have tests built and all. It's not tied to --enable-debug in any way.
I am sorry that I muddied the water by mistakenly linking the debug build issues with the test build issue. If I could excise all mention of debug from the previous comments, I would. So, to restate this in the terms available from this discussion: I believe that we should ship nightly builds that are built with --disable-tests, and we should also ship nightly builds that are built without --disable-tests. To state it concisely, having a subset of tests available to community members will be "a good thing".
Ray, the --enable-tests flag doesn't have any affect on what gets *shipped*, only what gets *built*. It will build a bunch of test programs, but those won't go anywhere beyond dist/bin.
I will have to clarify this with dbaron. I asked him and this is how I understood his response. If one uses the "-reftest" command-line flag with a released build, or a nightly, it has no effect. I had tried this. It has no effect. I enable both debug and tests and build the browser. When I build this, I can run reftests with that browser. We need to have a downloadable build that can run tests, that has the executables in dist/bin, that is enabled for running reftest. Whatever the things that are needed for this to be possible, that is what I am asking for.
(In reply to comment #6) > I will have to clarify this with dbaron. I asked him and this is how I > understood his response. If one uses the "-reftest" command-line flag with a > released build, or a nightly, it has no effect. I had tried this. It has no > effect. > > I enable both debug and tests and build the browser. When I build this, I can > run reftests with that browser. > > We need to have a downloadable build that can run tests, that has the > executables in dist/bin, that is enabled for running reftest. > > Whatever the things that are needed for this to be possible, that is what I am > asking for. Sounds like what you want is for us to publish debug builds, plus the entire contents of dist/bin ? For Firefox we only do one debug build that I know of, and it's purposely not published, which is fxdbug-linux tbox on the Firefox tinderbox tree. If we did a special packaging step (e.g. just zipped up dist/bin) and published that (e.g. to the experimental section of FTP) then I think that's would give you what you want. dbaron/bsmedberg/bz, is the above true and if so is there any reason for us not to publish this build? I think we can assume that Ray (or anyone else who wanted to run their own tests with a debug build) would be able to provide the reftest or any other test that they needed. Ray, I assume this is so you don't have to do your own custom build in order to work on tests, right? Also, I know robcee is producing these kinds of builds for the buildbot-driven reftests (cc'd) so maybe he has some input.
Does just --enable-tests not enable reftest? I didn't think reftest depended on --enable-debug. Did I miss something? I do think shipping some of the things built or stuck in dist/bin with --enable-tests with some of our nightlies might not be a bad idea. We used to do _some_ stuff like that, in fact (see QA menu in old suite builds), debug preferences in same, etc.
(In reply to comment #8) > Does just --enable-tests not enable reftest? I didn't think reftest depended > on --enable-debug. Did I miss something? Probably me misunderstanding the requirements :) > I do think shipping some of the things built or stuck in dist/bin with > --enable-tests with some of our nightlies might not be a bad idea. We used to > do _some_ stuff like that, in fact (see QA menu in old suite builds), debug > preferences in same, etc. Right. I guess I am confused what tests these are for, if it's *just* reftests then --enable-tests is ok. If the goal is to have a build or set of builds that can run all of our tests (e.g. leaktests and/or reftests) then I guess we'd want both debug and --enable-tests. There are a number of configure flags that enable things that we wouldn't put in the shipping product, but are useful for certain kinds of tests (trace-malloc and jprof come to mind), so we assume that anyone who wants to run these is compiling from source. Ray, I think I'd be less confused if I understood the scope of what you are asking. I was assuming from the summary and your first comment that what you would like to see is a set of binaries published by mozilla.org which could be used for any kind of test; that's pretty different from "the minimum change from a release build required for reftest", or "for leaktest", etc. which is what the discussion in this bug seems to be about. For a number of tests a custom build is required, and exactly how that build is done depends on the test.
This is the first paragraph from this bug: We have a community of users who are willing to download nightly builds and do things with them. We should allow them to download debug builds with tests, in addition to regular builds. If they are willing to run tests, we should help them do so. This request was a bit abstract. I was asked exactly what I was trying to do. I mentioned reftest and so on and so forth. Maybe I should have just repeated that first paragraph again. The goal is simple. We want to make it easy for community members to run tests. That is all I am asking for. From what I have seen, the easiest way to make it possible to allow community members to run tests would be to enable debug and testing, as you point out above. If there is a desire to clarify why enabling debug relates to enabling tests, can someone create a bug asking for that to be clarified?
Ray, enabling debug if it's not needed just makes it harder for people to download and run the builds, for the simple reason that debug builds are huge. My Linux debug Firefox dist/bin over here is about 640MB, for example. Just the layout library is 280MB. Stripping the debug symbols helps some; if we strip all those layout might be down to 15-20MB on Linux, with the entire Firefox probably around 60-100MB. I think we'll have much better success getting people to download these builds if they're about nightly-sized. If we can't do that, we can't do that, of course.
Can we experiment with producing these and see how big they are and what can be done to make them smaller? We do not have to advertise them until we agree that they are reasonably configured.
There's another issue with shipping a reftest-enabled build: We don't necessarily want people using it as their default browser. It might require extra work to repackage it and provide suitable documentation on how to use it. While I'm not disputing that it might be nice to have more tests included from the community, lowering the barrier to entry might not give us the best results. Is the extra effort required going to be worth the limited payoff?
What's wrong with people who use trunk nightlies right now using reftest-enabled builds? Put another way, is there something wrong with enabling tests in the default trunk nightlies we ship?
I'm not sure what the implications are, but I don't think we want to do this for our default nightly builds, which should be as-close-to-final-production-binary as we can make them. Another option: poll newsgroups, mozillazine and blogs to try and gauge level of interest before making this change?
OS: Mac OS X → All
Hardware: PC → All
I am not asking that we do not do the regular nightly builds. We should do the test-enabled build also. It sounds as though I need to clarify exactly what one gets from the '--enable-tests' only, vs the '--enable-tests --enable-debug' option. Hm. Consensus. It sounds good, but it sure takes a while. :-)
Reftest could be packaged as an extension too. Perhaps included in default nightly installer but not installed by default.
Ray, what good is a shipped build that can run reftest? Are you talking about a build that has the ability to run reftests (the infrastructure), or something that actually ships all of the reftests in the tree? I think we should consider turning the reftest infrastructure on by default, if it's small and doesn't pose security issues.
I would assume people want the official released Firefox binary to be as small as possible. I was not suggesting we put test code in everything that goes out. People seem to be getting hung up on issues with individual tests or individual test harnesses. Would it be easier to deal with if I filed separate bugs on each test harness and group of tests requesting that a nightly build be produced which contains that harness or that group? Then, the requests could be dealt with individually. To answer the question above, what I am asking for is that any user should be able to run a meaningful subset of the tests we have. That is all. If there are issues with any particular test, that would be a problem with that particular test. The main idea is that any user should be able to run a meaningful subset of the tests we have. So, I am asking that the build group build the regular nightly build and that they also build images that are configured, however they need to be configured, to run a reasonable subset of the available tests without requiring the user to build their own copy of the browser.
Ray, what do you think the value is of users who can't do their own builds being able to perform these tests? I tend to think that the value/cost proposition here is really low. So, I'd like to argue that it's not worthwhile for any user to be able to run our in-tree tests meaningfully, and this bug should be WONTFIX. Please argue with me ;-) I'd like to know what testing community you think could be involved, and what kinds of tests you think they'd perform. Would they repeat the automated tests already performed by the tinderbox (if so, why?). Would they do tests that are not automatable (don't we already do that with litmus and test days?).
First the nightly testers can detect failures in our test suites on certain configurations that we don't have up on Tinderbox. Second, and I think it's the most important, if people can easily run these tests (and since it's relatively easy to write the tests), we can hope to get more people involved in writing the tests. For example they could help with converting bugzilla testcases on older bugs to automated tests.
Nickolay, Ray: I get what you're saying, but I think the crux of the matter is what Ben is saying, if we lower the barrier to the point of non-existence, are the users really going to be contributing quality test-cases? How many users who aren't comfortable building their own application are comfortable enough with bugzilla and HTML/CSS/DOM to write a quality test case? How are they going to submit them to us? As I suggested in IRC yesterday, this sounds like an interesting case for a community-led project. There's nothing stopping anyone from building and releasing a test-enabled build to the world. We see it all the time with processor-specific optimized builds. I'm done. :)
Or, if we're focused on reftest, making the reftest harness an extension, and making it available on AMO (this sounds very easy to do).
Wow, what a lot of crossed wires :-) I believe Ray's logic is as follows: - The only skills needed to write reftests are an ability to read bugs, and an ability to write HTML and CSS - There are many people who can do this who are not capable of building Firefox (for example, web developers or people on Windows who can't afford MSVC) - If there is a way that nightly builds can run reftests, those people can use them to create new reftests when before they could not participate - This could seriously expand the pool of reftest-creators - This is a good thing, because there are 7000 bugs needing tests written - Therefore, we should find a way to make the nightly builds run reftests This can be achieved either by making the reftest-running code built all the time (even in released builds), or by turning it on only for nightlies (arguably bad if it makes them less like release builds, but if they are debug builds anyway, it's not a much bigger difference) or, if technically feasible, packaging the necessary extra bits of browser as an extension. Any of these will do. It doesn't matter too much which. Am I right? Gerv
You don't need to buy MSVC and haven't for quite some time, people need to stop throwing that out. Gerv, Ray, etc.: as Rob said in comment 22, you are free to provide these builds and demonstrate the value of them. You don't need special permission from anyone to make or distribute those builds, and you shouldn't let yourself be blocked from doing what you clearly believe has enormous value. Show us we're wrong!
I don't understand why this is so complicated -- isn't all that's needed a simple extension that will render two snippets of markup and compare them? I don't think we need to make the entire "reftest framework" run on nightly builds, though that certainly can't hurt. The people who are web developers aren't going to want to run firefox.exe -P blah -reftest etc. They're going to want to select Tools -> Reference Test, and paste/edit their stuff there (ideally with live preview). Once they have a good set of snippets they can submit them as bugs, or attach them to existing bugs. None of that needs --enable-tests or debug symbols or any of that stuff to be shipped.
Vlad is probably right here. What he is describing may not be very difficult. Coming into MoCo without previous experience, I thought that having another kind of build put out as a nightly would not be very difficult. I was obviously very mistaken. I am going to try to make sure that the proper tools are available to people to enable them to test as soon as they show any enthusiasm at all, with as little hassle as possible and as few hoops to jump through as possible. If anyone wants to suggest how I can base deal with this bug, I am open to suggestions.
Marking WONTFIX; that shouldn't imply that the community can't/shouldn't produce builds with reftests enabled or, as Vlad points out, an extension to help with this. If there's interest, feel free to reopen and assign to yourself.
Status: NEW → RESOLVED
Closed: 17 years ago
Resolution: --- → WONTFIX
Changing from RESOLVED WONTFIX to be a dupe of the new bug.
Resolution: WONTFIX → DUPLICATE
Hey, that bug isn't WONTFIX yet ;)
I don't see how this bug is a duplicate of that one. I thought this bug was requesting that we ship a "debug" or "test enabled" build, while that one is requesting that we run tests on a debug build, on the tinderbox.
Point taken. I spoke too soon.
Resolution: DUPLICATE → WONTFIX
Assignee: build → nobody
QA Contact: mozpreed → build
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.