Open Bug 640464 Opened 9 years ago Updated 8 years ago
Develop a way for tests in mozilla-central to be overridden in a suite build
We are constantly failing tests from mozilla-central, and in an effort to get our tree green I want to at least remove one of the bottlenecks with doing so. Waiting on reviews/approvals for Firefox (m-c) based tests. What I propose, is the following plan: create a |c-c/suite/test-overrides| directory. let our suite/ base makefile recurse into there (during the _end_ of tier_app) if we are an ENABLE_TESTS build. inside, have a Makefile that then does the logic for test installing. I propose using "simple" rules in the Makefile.in for the test-list, and let our tests simply overwrite the m-c based tests, making sure this all gets run before package-tests. Any test added here, _NEEDS_ to have a warning (todo is fine) added to said test that we are overriding it, and a BUG on file to track re-enabling the m-c based test. we should also have a configure arg to disable the test overriding (so we can, in theory setup a periodic builder to re-enable the m-c tests once and a while). The idea would be that we do not _WANT_ any tests lingering in our overrides, and will still ALWAYS want to get reviews and use the m-c copies. The benefits: keeps the logic for this out of buildbot and instead in the real build system, where it will also help local devs who choose to run tests. Once we are green staying that way should be easier! The Cons: Overwriting m-c tests is a PITA and a cause of divergence, but given the struggle we have had with getting [and keeping] our tree green, it may be our best shot. KAIRO: requesting status2.1 for a rough sign-off on this plan. Serge: Adding you here, as I think you would be a good candidate to help manage this plan/effort, and I will likely request review from you for some of the initial work here.
I have been wishing (mostly silently) for something like this for so long ;-> A few (future) ideas, for what they are worth: Depending on how it is implemented, part of the logic might be shared in m-c. I assume this could be used to override "MailNews" tests as well: I'm obviously thinking about xpcshell tests here, rather than mochitest-*/reftest-*. I wonder whether this could be extended to be usable to package(+override) Firefox (and maybe Thunderbird) tests we currently duplicate [sometimes because app APIs differ anyway] (or just (knowingly) miss to port). Maybe there could be a test template (for each kind of test suite) that the override logic could update with the test name (in the wanted warning) and copy+rename as an additional test, rather than modifying each test (to add the warning). (Though I'd prefer the next idea.) I wonder whether it could be about "patch" rather than "override", as it's done for some external libs in m-c. (So we could get enhancement for free or break+check when patch doesn't apply anymore.) [Else, I would be wishing for some kind of update warning like http://dev.seamonkey.at/?d=x&i=mozilla&m=c] Would that qualify as a GSoC project? ;->
Severity: normal → enhancement
OS: Windows XP → All
Hardware: x86 → All
Version: unspecified → Trunk
Serge, I'm against any auto-patch-during-build approach per se, as that calls for having awful conflicts in the builds. I have been thinking for quite some time that something in between manual porting of all code and keeping only diffs on our side and applying them to FF/TB files automatically would be good. This would not only be good for tests, but everything we do. That said, I don't have a good solution for this yet and it also is way larger than this bug, let's discuss that somewhere else and stick with the tight solution here for the moment. I'm in favor of doing that override solution with the clear rules Callek gives in comment #0, though.
(In reply to Justin Wood (:Callek) from comment #0) > The Cons: Overwriting m-c tests is a PITA and a cause of divergence, but > given the struggle we have had with getting [and keeping] our tree green, it > may be our best shot. (In reply to Robert Kaiser (:firstname.lastname@example.org) from comment #2) > Serge, I'm against any auto-patch-during-build approach per se, as that > calls for having awful conflicts in the builds. My idea of using diffs would likely include doing a checksum of the m-c test and reporting a warning/error when it changes (and/or patching fails). That way we would know (immediately) to check/port what was done in m-c. That would be a compromise between more work or missing changes.
You need to log in before you can comment on or make changes to this bug.