Closed Bug 368573 Opened 14 years ago Closed 6 years ago
Need a way to run tests (mochikit/reftest) that display as failed when assertion failures are encountered
Assertions in xpcshell tests are fatal; assertions in mochitest aren't. There isn't a way to run a GUI test which should fail if an assertion fails, but since we treat assertion failures as bugs there should be. Making mochitests fatal with assertions probably won't happen (yet -- I hold out hope for the future!), so this probably requires a new GUI test harness in which assertions *are* fatal. Alternately, maybe mochitest can be tweaked to be run with assertions as fatal for specific tests (probably requires invoking it twice), or maybe the proper GUI can be opened from xpcshell with enough pain (most of which would be in a head_*.js file).
How about making a list of assertions that the mochitests are known to hit, and considering it a failure if any other assertion is hit?
You could do that, but I suspect any such list would likely accumulate unnecessary entries over time as assertions are added/removed from the code being tested. Is it really a good idea to make core code changes possibly also necessitate changes to the list of assertions? I don't think it is.
From bug 370767 comment 0, by Nickolay: Assertions are bad and there was talk about treating them the same way as crashes. Right now it's a bit unclear what happens if something asserts. From what I could figure out on IRC, they are ignored now. In any case, it would be nice if a) they were not ignored b) just a particular test was marked as failed and the testsuite continued to run, since most assertions are not really -that- fatal. This applies to all of our test harnesses ideally, in particular mochitest and reftest. The relevant snippet of #developers log: <asqueella> sayrer: is the idea that mochitests fail on assertion failures? <asqueella> if not, how does one go about testing lack of assertion in certain cases? <sayrer> asqueella: there's nothing set up for that <asqueella> it's a pity <asqueella> so it ignores assertions? <sayrer> it doesn't operate at that level <sayrer> tinderbox could watch for assertions if we wanted <asqueella> well <asqueella> yeah <asqueella> that's what I'm talkign about I guess <sayrer> talk to robcee then :) <robcee> sayrer: I just run it from the command line on the test box and ran through perfectly <shaver> it wouldn't be too hard to let mochitests detect assertions <shaver> add a static int assertCount; <shaver> have NS_ASSERT increment it <shaver> add two little functions to a service or whatever to clear and sample the variable <shaver> and robert is your mother's brother <sayrer> patches welcome <sayrer> asserts are the least of worries, really <asqueella> why? <sayrer> since the trunk is so horked <asqueella> they're supposed to be fatal <asqueella> the state of trunk shouldn't affect the test harnesses :) <sayrer> I'm not big on test harnesses that always fail <sayrer> ymmv
Summary: Need a way to run tests in which assertions are fatal which need a GUI → Need a way to run tests (mochikit/reftest) that display as failed when assertion failures are encountered
Are mochitests even run on debug builds? That would be a first step to getting useful assertions...
I don't think so, which is actually rather bad: xpcshell tests explicitly fail if they trip assertions (but of course only in debug builds).
we have bug 372581 opened requesting debug builds. I think this is important and we should get some debug builds setup and running.
Everything left in Core:Testing is going to Testing:General. Filter on CleanOutCoreTesting to ignore.
Component: Testing → General
Product: Core → Testing
QA Contact: testing → general
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #9) > Dupe of bug 472557 or bug 404077 (or the reverse?) Yes.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 404077
You need to log in before you can comment on or make changes to this bug.