Closed
Bug 368573
Opened 18 years ago
Closed 10 years ago
Need a way to run tests (mochikit/reftest) that display as failed when assertion failures are encountered
Categories
(Testing :: General, defect)
Testing
General
Tracking
(Not tracked)
RESOLVED
DUPLICATE
of bug 404077
People
(Reporter: Waldo, Unassigned)
References
Details
Assertions in xpcshell tests are fatal; assertions in mochitest aren't. There isn't a way to run a GUI test which should fail if an assertion fails, but since we treat assertion failures as bugs there should be.
Making mochitests fatal with assertions probably won't happen (yet -- I hold out hope for the future!), so this probably requires a new GUI test harness in which assertions *are* fatal. Alternately, maybe mochitest can be tweaked to be run with assertions as fatal for specific tests (probably requires invoking it twice), or maybe the proper GUI can be opened from xpcshell with enough pain (most of which would be in a head_*.js file).
Comment 1•18 years ago
|
||
How about making a list of assertions that the mochitests are known to hit, and considering it a failure if any other assertion is hit?
Reporter | ||
Comment 2•18 years ago
|
||
You could do that, but I suspect any such list would likely accumulate unnecessary entries over time as assertions are added/removed from the code being tested. Is it really a good idea to make core code changes possibly also necessitate changes to the list of assertions? I don't think it is.
Comment 3•18 years ago
|
||
From bug 370767 comment 0, by Nickolay:
Assertions are bad and there was talk about treating them the same way as
crashes. Right now it's a bit unclear what happens if something asserts. From
what I could figure out on IRC, they are ignored now.
In any case, it would be nice if
a) they were not ignored
b) just a particular test was marked as failed and the testsuite continued to
run, since most assertions are not really -that- fatal.
This applies to all of our test harnesses ideally, in particular mochitest and
reftest.
The relevant snippet of #developers log:
<asqueella> sayrer: is the idea that mochitests fail on assertion failures?
<asqueella> if not, how does one go about testing lack of assertion in certain
cases?
<sayrer> asqueella: there's nothing set up for that
<asqueella> it's a pity
<asqueella> so it ignores assertions?
<sayrer> it doesn't operate at that level
<sayrer> tinderbox could watch for assertions if we wanted
<asqueella> well
<asqueella> yeah
<asqueella> that's what I'm talkign about I guess
<sayrer> talk to robcee then :)
<robcee> sayrer: I just run it from the command line on the test box and ran
through perfectly
<shaver> it wouldn't be too hard to let mochitests detect assertions
<shaver> add a static int assertCount;
<shaver> have NS_ASSERT increment it
<shaver> add two little functions to a service or whatever to clear and sample
the variable
<shaver> and robert is your mother's brother
<sayrer> patches welcome
<sayrer> asserts are the least of worries, really
<asqueella> why?
<sayrer> since the trunk is so horked
<asqueella> they're supposed to be fatal
<asqueella> the state of trunk shouldn't affect the test harnesses :)
<sayrer> I'm not big on test harnesses that always fail
<sayrer> ymmv
Blocks: 379405
Summary: Need a way to run tests in which assertions are fatal which need a GUI → Need a way to run tests (mochikit/reftest) that display as failed when assertion failures are encountered
Comment 5•18 years ago
|
||
Are mochitests even run on debug builds? That would be a first step to getting useful assertions...
Reporter | ||
Comment 6•18 years ago
|
||
I don't think so, which is actually rather bad: xpcshell tests explicitly fail if they trip assertions (but of course only in debug builds).
Comment 7•17 years ago
|
||
we have bug 372581 opened requesting debug builds. I think this is important and we should get some debug builds setup and running.
Comment 8•16 years ago
|
||
Everything left in Core:Testing is going to Testing:General. Filter on CleanOutCoreTesting to ignore.
Component: Testing → General
Product: Core → Testing
QA Contact: testing → general
Comment 9•16 years ago
|
||
Dupe of bug 472557 or bug 404077 (or the reverse?)
Comment 10•10 years ago
|
||
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #9)
> Dupe of bug 472557 or bug 404077 (or the reverse?)
Yes.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → DUPLICATE
You need to log in
before you can comment on or make changes to this bug.
Description
•