Closed Bug 368573 Opened 14 years ago Closed 6 years ago

Need a way to run tests (mochikit/reftest) that display as failed when assertion failures are encountered

Categories

(Testing :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 404077

People

(Reporter: Waldo, Unassigned)

References

Details

Assertions in xpcshell tests are fatal; assertions in mochitest aren't.  There isn't a way to run a GUI test which should fail if an assertion fails, but since we treat assertion failures as bugs there should be.

Making mochitests fatal with assertions probably won't happen (yet -- I hold out hope for the future!), so this probably requires a new GUI test harness in which assertions *are* fatal.  Alternately, maybe mochitest can be tweaked to be run with assertions as fatal for specific tests (probably requires invoking it twice), or maybe the proper GUI can be opened from xpcshell with enough pain (most of which would be in a head_*.js file).
Blocks: 367489
Blocks: 370255
Blocks: 372483
How about making a list of assertions that the mochitests are known to hit, and considering it a failure if any other assertion is hit?
You could do that, but I suspect any such list would likely accumulate unnecessary entries over time as assertions are added/removed from the code being tested.  Is it really a good idea to make core code changes possibly also necessitate changes to the list of assertions?  I don't think it is.
From bug 370767 comment 0, by Nickolay:

Assertions are bad and there was talk about treating them the same way as
crashes. Right now it's a bit unclear what happens if something asserts. From
what I could figure out on IRC, they are ignored now.

In any case, it would be nice if 
a) they were not ignored
b) just a particular test was marked as failed and the testsuite continued to
run, since most assertions are not really -that- fatal.

This applies to all of our test harnesses ideally, in particular mochitest and
reftest.

The relevant snippet of #developers log:

<asqueella> sayrer: is the idea that mochitests fail on assertion failures?
<asqueella> if not, how does one go about testing lack of assertion in certain
cases?
<sayrer> asqueella: there's nothing set up for that
<asqueella> it's a pity
<asqueella> so it ignores assertions?
<sayrer> it doesn't operate at that level
<sayrer> tinderbox could watch for assertions if we wanted
<asqueella> well
<asqueella> yeah
<asqueella> that's what I'm talkign about I guess
<sayrer> talk to robcee then :)
<robcee> sayrer: I just run it from the command line on the test box and ran
through perfectly
<shaver> it wouldn't be too hard to let mochitests detect assertions
<shaver> add a static int assertCount;
<shaver> have NS_ASSERT increment it
<shaver> add two little functions to a service or whatever to clear and sample
the variable
<shaver> and robert is your mother's brother
<sayrer> patches welcome
<sayrer> asserts are the least of worries, really
<asqueella> why?
<sayrer> since the trunk is so horked 
<asqueella> they're supposed to be fatal
<asqueella> the state of trunk shouldn't affect the test harnesses :)
<sayrer> I'm not big on test harnesses that always fail
<sayrer> ymmv
Blocks: 379405
Summary: Need a way to run tests in which assertions are fatal which need a GUI → Need a way to run tests (mochikit/reftest) that display as failed when assertion failures are encountered
Duplicate of this bug: 370767
Are mochitests even run on debug builds?  That would be a first step to getting useful assertions...
I don't think so, which is actually rather bad: xpcshell tests explicitly fail if they trip assertions (but of course only in debug builds).
Blocks: 380625
Blocks: 385276
we have bug 372581 opened requesting debug builds. I think this is important and we should get some debug builds setup and running.
Blocks: 385246
Blocks: 370875
Blocks: 394120
Blocks: 386807
Blocks: 383872
Blocks: 385228
Blocks: 385880
Blocks: 385533
Blocks: 371483
Blocks: 367673
Blocks: 375058
Blocks: 397448
Blocks: 382636
Blocks: 387182
Blocks: 385265
Blocks: 397518
Everything left in Core:Testing is going to Testing:General. Filter on CleanOutCoreTesting to ignore.
Component: Testing → General
Product: Core → Testing
QA Contact: testing → general
Dupe of bug 472557 or bug 404077 (or the reverse?)
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #9)
> Dupe of bug 472557 or bug 404077 (or the reverse?)

Yes.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 404077
No longer blocks: 385228
No longer blocks: 471979
You need to log in before you can comment on or make changes to this bug.