Closed Bug 451474 Opened 12 years ago Closed 11 years ago

When a (TUnit) |make check| test fails, 'make' stops and following tests are silently skipped

Categories

(Testing :: XPCShell Harness, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED WORKSFORME

People

(Reporter: sgautherie, Unassigned)

References

Details

Spun off from bug 447999 comment 9:
{{
Serge Gautherie   2008-08-20 12:44:28 PDT

With that fix, MacOSX TB TUnit jumped from "233/1" to "327/0".
MacOSX SM is at "234/1", while Linux and Windows are at 303-304/0.
It looks like the failure of that test is preventing other tests to be run :-(
Could that be looked into and fixed too ?
}}

***

IIRC, Ted checked and confirmed:
{{
TUnit within a file ends on error

xpcshell (TUnit) tests are javascript files
in one test file, if a test fails, no more tests in that file are run
actually, it may stop the make process too

make is stopping when it hits that failure
so it just doesn't run the rest of the tests

biesi mentioned it at a session at the summit
}}

***

Examples:

Now Green:
233/1 : http://tinderbox.mozilla.org/showlog.cgi?log=Thunderbird/1219246711.1219247745.30775.gz&fulltext=1
327/0 : http://tinderbox.mozilla.org/showlog.cgi?log=Thunderbird/1219255809.1219257125.23027.gz&fulltext=1

Still Orange:
234/1 : http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1219269497.1219274130.4319.gz&fulltext=1

*****

Ideally, a failing test should let the following tests run (as it does for the other kind of tests).

But, if that is not possible, the '/1' should be replaced by '/FAILED'.
Or, if the total number of tests is known, by '/NumberOfFailedAndSkippedTests'.

The current issue is that '1':
*possibly underestimates the number of failing tests: reduced test coverage
*doesn't make it obvious (enough) that the priority is to fix/restore this situation.
*total number of test (good + bad) varies :-(
(In reply to comment #0)
> Ideally, a failing test should let the following tests run (as it does for the
> other kind of tests).

Well, mochitest wuld seem to have it too:

http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1219259846.1219264157.9072.gz
Linux comm-central dep unit test on 2008/08/20 12:17:26
mochitest 61728/0/1615

http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1219264315.1219269700.24884.gz
Linux comm-central dep unit test on 2008/08/20 13:31:55
mochitest 56618/5/1615
Depends on: 485736
Version: unspecified → Trunk
Depends on: 485672
OS: Mac OS X → All
Hardware: x86 → All
Summary: When a 'TUnit' test fails, 'make' stops and following tests are silently skipped → When a (TUnit) |make check| test fails, 'make' stops and following tests are silently skipped
Duplicate of this bug: 477405
This works differently now with the top-level xpcshell-tests target.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.