362433 - xpcshell-tests: add dump() to each do_check_*(), at least

Status: RESOLVED FIXED

(Testing :: XPCShell Harness, enhancement)

Description

[(not reading, please use [e-mail] instead)]

[RFE] option to "make check" be more verbose

When I run the tests, and they pass I get something like this:

sh-3.1$ pwd
/cygdrive/c/builds/trunk/mozilla
sh-3.1$ cd ff-debug/browser/components/places/tests/
sh-3.1$ make check
../../../../dist/bin/test_all.sh ../../../../dist/bin/test_places
../../../../dist/bin/test_places/test_bookmarks.js: PASS 
../../../../dist/bin/test_places/test_livemarks.js: PASS 

Is there a way we could add a "make checkverbose" target so that we'd see more of what's going on?  I was thinking about http://lxr.mozilla.org/seamonkey/source/tools/test-harness/xpcshell-simple/head.js and doing a dump() on all do_check_*() functions.

robert, what do you think?

Comment 1

[Rob Campbell [:rc] (:robcee)]

I think that's fine if-wanted, but the tests themselves will have to be rewritten to handle the extra verbosity. An integer value for verbosity a la xUnit might be nice. e.g., 0 = terse, 1 = standard 2+ = verbose ?

Comment 2

[Robert Sayre]

(In reply to comment #1)
> I think that's fine if-wanted, but the tests themselves will have to be
> rewritten to handle the extra verbosity. An integer value for verbosity a la
> xUnit might be nice. e.g., 0 = terse, 1 = standard 2+ = verbose ?
> 

Let's use the same thing mochikit does-- a Log4J approach. DEBUG, INFO, ERROR, etc. Test failures are logged at ERROR level.

Comment 3

[Rob Campbell [:rc] (:robcee)]

make check output on the mac is different from the other platforms. If I run make -k check in my objdir, I get notice that deps are being updated, but it doesn't tell me where. Linux and Windows both report which directories are being entered and left.

Comment 4

[Serge Gautherie (:sgautherie)]

(In reply to comment #0)
> http://lxr.mozilla.org/seamonkey/source/tools/test-harness/xpcshell-simple/head.js
> doing a dump() on all do_check_*() functions.

Just had this idea too while modifying a test ;-)

(In reply to comment #1)
> I think that's fine if-wanted, but the tests themselves will have to be
> rewritten to handle the extra verbosity.

Let's morph this bug:
add dump()s to what we have now (only), which will go to the individual log files,
then leave the rest to bug 468196.

(In reply to comment #3)
> make check output on the mac is different from the other platforms. If I run
> make -k check in my objdir, I get notice that deps are being updated, but it
> doesn't tell me where. Linux and Windows both report which directories are
> being entered and left.

That's unrelated to this bug, is it?

Comment 5

[Serge Gautherie (:sgautherie)]

Attachment: (Av1) Log check results [Checkin: See comment 8 & 9]

The output is still a little raw, but this is a major enhancement yet.


[Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.2a1pre) Gecko/20090531 Minefield/3.6a1pre] (mozilla-central-win32-unittest/1243808976) (W2Ksp4)
(http://hg.mozilla.org/mozilla-central/rev/ab395a1916be)

628 TEST-PASS = 579 with checks + 49 without checks.

Comment 6

[Rob Campbell [:rc] (:robcee)]

Comment on attachment 381133 [details] [diff] [review]
(Av1) Log check results
[Checkin: See comment 8 & 9]

I think this looked pretty good on a quick pass.

coop, do you have time to give this a once-over?

Comment 7

[Chris Cooper [:coop] (he/him)]

Comment on attachment 381133 [details] [diff] [review]
(Av1) Log check results
[Checkin: See comment 8 & 9]

> function do_test_pending() {
>-  dump("*** test pending\n");
>-  _tests_pending++;
>+  dump("TEST-INFO | (xpcshell/head.js) | test " + (++_tests_pending) +
>+         " pending\n");
> }

My only concern is incrementing _tests_pending while we're concatenating. I'd prefer that as two distinct statements.

Comment 8

[Serge Gautherie (:sgautherie)]

Comment on attachment 381133 [details] [diff] [review]
(Av1) Log check results
[Checkin: See comment 8 & 9]


http://hg.mozilla.org/mozilla-central/rev/07971f68e728
(Av1a) Log check results

Comment 9

[Serge Gautherie (:sgautherie)]

Comment on attachment 381133 [details] [diff] [review]
(Av1) Log check results
[Checkin: See comment 8 & 9]


http://hg.mozilla.org/releases/mozilla-1.9.1/rev/3bab31a7104f

after fixing context for
{
patching file testing/xpcshell/head.js
Hunk #2 FAILED at 40
1 out of 5 hunks FAILED
patching file testing/xpcshell/runxpcshelltests.py
Hunk #1 FAILED at 14
Hunk #2 FAILED at 208
2 out of 2 hunks FAILED
}

Comment 10

[Serge Gautherie (:sgautherie)]

verified1.9.1, per bug 496776 comment 4.