765224 - (logspam) [Meta] Make job logs less verbose for non-failures

Status: NEW

(Testing :: General, defect)

Description

[Ed Morley [:emorley]]

Current log sizes for a couple of platforms (didn't intend for both to be OS X, but hey):
lion-debug_test-crashtest	 15,201K
lion-debug_test-jetpack	 197K
lion-debug_test-jsreftest	 15,918K
lion-debug_test-mochitest-other	 32,999K
lion-debug_test-mochitests-1	 28,559K
lion-debug_test-mochitests-2	 40,913K
lion-debug_test-mochitests-3	 6,229K
lion-debug_test-mochitests-4	 38,213K
lion-debug_test-mochitests-5	 3,183K
lion-debug_test-reftest	 11,258K
lion-debug_test-xpcshell	 805K
snowleopard-debug_test-crashtest	 15,046K
snowleopard-debug_test-jetpack	 166K
snowleopard-debug_test-jsreftest	 15,917K
snowleopard-debug_test-mochitest-other	 33,410K
snowleopard-debug_test-mochitests-1	 28,511K
snowleopard-debug_test-mochitests-2	 41,473K
snowleopard-debug_test-mochitests-3	 6,480K
snowleopard-debug_test-mochitests-4	 38,412K
snowleopard-debug_test-mochitests-5	 3,305K
snowleopard-debug_test-reftest	 11,247K
snowleopard-debug_test-xpcshell	 806K

This causes several problems:
1) Hangs every time someone opens the full log in the browser (ie bug 762710).
2) Logs to be truncated if they go over the 50MB limit. As much as someone whose name starts with 'A' would like to raise this limit; it's not the way forwards ;-)
3) Increases log storage requirements.
4) Increases time taken for TBPL to download/parse the logs.

I would like to:
1) Modify the handful of tests that are responsible for a disproportionate amount of the output.
2) Think about a more general 'elided entries' type solution, using an approach similar to bug 763169's part C patch.

Note: This would be to cut down on the output of INFO TEST-PASSes, not the failure cases.

Comment 1

[Ed Morley [:emorley]]

Meant to say:

Only consideration to the elided approach would be to ensure we don't exceed the 1200s timeout with no output.

Comment 2

[Ed Morley [:emorley]]

Bug 794458 and bug 794453 have already reduced mochitest-other logs by a fair amount (eg linux opt from 163K to 127K lines) :-)

Comment 3

[Andrew McCreight [:mccr8]]

I found a few tests that bloat up the logs by a fair amount for no real good reason.

Comment 4

[Ed Morley [:emorley]]

(In reply to Andrew McCreight [:mccr8] from comment #3)
> I found a few tests that bloat up the logs by a fair amount for no real good
> reason.

Much appreciated, thank you! :-D

Comment 5

[(not currently active) Ted Mielczarek]

It might be tricky, but if we could rework Mochitest to do something like the xpcshell harness that would be nice. xpcshell tests only display a single TEST-PASS line for the whole test file unless it fails, in which case they display a TEST-UNEXPECTED-FAIL and dump the entire log.

Comment 6

[ISHIKAWA, Chiaki]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #5)
> It might be tricky, but if we could rework Mochitest to do something like
> the xpcshell harness that would be nice. xpcshell tests only display a
> single TEST-PASS line for the whole test file unless it fails, in which case
> they display a TEST-UNEXPECTED-FAIL and dump the entire log.

This is a great approach.

Except: I have found that some test platforms, notably |make mozmill| test suite of
C-C TB fails to notice there are serious JavaScript errors and such during test execution, but if the end result is OK (heaven knows how much we can trust the conclusion when we see
undefined variables, etc. along the way),
it announces success of the test.

So as long as the test harness properly picks up problematic warnings/errors properly,
I agree with the approach in general.

Secondly, we really need to have an INFO-level output.
Today, we have
ERROR (or stacktrace for serious errors coming from some form of asserts.)
WARNING (NS_WARNING comes to my mind)

But I notice after looking at voluminous log from |make mozmill| for a few years now,
that most of the messages are actually
INFO (a piece of information for developers to guide their future efforts by
giving some information on the run-time behavior of the code, etc.)

By creating a macro/function to print out 
INFO: blah...
in the log and replacing inappropriate NS_WARNING and other WARNING-level output routines in the source code,
we can cope with REAL warnings and errors in a saner way.
We can even weed out INFO-level messages fromt the automatic build logs, etc.

Local test or specially run log can record INFO-level messages.

Just a thought.

TIA

Comment 7

[(not currently active) Ted Mielczarek]

(In reply to ISHIKAWA, Chiaki from comment #6)
> (In reply to Ted Mielczarek [:ted.mielczarek] from comment #5)
> > It might be tricky, but if we could rework Mochitest to do something like
> > the xpcshell harness that would be nice. xpcshell tests only display a
> > single TEST-PASS line for the whole test file unless it fails, in which case
> > they display a TEST-UNEXPECTED-FAIL and dump the entire log.
> 
> This is a great approach.

This got implemented in bug 937181.