156959 - Incorrect warnings on number of tests in nightly QA reports

Status: RESOLVED FIXED

(NSS :: Test, defect)

Description

[Bishakha Banerjee]

The nightly QA reports these warnings, e.g.,

WARNING - more tests than expected on adleman.1/results.html ( 252)
WARNING - more tests than expected on adleman.2/results.html ( 252)
WARNING - more tests than expected on diffie.1/results.html ( 252)
WARNING - more tests than expected on diffie.2/results.html ( 252)

This is due to a value hardcoded in the scripts which is incorrect for the
present configuration at Netscape. The details of the bug are:

TOTAL_TESTS (which PASSED_LINES checks against is hardcoded to be 244, also for
the tip). Since PASSED_LINES echoes out to be "252" which is greater than "244",
this warning appears. I have to estimate the number of tests we should be doing
and correct that appropriately.

This is what the particular piece of code does:

        PASSED_LINES=""
        PASSED_LINES=`grep bgcolor=lightGreen $w | wc -l`
        if [ -n "$PASSED_LINES" -a "$PASSED_LINES" -lt "$TOTAL_TESTS" ] ; then
            BUILDPLATFORM=`grep Platform $w | sed -e 's/<H4>Platform:/    /'
               -e 's/<BR>//'`
            MACHINE=`echo $w | sed -e "s/.results.html//"`
            #MACHINE=`echo $w | sed -e "s/\.[0-9]*.results.html//"`
            setQAsysvars
            html_line incomplete >>$TMP_HTML_FILE
        elif [ "$PASSED_LINES" -gt "$TOTAL_TESTS" ] ; then
            echo "WARNING - more tests than expected on $w ($PASSED_LINES)"
>>$WARNINGLIST
        fi

[bishakhabanerjee@rivest tests]$ grep "PASSED_LINES" -r .
./qa_stat:        PASSED_LINES=""
./qa_stat:        PASSED_LINES=`grep bgcolor=lightGreen $w | wc -l`
./qa_stat:        if [ -n "$PASSED_LINES" -a "$PASSED_LINES" -lt "$TOTAL_TESTS"]
; then
./qa_stat:        elif [ "$PASSED_LINES" -gt "$TOTAL_TESTS" ] ; then
./qa_stat:            echo "WARNING - more tests than expected on $w
($PASSED_LINES)" >> $WARNINGLIST

[bishakhabanerjee@rivest tests]$ grep "TOTAL_TESTS" -r .
./qa_stat:#TOTAL_TESTS=106
./qa_stat:TOTAL_TESTS=244 #3.4
./qa_stat:#TOTAL_TESTS=123 #3.3.2
./qa_stat:BCT_TOTAL_TESTS=122 #3.2.2
./qa_stat:#TOTAL_TESTS=133 #tip
./qa_stat:        if [ -n "$PASSED_LINES" -a "$PASSED_LINES" -lt "$TOTAL_TESTS"]
; then
./qa_stat:        elif [ "$PASSED_LINES" -gt "$TOTAL_TESTS" ] ; then
./qa_stat:    TOTAL_TESTS=$BCT_TOTAL_TESTS

Comment 1

[Sonja Mirtitsch]

what we are running here has 252 as total tests. Do you want me to check it in?

Comment 2

[Bishakha Banerjee]

Well, it'll prevent the warning from getting printed, but what we should really
do is estimate the number here, and use that.

Its not necessary to check it in, since I already have other customizations of
the scripts here at Netscape. Anthony has created space on
share/builds/sbstools/nsstools/nssqa where the customised scripts reside, and
where the crontabs look to.

Comment 3

[Sonja Mirtitsch]

OK, that's probably better. We should occasionally compare our customizations so
we don't divert too far though.
Just a few words on the background of the number of tests: In the dark ages of
automated QA, tests were hanging, or killed, or just died, and it was hard to
find out if QA was complete or not. I started to see if "$PASSED_LINES" -lt
"$TOTAL_TESTS" and make it an "Incomplete" if so. It occured to me, that as more
tests are added it would probably get to the same point again if the value of
TOTAL_TESTS was not updated on a regular basis - so I put the warning in.

Comment 4

[Bishakha Banerjee]

Attachment: patch

fixed in our local scripts here at Netscape after estimating the number of
tests we do currently is 252.

If the number changes in the future, this will need to be modified accordingly.


Submitting patch to update the script in CVS since both Sun and Netscape are
using the same numbers currently.

Comment 5

[Bishakha Banerjee]

fix checked into tip.

Comment 6

[Bishakha Banerjee]

fix verified in nightly QA, so marking this bug fixed.