332272 - add core detection functionality to all.sh

Status: RESOLVED FIXED

(NSS :: Test, defect, P1)

Description

[Alexei Volkov]

all.sh should be able to detect any cores that there dumped during a test run.
It should be done in test post execution step.

Neil, does window have an ability to create core files(or any other process mem dumps)? I have an impression that it can do so, if VS is installed, but not sure about it.

Comment 1

[Julien Pierre]

I think DrWatson allows you to get some sort of dump with stacks, but it's not nearly as useful as a core file. You need to configure the machine in a special way, and tell it where to store the files. I wouldn't put a high priority on making this work on Windows. Let's make it work on all the Unix platforms first.

Let's assume all our Unix systems are configured to have a core file name that includes the string "core" and to dump core where the program executes, ie. in tests_results .

I think the core detection should be done not just in all.sh be in each of the individual scripts, ie. cipher.sh, ssl.sh, etc. Sometimes I run only one of those scripts and I want to look at results.html and make sure it found any cores. The core detection test should run only once, either in the individual test script, or in all.sh .

Comment 2

[Julien Pierre]

Adding bug 315798 as a dependency. We won't be able to detect core files if we don't know for sure where our programs will dump them.

Comment 3

[Julien Pierre]

This task is required to complete testing of selfserv reference leaks. Marking as a blocker for bug 331413 .

Comment 4

[Alexei Volkov]

Attachment: possible solution(unix version)

will run as a part of clean up after each script. Will track core generated by each script individually

Comment 5

[Alexei Volkov]

Comment on attachment 216800 [details] [diff] [review]
possible solution(unix version)

Since cleanup.sh is called from every script we execute in all.sh, all script will inherit thiat functionality. 

I've also verified that script works on HP-UX as well.

Comment 6

[Alexei Volkov]

Comment on attachment 216800 [details] [diff] [review]
possible solution(unix version)

result file was not updating with info

Comment 7

[Julien Pierre]

Comment on attachment 216800 [details] [diff] [review]
possible solution(unix version)

Two problems :

a) the result doesn't appear correctly in the table in results.html

b) when running all.sh, the core file check is repeated as part of each subscript. This is unnecessary and even misleading, since core files from previous subscripts will be found. 
I suggest that when running all.sh, the test be performed only once, at the end, and not in each subscript. It should be performed in the subscripts only when they are run individually and not from all.sh . You can use environment variables to detect this.

Comment 8

[Alexei Volkov]

Attachment: an example of a result file

Comment 9

[Alexei Volkov]

(In reply to comment #7)
> (From update of attachment 216800 [details] [diff] [review] [edit])
> Two problems :
> 
> a) the result doesn't appear correctly in the table in results.html
> 
This is why I've removed the previous attachment. New patch does fix the problem.

> b) when running all.sh, the core file check is repeated as part of each
> subscript. This is unnecessary and even misleading, since core files from
> previous subscripts will be found. 
This is not correct. Each time a core is found it get checked against CORELIST_FILE. If the core with the path already exists, it is not detected as new core file and not reported. If it is a new core file, it get listed in a CORELIST_FILE. 

Collisions like, for example, one program core overwrites another old core file, is the problem. Check sum is calculated to solve it. See new attachment.

Comment 10

[Alexei Volkov]

Attachment: core detection patch

I disagree with Julien. It is good to check core after each script since core files can collide with eachother.
New patch can identify collisions.

Comment 11

[Nelson Bolyard (seldom reads bugmail)]

Alexei,  I think the following bit of shell code does the same as your
code, without the O(N^2) loops.  Please test this and see if it is 
satisfactory.  

detect_core() {
    testRunDir="$1"

    echo Core list:
    cat $CORELIST_FILE

    html "<TABLE BORDER=1>"
    [ ! -f $CORELIST_FILE ] && touch $CORELIST_FILE
    mv $CORELIST_FILE ${CORELIST_FILE}.old
    coreStr=$(find $testRunDir -type f -name '*core*')
    res=0
    if [ -n "$coreStr" ]; then
        sum $coreStr > $CORELIST_FILE
        res=$(cat $CORELIST_FILE ${CORELIST_FILE}.old | sort | uniq -u | wc -l)
    fi
    html_msg $res 0  "Script $SCRIPTNAME: Post execution check for core files."
    html "</TABLE><BR>"

}

if [ "${OS_ARCH}" != "WINNT" \
  -a "${OS_ARCH}" != "WIN95" \
  -a "${OS_ARCH}" != "OS2" ]; then
    detect_core ${HOSTDIR}
fi

Comment 12

[Nelson Bolyard (seldom reads bugmail)]

er, one more change.  Change

    echo Core list:
    cat $CORELIST_FILE

to 

    echo Core list:
    cut -f 3 $CORELIST_FILE

Comment 13

[Alexei Volkov]

Nelson, it is very compact script, and it produses proper results.

Comment 14

[Nelson Bolyard (seldom reads bugmail)]

Alexei, Thank you for working on this bug. 
Please integrate my patch with yours and create another patch for review.
Thanks.

Comment 15

[Alexei Volkov]

Attachment: patch

the patch detects core files and reports "Core" status for each test.

Comment 16

[Nelson Bolyard (seldom reads bugmail)]

Comment on attachment 217467 [details] [diff] [review]
patch

r=nelson, with some suggested changes.

First, maybe you want to use "$@" instead of "$*" in some places.
Second:

>     HTML_FAILED='</TD><TD bgcolor=red>Failed</TD><TR>'
>+    HTML_FAILED_CORE='</TD><TD bgcolor=red>Core</TD><TR>'

Make that

>+    HTML_FAILED_CORE='</TD><TD bgcolor=red>Failed Core</TD><TR>'

so that a search for "Failed" will find all the failures, including 
the ones with core.

Comment 17

[Alexei Volkov]

I agree, "$@" is a better choice as it preserves caller arguments, but it does not really matter in this situation.

"Core" and change to "Failed Core" as well. 

Comment 18

[Alexei Volkov]

Patch modified according review comment #16 and checked into the tip.

/cvsroot/mozilla/security/nss/tests/common/init.sh,v  <--  init.sh
new revision: 1.42; previous revision: 1.41

Comment 19

[Alexei Volkov]

Checking into 3.11 branch.
/cvsroot/mozilla/security/nss/tests/common/init.sh,v  <--  init.sh
new revision: 1.40.24.1; previous revision: 1.40