436845 - False failure detected in NSPR testing script.

Status: RESOLVED FIXED

(NSPR :: NSPR, defect)

Description

[Slavomir Katuscak]

Log file:
---
BEGIN TEST: udpsrv (2008-04-31 04:39:23)


END TEST: udpsrv (2008-04-31 04:46:07)
TEST STATUS: udpsrv = FAILED (errno 256)
---

From Perl documentation:
- The Termination Status of the terminated child is returned in 
  the special variable $?.  To get the Exit Status divide this by 
  256.

When I was searching which script we use to run NSPR tests, I found that we don't use runtest.sh from nsprpub/pr/tests directory, but we use runtests.pl from our local ns/securityqa/nightly. I'm going to fix the problem locally (divide by 256) and I suggest to add this file to mozilla CVS and change it's testing path.

Comment 1

[Slavomir Katuscak]

Attachment: runtests.pl

Copy of script already used in nightly testing. I haven't reviewed all the content, just copied + fixed problem with return values.

Comment 2

[Christophe Ravel]

Slavo,

Could you give me the diffs between this version in attachment and the version we have been using for a while ? I'd like to have you changes only.

Christophe.

Comment 3

[Slavomir Katuscak]

Attachment: Diff with changes.

This is a diff from local CVS.

Comment 4

[Christophe Ravel]

Comment on attachment 323353 [details] [diff] [review]
runtests.pl

r+ christophe.

Comment 5

[Wan-Teh Chang]

Why don't you use runtests.sh?  If you check in runtests.pl,
please remove runtests.sh from CVS.

Comment 6

[Slavomir Katuscak]

I don't know why, I found it out when I was troubleshooting failure above. Btw are there any other scripts using runtests.sh ? If there are, I would rather keep it in CVS.

Comment 7

[Christophe Ravel]

I had to rewrite runtests.sh in Perl because it wasn't working properly on Windows and there wasn't any way to fix it with the poor support of shell on Windows. Perl allows to monitor child processes more effectively and avoid having a single test block the whole sequence.

It wasn't yet proposed for Open Source because I wanted to give it some time to prove it is reliable.

Note: you have to add the Mozilla license text if you are going to commit this into the Mozilla CVS server.

Comment 8

[Nelson Bolyard (seldom reads bugmail)]

Christophe, what shell problem did you encounter on Windows? 
Perhaps I can help find a solution.
I would prefer that we use shell scripts rather than perl when we can.

Comment 9

[Christophe Ravel]

The main issue is that when you run a program from a main shell script, an intermediate shell script is created to run this program. The child pid you get is the pid of the shell script, not the program you want to test. So you can't kill this program if it is still running after the timeout.

Comment 10

[Slavomir Katuscak]

Attachment: New patch.

Adding license text, removing old shell script.

Comment 11

[Christophe Ravel]

Comment on attachment 324769 [details] [diff] [review]
New patch.

The license text should be changed from JSS to NSPR (see runtests.sh for an example)

Near the end, you can remove the commented line that I used for testing and didn't remove:

+#@progs = ("multiwait");

Comment 12

[Slavomir Katuscak]

Attachment: New patch.

Comment 13

[Christophe Ravel]

Comment on attachment 324943 [details] [diff] [review]
New patch.

r+=christophe

Comment 14

[Wan-Teh Chang]

Like Nelson, I'd also prefer to avoid the dependency on
Perl.  But I don't want that to become an obstacle to
progress.

Christophe, do you know why this shell problem (with killing
a program after the timeout on Windows) doesn't affect NSS's
test scripts?

Comment 15

[Christophe Ravel]

If I am not mistaken, for NSS we start selfserv in the background. Selfserv writes its pid in a file. We use this pid to kill it.

For the NSPR tests, we just start a test program. We don't know the actual pid of the process (on Windows).

The previous NSPR test script (runtests.sh) treated Windows differently for this reason. Basically, it was doing:

if [ $OS_PLATFORM = "Windows_95" ] || [ $OS_PLATFORM = "Windows_98" ] || [ $OS_PLATFORM = "Windows_NT" ] || [ $OS_PLATFORM = "OS/2" ] ; then
	for prog in $TESTS
	do
# This is Windows. Don't start the test program in the background
		./$prog >> ${LOGFILE} 2>&1
# Wait until it returns... if it does
	done
else
	for prog in $TESTS
	do
# This is Unix. Start the test program in the background
		./$prog >> ${LOGFILE} 2>&1 &
# Wait until it returns or time out
	done
fi

This was obviously a shortcoming and I didn't find a reliable way to fix it on Windows with just shell (MKS shell or Cygwin shell or MSYS shell).

The use of Perl is only to automate the NSPTR test runs, not to build it nor build the individual test programs.
- On Unix, we have Perl installed with the OS most of the time.
- On Windows, the previous script was not providing a good enough result (the whole test would hang forever at some points). The new Perl script offers a solution for those who want to run the tests in an automated way.

Comment 16

[Slavomir Katuscak]

Wan-Teh, I tried to commit latest patch to CVS, but I don't have write access there. Should I contact someone to get CVS access to NSPR directory, or would you commit it ? 

Comment 17

[Wan-Teh Chang]

Comment on attachment 324943 [details] [diff] [review]
New patch.

I can check in this patch for you.  Can you
use an indentation level of 4, and try to
avoid lines longer than 80 characters?

It seems that the file is indented using tabs.
So it would be nice to find a tool that can
expand the tabs to four spaces.

Comment 18

[Nelson Bolyard (seldom reads bugmail)]

> it would be nice to find a tool that can expand the tabs to four spaces.

The program pr is such a tool.  
   pr -e4 -l 1 < src > dst

(Note, some implementations use just -l instead of -l 1 to disable headers.)

Comment 19

[Wan-Teh Chang]

Attachment: New patch (tabs changed to 4 spaces; checked in)

Nelson, thanks for the pr command line.  I used it to
expand tabs to 4 spaces and checked in runtests.pl.  I
will remove runtests.sh from CVS later.

RCS file: /cvsroot/mozilla/nsprpub/pr/tests/runtests.pl,v
done
Checking in runtests.pl;
/cvsroot/mozilla/nsprpub/pr/tests/runtests.pl,v  <--  runtests.pl
initial revision: 1.1
done