156296 - Backward compatibility tests on Linuxes failing on nightly builds of tip

Status: RESOLVED FIXED

(NSS :: Test, defect, P1)

Description

[Bishakha Banerjee]

I see from the script that it fails while trying to create CA, while trying to
execute this part of all.sh

CU_ACTION="Creating CA Cert DB"
certu -N -d . -f ${R_PWFILE} 2>&1
if [ "$RET" -ne 0 ]; then
    Exit 5 "Fatal - failed to create CA"
fi

The failure in the output file is listed as this (certutil - command not found):

certutil -N -d . -f ../tests.pw.7974
/share/builds/sbsrel2/nss/nss35/../nss322/builds/20010820.1/y2sun2_Solaris8/mozilla/dist/../security/nss/tests/all.sh:
certutil: command not found
cert.sh: Exit: 5 Fatal - failed to create CA

I checked all the paths, they appear OK.  The thing I haven't checked is whether
"tests.pw.7974" (I assume a temp file with that particular process ID) exists or
not during the execution. The same script works OK on the Solaris and AIX machines.

Comment 1

[Sonja Mirtitsch]

We got the builds to work here, Linux 2.4 is building right now, and I'll see if
I can reproduce the behavior with a tip build

Comment 2

[Bishakha Banerjee]

Sonja reported that the Linux BC tests on the tip passed yesterday. The scripts
she used the scripts that are checked in to CVS.

So it must be a Netscape specific problem. I "echoed" the paths, SHLIBPATHs and
LD_LIBRARY_PATHS, and they all appear OK. In any case, the same scripts work on
other Unix systems here.

Here's the path info echoed out:
TESTSCRIPTDIR /share/builds/sbsrel2/nss/nsstip/../nss322/builds/20010820.1/y2sun
2_Solaris8/mozilla/dist/../security/nss/tests
TESTDIR /share/builds/sbsrel2/nss/nsstip/builds/20020710.1/spd04_Solaris8/mozill
a/tests_results/security/bct
PATH  /share/builds/sbsrel2/nss/nsstip/../nss322/builds/20010820.1/y2sun2_Solari
s8/mozilla/dist/../security/nss/tests:/share/builds/sbsrel2/nss/nsstip/../nss322
/builds/20010820.1/y2sun2_Solaris8/mozilla/dist/Linux2.4_x86_glibc_PTH_DBG.OBJ/b
in:/lib:/usr/lib:/bin:/sbin:/usr/bin:/usr/sbin:.:/u/bishakhabanerjee/bin:/tools/
ns/bin:/usr/ccs/bin:/usr/dist/local/exe:/usr/bin/X11:/usr/audio/bin:/u/sonmi/bin
:/usr/local/bin:/usr/X11R6/bin
SHLIBPATH /share/builds/sbsrel2/nss/nsstip/builds/20020710.1/spd04_Solaris8/mozi
lla/dist/Linux2.4_x86_glibc_PTH_DBG.OBJ/lib
LIBPATH /share/builds/sbsrel2/nss/nsstip/builds/20020710.1/spd04_Solaris8/mozill
a/dist/Linux2.4_x86_glibc_PTH_DBG.OBJ/lib


Next, Sonja gave me some good suggestions. From her email:
"
which compiler and redhat version (cat /etc/redhat*) are you using on Linux?
There are a lot of incompatibilities, we were using a homegrown gcc 3.0.x for a
while, which will break backward compatibility to 2.96. Also, if the NSS version
that you are running against was built on Linux 2.2 (Redhat 6.x) and the 2.4
directory (redhat 7.x) is just a symlink, it will cause troubles. You might want
to check with Anthony on that.
"

Since we (NSS lab) did not have a Linux 2.2 machine, I had not been running
nightly QA on it. Hence it was also not obvious here. I used Anthony's Linux 2.2
machine, and BC tests worked. With Anthony's permission, I added "spd09" to the
nightly QA list. It looks like there were probably space issues on sbsrel2
yesterday, some of the builds failed, and some of the nightly QA is reported as
incomplete (the files got created, but nothing written in them). I have also
lost my results from yesterday testing, since Anthony removed all build and QA
tress till yesterday to make space.

However, what I had found is that we are currently building on Red Hat Linux
Advanced Server, and our BC testing is done against either shabadoo_Linux2.2
(for Lin 2.2) and y2sun2_Solaris8 (for Lin 2.4). 
I also tested on "spd16", the machine Anthony uses to build the other Linux
build, and tested against scripts in the y2Sun2_Solaris8 directory. Those tests
failed. Also, the SHLIBPATH and the LIBPATH point to libraries in
spd04_Solaris8/mozilla/dist/Linux2.4_x86_glibc_PTH_DBG.OBJ/lib

Sonja, the build that you tested against, did you build on a Lin 2.4 machine?

-Bishakha

Comment 3

[Wan-Teh Chang]

This bug is caused by a local configuration problem at Netscape.
Our BC test baseline NSS 3.2.2 build has Linux2.4*.OBJ as symlinks
pointing to nonexistent directories.  We should ask Anthony to
remove those Linux2.4*OBJ symlinks and build
nss/nss322/builds/20010820.1 on Linux 2.4.

Comment 4

[Bishakha Banerjee]

Hi Wan-Teh,

I've already asked Anthony to build on a Linux 2.4 machine. He was going to
build with the NSS_3_2_2_RTM tag, which is the same as the 20010820.1 build. He
will be using on one running RedHat 7.1, as that is probably what was there at
the time.

-Bishakha

Comment 5

[Bishakha Banerjee]

fixed.

Comment 6

[Bishakha Banerjee]

marking fixed