168575 - certutil failures on HP-UX 64 bit

Status: RESOLVED WORKSFORME

(NSS :: Tools, defect, P3)

Description

[Sonja Mirtitsch]

first problem was a hang of certutil -L, whicjh I killed. The parentprocess was
restarted (by the shell?) log and resultfiles were re-used, I assume so were
already created databases etc. for the certs for the regular tests (extended
were not yet started)

Is this a critical extension [y/n]? 
cert.sh: Exporting Root Cert --------------------------
certutil -L -n clientCA -r -d . -o root.cert
cert.sh: Certutil Tests ===============================

after this the cert.sh script started over, in the same directory tree. Certs
for the regular tests were created fine this time, when it started on the
extended tests (new directories) following tests failed:

Import serverCA CA -t C,C,C for charm.red.iplanet.com (ext.) (255) 
Import chain-1-serverCA CA -t u,u,u for charm.red.iplanet.com (ext.) (255)
Import Server Root CA -t C,C,C for ExtendedSSLUser (ext.) (255)

certutil -A -n serverCA -t C,C,C -f ../tests.pw.10309 -d . -i
/share/builds/mccrel3/nss/nsstip/builds/20020913.1/booboo_Solaris8/mozilla/tests_results/security/charm.4/serverCA/serverCA.ca.cert
certutil: could not obtain certificate from file: Failure to load dynamic library.
cert.sh ERROR: Import serverCA CA -t C,C,C for charm.red.iplanet.com (ext.) 
failed 255
cert.sh: Import chain-1-serverCA CA -t u,u,u for charm.red.iplanet.com (ext.) 
--------------------------
certutil -A -n chain-1-serverCA -t u,u,u -f ../tests.pw.10309 -d . -i
/share/builds/mccrel3/nss/nsstip/builds/20020913.1/booboo_Solaris8/mozilla/tests_results/security/charm.4/serverCA/chain-1-serverCA.ca.cert
certutil: could not obtain certificate from file: Failure to load dynamic library.
cert.sh ERROR: Import chain-1-serverCA CA -t u,u,u for charm.red.iplanet.com
(ext.)  failed 255
cert.sh: Import chain-2-serverCA CA -t u,u,u for charm.red.iplanet.com (ext.) 
--------------------------
certutil -A -n chain-2-serverCA -t u,u,u -f ../tests.pw.10309 -d . -i
/share/builds/mccrel3/nss/nsstip/builds/20020913.1/booboo_Solaris8/mozilla/tests_results/security/charm.4/serverCA/chain-2-serverCA.ca.cert
Client Cert
cert.sh: Initializing ExtendedSSLUser's Cert DB (ext.) --------------------------
certutil -N -d . -f ../tests.pw.10309
cert.sh: Generate Cert Request for ExtendedSSLUser (ext) --------------------------
certutil -s "CN=ExtendedSSLUser, E=[e-mail], O=BOGUS NSS,
L=Mountain View, ST=California, C=US" -R -d . -f ../tests.pw.10309 -z
../tests_noise.10309 -o req


As expected in this case all extended ssl tests, depending on the ext. certs
failed too.

I have seen hangs and core-dumps on 64 bit tinderbox tests on this machine too,
the machine runs 2 tinderboxes, plus nighly QA 32 and 64 bit. I am installing a
gdb that I will use the next time tinderbox hangs.

Comment 1

[Wan-Teh Chang]

The gdb that really works is installed in /opt/langtools/bin.
There are gdb32 and gdb64.  gdb is a symlink to gdb32.

It will be useful to get the stack trace in the core file.
Thanks.

Comment 2

[Sonja Mirtitsch]

Attachment: /home/sonmi/tmp/hp64/charm.4/result.html

Comment 3

[Sonja Mirtitsch]

Attachment: output.log

Comment 4

[Bishakha Banerjee]

Just some additional information:

I have run the QA here on five 64 bit HP UX 11.00 machines today, and have not
seen a single failure

Comment 5

[Sonja Mirtitsch]

> The gdb that really works is installed in /opt/langtools/bin.

Thanks.

I found that I had saved the core it dumped on 9/9 by dbtest, but I could not
debug it:

/opt/langtools/bin/gdb64
/share/builds/mccrel3/nss/nsstip/builds/20020909.1/booboo_Solaris8/mozilla/dist/HP-UXB.11.00_64_DBG.OBJ/bin/dbtest
-core core
HP gdb 2.1
Copyright 1986 - 1999 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 2.1 (based on GDB 5.0-hpwdb-20000630)
Wildebeest is free software, covered by the GNU General Public License, and
you are welcome to change it and/or distribute copies of it under certain
conditions.  Type "show copying" to see the conditions.  There is
absolutely no warranty for Wildebeest.  Type "show warranty" for details.
Wildebeest was built for PA-RISC 2.0 (wide), HP-UX 11.00.
..
Core was generated by `dbtest'.
Program terminated with signal 4, Illegal instruction.
Error trying to get information about dynamic linker.
#0  0x800003ffff78d9c8 in ?? ()
(gdb)  bt
#0  0x800003ffff78d9c8 in ?? ()
#1  0x800003ffff78d8c8 in ?? ()
Cannot access memory at address 0xc00000000009e45f
(gdb) 


> I have run the QA here on five 64 bit HP UX 11.00 machines today, and have not
> seen a single failure

tinderbox failures here show up every 200+ runs (~ of 50 QA runs / day x 2 (DBG
and OPT) failure shows up 2-3 times a week) 

Comment 6

[Sonja Mirtitsch]

I saw failures on yesterday's QA again.

a process was hanging, ps showed it as all.sh, after killing it QA resumed, but
a test failed - Import ExtendedSSLUser's Cert -t u,u,u (ext) (137)
from the log it looks like I killed a certutil, rather than all.sh

Note that there should only 2 processes named all.sh show in the ps, the real
all.sh (PID 7506) , and the sourced cert.sh (PID 7541)

If I see hangs again, I will try to guess what process it is, and attempt to
attach a debugger. 

What patch level do you have on the HP-UX machines in M.V.? I noticed that there
were patches on Sun's machine orville, that caused the system to fail tests. I
am also not certain how to look up patch levels on HP


cert.sh: Sign ExtendedSSLUser's Request (ext) --------------------------
certutil -C -c chain-2-clientCA -m 1 -v 60 -d
/share/builds/mccrel3/nss/nsstip/builds/20020917.1/booboo_Solaris8/mozilla/tests_results/security/charm.4/clientCA
-i req -o ExtendedSSLUser.cert -f ../tests.pw.7506
cert.sh: Import ExtendedSSLUser's Cert -t u,u,u (ext) --------------------------
certutil -A -n ExtendedSSLUser -t u,u,u -d . -f ../tests.pw.7506 -i
ExtendedSSLUser.cert
/share/builds/mccrel3/nss/nsstip/builds/20020917.1/booboo_Solaris8/mozilla/security/nss/tests/all.sh[97]:
7625 Killed
cert.sh ERROR: Import ExtendedSSLUser's Cert -t u,u,u (ext) failed 137
cert.sh: Import Server Root CA -t C,C,C for ExtendedSSLUser (ext.)
--------------------------



charm[svbld]:/tmp_mnt/u/svbld> ps -u svbld | grep -v pts
   PID TTY       TIME COMMAND
  7541 ?         0:00 all.sh
  7506 ?         0:00 all.sh
  7625 ?        1178:58 all.sh
 24067 ?         0:00 sh
 24075 ?         0:00 sh
charm[svbld]:/tmp_mnt/u/svbld> ps -fu svbld | grep -v pts
     UID   PID  PPID  C    STIME TTY       TIME COMMAND
   svbld  7541  7506  0  Sep 17  ?         0:00 /bin/sh
/share/builds/mccrel3/nss/nsstip/builds/20020917.1/b
   svbld  7506 24075  0  Sep 17  ?         0:00 /bin/sh
/share/builds/mccrel3/nss/nsstip/builds/20020917.1/b
   svbld  7625  7541 255  Sep 17  ?        1179:03 /bin/sh
/share/builds/mccrel3/nss/nsstip/builds/20020917.1/b
   svbld 24067   941  0  Sep 17  ?         0:00 sh -c /bin/sh
/u/svbld/ns/securityqa/nssqa -cron
   svbld 24075 24067  0  Sep 17  ?         0:00 /bin/sh
/u/svbld/ns/securityqa/nssqa -cron
charm[svbld]:/tmp_mnt/u/svbld> ps -lu svbld | grep -v pts
  F S        UID   PID  PPID  C PRI NI             ADDR   SZ            WCHAN
TTY       TIME COMD
  1 S       7865  7541  7506  0 158 22         427b1c00  117          1fb4a40 ?
        0:00 all.sh
  1 S       7865  7506 24075  0 158 22         47042400   97          1fba440 ?
        0:00 all.sh
  1 R       7865  7625  7541 255 245 22         470c6600  117                - ?
       1179:10 all.sh
  1 S       7865 24067   941  0 158 22         45e43a00   61          1fca340 ?
        0:00 sh
  1 S       7865 24075 24067  0 158 22         41779300  146          1fca700 ?
        0:00 sh
charm[svbld]:/tmp_mnt/u/svbld>
charm[svbld]:/tmp_mnt/u/svbld> kill 7625
charm[svbld]:/tmp_mnt/u/svbld> ps -u svbld | grep -v pts
   PID TTY       TIME COMMAND
  7541 ?         0:00 all.sh
  7506 ?         0:00 all.sh
  7625 ?        1180:02 all.sh
 24067 ?         0:00 sh
 24075 ?         0:00 sh
charm[svbld]:/tmp_mnt/u/svbld> kill -9 7625
charm[svbld]:/tmp_mnt/u/svbld>  ps -u svbld | grep -v pts
   PID TTY       TIME COMMAND
  7541 ?         0:00 all.sh
  7506 ?         0:00 all.sh
 26224 ?         0:00 certutil
 24067 ?         0:00 sh
 24075 ?         0:00 sh
charm[svbld]:/tmp_mnt/u/svbld>

$
Import ExtendedSSLUser's Cert -t u,u,u (ext) (137)

Comment 7

[Wan-Teh Chang]

I don't know what patch levels we have on our HP-UX
machines here.

Comment 8

[Bishakha Banerjee]

This is the software,patch and kernel info on the HP machine 'sbshp1' we use to
run our builds and tests:

------------------------------------------------------------------------------------------
  B3901BA                       B.11.01.20     HP C/ANSI C Developer's Bundle
for HP-UX 11.00 (S800)
  B3913DB                       C.03.25        HP aC++ Compiler (S800)
  B5456CA                       C.01.17.00     HP-UX Development Kit for Java*
  B6256CA                       B.11.01.06     HP aC++ Ecom Compiler (S800)
  HPUXEng64RT                   B.11.00        English HP-UX 64-bit Runtime
Environment
  J3515A                        B.11.00.04     100BT/9000 GSC for B, C, D, and J
Series
  OnlineDiag                    B.11.00.10.12  HPUX 11.0 Support Tools Bundle
  XSWGR1100                     B.11.00.46     HP-UX General Release Patches,
October 1999
  XSWHWCR1100                   B.11.00.47     Hardware Enablement and Critical
Patches for HP-UX 11.00 (December 1999)
#
# Product(s) not contained in a Bundle:
#

  PHCO_18685                    B.11.00.04     HP Array Manager/60 cumulative patch
  PHKL_18490                    1.0            diag0 handle more devices,fix
data corrupton
  PHKL_19060                    1.0            LVM Cumulative patch
  tusc                          7.1            tusc
------------------------------------------------------------------------------------------
 

Also,

The following kernel parameters on 3 Oct. 2000 at 12 o'clock noon.
    1. maxfiles.  old value = 60.  new value = 100.
    2. nkthread.  old value = 499.  new value = 1328.
    3. max_thread_proc.  old value = 64.  new value = 512.
    4. maxusers.  old value = 32.  new value = 64.
    5. maxuprc.  old value = 75.  new value = 512.
    6. nproc.  old formula = 20+8*MAXUSERS, which evaluated to 276.
       new value (note: not a formula) = 750.

Comment 9

[Sonja Mirtitsch]

more failures on the tinderbox, the shell - all.sh dumped a core here, we have a
patchlevel on charm which is higher than yours, maybe I'll try to have them
reinstall it. I'll check the kernelparameters too.

file core
core: core file from 'all.sh' - receiverd SIGILL

fips.sh: List the FIPS module certificates -----------------
certutil -d ../fips -L
FIPS_PUB_140-1_Test_Certificate                              Cu,Cu,Cu
fips.sh: Delete the certificate and key fr 
om the FIPS module
certutil -d ../fips -F -n FIPS_PUB_140-1_Test_Certificate -f ../tests.fipspw.16390
/export/nss_tbx_hpux11.00_64/builds/tinderbox/HP-UX-11.00-64/mozilla/security/nss/tests/all.sh[97]:
20043 Illegal instruction(coredump)
=============


# uname -a
HP-UX charm B.11.00 U 9000/800 1549706547 unlimited-user license
# swlist
# Initializing...
# Contacting target "charm"...
#
# Target:  charm:/
#

#
# Bundle(s):
#

  B3901BA               B.11.02.03     HP C/ANSI C Developer's Bundle for HP-UX
11.00 (S800) 
  B3913DB               C.03.30.03     HP aC++ Compiler (S800) 
  B5456CA               C.01.18.04     HP-UX Development Kit for Java* 
  HPUXEng64RT           B.11.00        English HP-UX 64-bit Runtime Environment

  OnlineDiag            B.11.00.18.09  HPUX 11.0 Support Tools Bundle, Sep 2001

  QPK1100               B.11.00.54.7   Quality Pack for HP-UX 11.00, September
2001 
  XSWHWCR1100           B.11.00.54.6   HP-UX Hardware Enablement and Critical
Patches, September 2001 

Comment 10

[Julien Pierre]

There was no update in almost 5 years on this bug. I don't believe it was ever reproduced. Slavo, if you have seen this, feel free to reopen this bug.

Comment 11

[Nelson Bolyard (seldom reads bugmail)]

Julien, you resolved this bug as fixed.  
In what release was it fixed?
Please set the target fix to record that.