348905 - strsclnt segfault + selfserv HDX PR_Read Error

Status: NEW

(NSS :: Libraries, defect, P3)

Description

[Slavomir Katuscak]

sl.sh: Stress TLS RC4 128 with MD5 (client auth) ----
selfserv -D -p 8444 -d ../ext_server -n nssamdrhel3.red.iplanet.com  \
         -e nssamdrhel3.red.iplanet.com-ec -w nss -r -r -i ../tests_pid.23838  &
selfserv started at Wed Aug 16 03:06:28 PDT 2006
tstclnt -p 8444 -h nssamdrhel3.red.iplanet.com -B -s -q \
        -d ../ext_client < /share/builds/mccrel3/security/securitytip/builds/20060816.1/nssamdrhel3_RHEL3/mozilla/security/nss/tests/ssl/sslreq.dat
strsclnt -q -p 8444 -d ../ext_client -B -s -w nss -c 100 -C c -N -n ExtendedSSLUser \
          nssamdrhel3.red.iplanet.com
strsclnt started at Wed Aug 16 03:06:28 PDT 2006
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
strsclnt: -- SSL: Server Certificate Validated.
selfserv: HDX PR_Read returned error -5961:
TCP connection reset by peer.
./all.sh: line 738:  1258 Segmentation fault      (core dumped) strsclnt -q -p ${PORT} -d ${P_R_CLIENTDIR} ${CLIENT_OPTIONS} -w nss $cparam $verbose ${HOSTADDR}
strsclnt completed at Wed Aug 16 03:06:29 PDT 2006
<TR><TD>Stress TLS RC4 128 with MD5 (client auth). Core file is detected.
ssl.sh: Stress TLS RC4 128 with MD5 (client auth) produced a returncode of 139, expected is 0.  FAILED
selfserv: normal termination

Appeared in nightly testing 20060816.1 securitytip on nssamdrhel3, Linux2.4_x86_64_glibc_PTH_64_OPT.OBJ.

From log is not clear for me what exactly failed, first error is in selfserv, but it has terminated normally. It looks like an error on 738 line of all.sh, but this script doesn't have so much lines.

Core wasn't found in /share/builds/mccrel3/security/securitytip/builds.

Comment 1

[Nelson Bolyard (seldom reads bugmail)]

got stack trace?

Comment 2

[Slavomir Katuscak]

No core, no stack trace. 

Comment 3

[Nelson Bolyard (seldom reads bugmail)]

The output in comment 0 clearly says: "Core file is detected."

No core, no stack trace, no bug.  

Comment 4

[Julien Pierre]

Slavo,

There is a core . You need to grep for the word core, not just look for a file named "core" . On some machines we configured core files to have the process id and the process name where feasible . This is the case here .

[jp96085@monstre]/share/builds/mccrel3/security/securitytip/builds/20060816.1/nssamdrhel3_RHEL3/mozilla/tests_results 119 % find . | grep core
./security/nssamdrhel3.4/ext_client/core.1258

Here is the stack from gdb when loading this core . Unfortunately it's from an optimized build :

#0  0x0000002a963b5de5 in SECITEM_CompareItem ()
   from /share/builds/mccrel3/security/securitytip/builds/20060816.1/nssamdrhel3_RHEL3/mozilla/dist/Linux2.4_x86_64_glibc_PTH_64_OPT.OBJ/lib/libfreebl3.so
#1  0x0000002a963929d3 in get_blinding_params ()
   from /share/builds/mccrel3/security/securitytip/builds/20060816.1/nssamdrhel3_RHEL3/mozilla/dist/Linux2.4_x86_64_glibc_PTH_64_OPT.OBJ/lib/libfreebl3.so
#2  0x0000002a96392e87 in rsa_PrivateKeyOp ()
   from /share/builds/mccrel3/security/securitytip/builds/20060816.1/nssamdrhel3_RHEL3/mozilla/dist/Linux2.4_x86_64_glibc_PTH_64_OPT.OBJ/lib/libfreebl3.so
#3  0x0000002a9624223b in RSA_PrivateKeyOpDoubleChecked ()
   from /share/builds/mccrel3/security/securitytip/builds/20060816.1/nssamdrhel3_RHEL3/mozilla/dist/Linux2.4_x86_64_glibc_PTH_64_OPT.OBJ/lib/libsoftokn3.so
#4  0x0000002a9624107a in RSA_Sign ()
   from /share/builds/mccrel3/security/securitytip/builds/20060816.1/nssamdrhel3_RHEL3/mozilla/dist/Linux2.4_x86_64_glibc_PTH_64_OPT.OBJ/lib/libsoftokn3.so
#5  0x0000002a96236223 in NSC_Sign ()
   from /share/builds/mccrel3/security/securitytip/builds/20060816.1/nssamdrhel3_RHEL3/mozilla/dist/Linux2.4_x86_64_glibc_PTH_64_OPT.OBJ/lib/libsoftokn3.so
#6  0x0000002a958f8d14 in PK11_Sign ()
   from /share/builds/mccrel3/security/securitytip/builds/20060816.1/nssamdrhel3_RHEL3/mozilla/dist/Linux2.4_x86_64_glibc_PTH_64_OPT.OBJ/lib/libnss3.so
#7  0x0000002a95676552 in ssl3_SignHashes ()
   from /share/builds/mccrel3/security/securitytip/builds/20060816.1/nssamdrhel3_RHEL3/mozilla/dist/Linux2.4_x86_64_glibc_PTH_64_OPT.OBJ/lib/libssl3.so
#8  0x0000002a9567a92b in ssl3_SendCertificateVerify ()
   from /share/builds/mccrel3/security/securitytip/builds/20060816.1/nssamdrhel3_RHEL3/mozilla/dist/Linux2.4_x86_64_glibc_PTH_64_OPT.OBJ/lib/libssl3.so
#9  0x0000002a9567bb5a in ssl3_HandleServerHelloDone ()
   from /share/builds/mccrel3/security/securitytip/builds/20060816.1/nssamdrhel3_RHEL3/mozilla/dist/Linux2.4_x86_64_glibc_PTH_64_OPT.OBJ/lib/libssl3.so
#10 0x0000002a9567e958 in ssl3_HandleHandshakeMessage ()
   from /share/builds/mccrel3/security/securitytip/builds/20060816.1/nssamdrhel3_RHEL3/mozilla/dist/Linux2.4_x86_64_glibc_PTH_64_OPT.OBJ/lib/libssl3.so
#11 0x0000002a9567eca8 in ssl3_HandleHandshake ()
   from /share/builds/mccrel3/security/securitytip/builds/20060816.1/nssamdrhel3_RHEL3/mozilla/dist/Linux2.4_x86_64_glibc_PTH_64_OPT.OBJ/lib/libssl3.so
#12 0x0000002a9567efba in ssl3_HandleRecord ()
   from /share/builds/mccrel3/security/securitytip/builds/20060816.1/nssamdrhel3_RHEL3/mozilla/dist/Linux2.4_x86_64_glibc_PTH_64_OPT.OBJ/lib/libssl3.so
#13 0x0000002a9567fb0c in ssl3_GatherCompleteHandshake ()
   from /share/builds/mccrel3/security/securitytip/builds/20060816.1/nssamdrhel3_RHEL3/mozilla/dist/Linux2.4_x86_64_glibc_PTH_64_OPT.OBJ/lib/libssl3.so
#14 0x0000002a95681369 in ssl_GatherRecord1stHandshake ()
   from /share/builds/mccrel3/security/securitytip/builds/20060816.1/nssamdrhel3_RHEL3/mozilla/dist/Linux2.4_x86_64_glibc_PTH_64_OPT.OBJ/lib/libssl3.so
#15 0x0000002a956852dd in ssl_Do1stHandshake ()
   from /share/builds/mccrel3/security/securitytip/builds/20060816.1/nssamdrhel3_RHEL3/mozilla/dist/Linux2.4_x86_64_glibc_PTH_64_OPT.OBJ/lib/libssl3.so
#16 0x0000002a9568671b in ssl_SecureSend ()
---Type <return> to continue, or q <return> to quit---
   from /share/builds/mccrel3/security/securitytip/builds/20060816.1/nssamdrhel3_RHEL3/mozilla/dist/Linux2.4_x86_64_glibc_PTH_64_OPT.OBJ/lib/libssl3.so
#17 0x0000002a9568a36b in ssl_Send ()
   from /share/builds/mccrel3/security/securitytip/builds/20060816.1/nssamdrhel3_RHEL3/mozilla/dist/Linux2.4_x86_64_glibc_PTH_64_OPT.OBJ/lib/libssl3.so
#18 0x0000002a95c6aa7d in PR_Send ()
   from /share/builds/mccrel3/security/securitytip/builds/20060816.1/nssamdrhel3_RHEL3/mozilla/dist/Linux2.4_x86_64_glibc_PTH_64_OPT.OBJ/lib/libnspr4.so
#19 0x0000000000405cb8 in handle_connection ()
#20 0x00000000004060ef in do_connects ()
#21 0x0000000000405694 in thread_wrapper ()
#22 0x0000002a95c81ffe in _pt_root ()
   from /share/builds/mccrel3/security/securitytip/builds/20060816.1/nssamdrhel3_RHEL3/mozilla/dist/Linux2.4_x86_64_glibc_PTH_64_OPT.OBJ/lib/libnspr4.so
#23 0x0000002a95db6c6b in start_thread () from /lib64/tls/libpthread.so.0
#24 0x0000002a9609fa83 in thread_start () from /lib64/tls/libc.so.6

Comment 5

[Julien Pierre]

One more thing, the core is from strsclnt, not selfserv, as evidenced by the debugger output when loading the core :

Core was generated by `strsclnt -q -p 8444 -d ../ext_client -B -s -w nss -c 100 -C c -N -n ExtendedSSL'.
Program terminated with signal 11, Segmentation fault.

This is consistent with the test output which says :

"selfserv: normal termination" .
The message comes from selfserv itself, so we know selfserv didn't crash.

The test output also contains the following message :
strsclnt completed at Wed Aug 16 03:06:29 PDT 2006

But that message comes from ssl.sh, not from strsclnt . It is misleading .
Upon success, strsclnt would output the cache statistics, and they are missing here.

So, I'm changing the description of this bug to be a strsclnt segfault, not selfserv.

Comment 6

[Julien Pierre]

Unfortunately, because of the wonderful OS that is RHEL3, and the fact that each thread is a different process, gdb can't see any stack from any other thread than the one that crashed.

(gdb) info threads
  9 process 1258  0x474f423d4f2c6d6f in ?? ()
  8 process 1261  0x474f423d4f2c6d6f in ?? ()
  7 process 1262  0x474f423d4f2c6d6f in ?? ()
  6 process 1263  0x616c432061746e61 in ?? ()
  5 process 1265  0x656872646d617373 in ?? ()
  4 process 1266  0x474f423d4f2c6d6f in ?? ()
  3 process 1267  0x474f423d4f2c6d6f in ?? ()
  2 process 1268  0x474f423d4f2c6d6f in ?? ()
* 1 process 1264  0x0000002a963b5de5 in SECITEM_CompareItem ()
   from /share/builds/mccrel3/security/securitytip/builds/20060816.1/nssamdrhel3_RHEL3/mozilla/dist/Linux2.4_x86_64_glibc_PTH_64_OPT.OBJ/lib/libfreebl3.so
(gdb) thread 2
[Switching to thread 2 (process 1268)]#0  0x474f423d4f2c6d6f in ?? ()
(gdb) w
Ambiguous command "w": watch, wh, whatis, where, while, while-stepping, winheight, ws.
(gdb) where
#0  0x474f423d4f2c6d6f in ?? ()
Cannot access memory at address 0x532c77656956206e
(gdb) thread 9
[Switching to thread 9 (process 1258)]#0  0x474f423d4f2c6d6f in ?? ()
(gdb) where
#0  0x474f423d4f2c6d6f in ?? ()
Cannot access memory at address 0x532c77656956206e
(gdb) thread 3
[Switching to thread 3 (process 1267)]#0  0x474f423d4f2c6d6f in ?? ()
(gdb) where
#0  0x474f423d4f2c6d6f in ?? ()
Cannot access memory at address 0x532c77656956206e
(gdb) thread 7
[Switching to thread 7 (process 1262)]#0  0x474f423d4f2c6d6f in ?? ()
(gdb) where
#0  0x474f423d4f2c6d6f in ?? ()
Cannot access memory at address 0x532c77656956206e
(gdb) thread 5
[Switching to thread 5 (process 1265)]#0  0x656872646d617373 in ?? ()
(gdb) where
#0  0x656872646d617373 in ?? ()
Cannot access memory at address 0x746972756365732f
(gdb) thread 4
[Switching to thread 4 (process 1266)]#0  0x474f423d4f2c6d6f in ?? ()
(gdb) where
#0  0x474f423d4f2c6d6f in ?? ()
Cannot access memory at address 0x532c77656956206e
(gdb) thread 3
[Switching to thread 3 (process 1267)]#0  0x474f423d4f2c6d6f in ?? ()
(gdb) where
#0  0x474f423d4f2c6d6f in ?? ()
Cannot access memory at address 0x532c77656956206e
(gdb) thread 2
[Switching to thread 2 (process 1268)]#0  0x474f423d4f2c6d6f in ?? ()
(gdb) where
#0  0x474f423d4f2c6d6f in ?? ()
Cannot access memory at address 0x532c77656956206e
(gdb) thread 1