110845 - MD5 digest fails, coredumps, stresstest hangs

Status: RESOLVED DUPLICATE of bug 110498

(NSS :: Libraries, defect, P1)

Description

[Sonja Mirtitsch]

Solaris 8 and 9 64 bit.

ssl.sh: SSL tests ===============================
ssl.sh: SSL Cipher Coverage ===============================
selfserv -p 8443 -d ../server -n kentuckyderby.red.iplanet.com \
         -w nss -c ABCDEFabcdefghijklm -i ../tests_pid.4303  &
selfserv started at Mon Nov 19 12:14:42 PST 2001
tstclnt -p 8443 -h kentuckyderby -q -d . <
/share/builds/mccrel/nss/nss322/builds/20010820.1/y2sun2_Solaris8/mozilla/security/nss/tests/ssl/sslreq.txt

ssl.sh: running SSL2 RC4 128 WITH MD5 ----------------------------
tstclnt -p 8443 -h kentuckyderby -c A -T \
        -f -d . <
/share/builds/mccrel/nss/nss322/builds/20010820.1/y2sun2_Solaris8/mozilla/security/nss/tests/ssl/sslreq.txt
tstclnt: write to SSL socket failed: MD5 digest function failed.

....
selfserv -p 8443 -d ../server -n kentuckyderby.red.iplanet.com \
         -w nss -r -r -i ../tests_pid.4303  &
selfserv started at Mon Nov 19 12:15:01 PST 2001
tstclnt -p 8443 -h kentuckyderby -q -d . <
/share/builds/mccrel/nss/nss322/builds/20010820.1/y2sun2_Solaris8/mozilla/security/nss/tests/ssl/sslreq.txt

tstclnt -p 8443 -h kentuckyderby -f -d . -T -n TestUser -w bogus \
        <
/share/builds/mccrel/nss/nss322/builds/20010820.1/y2sun2_Solaris8/mozilla/security/nss/tests/ssl/sslreq.txt
tstclnt: write to SSL socket failed: MD5 digest function failed.
ssl.sh: SSL3 Require client auth (bad password) produced a returncode of 254,
expected is 254 PASSED
4799 Terminated
ssl.sh: SSL3 Require client auth (client auth) ----
selfserv -p 8443 -d ../server -n kentuckyderby.red.iplanet.com \
         -w nss -r -r -i ../tests_pid.4303  &
selfserv started at Mon Nov 19 12:15:02 PST 2001
tstclnt -p 8443 -h kentuckyderby -q -d . <
/share/builds/mccrel/nss/nss322/builds/20010820.1/y2sun2_Solaris8/mozilla/security/nss/tests/ssl/sslreq.txt

tstclnt -p 8443 -h kentuckyderby -f -d . -T -n TestUser -w nss \
        <
/share/builds/mccrel/nss/nss322/builds/20010820.1/y2sun2_Solaris8/mozilla/security/nss/tests/ssl/sslreq.txt
tstclnt: write to SSL socket failed: MD5 digest function failed.
ssl.sh: SSL3 Require client auth (client auth) produced a returncode of 254,
expected is 0 FAILED
4816 Segmentation Fault - core dumped

...

ssl.sh: SSL Stress Test ===============================
ssl.sh: Stress SSL2 RC4 128 with MD5 ----
selfserv -p 8443 -d ../server -n kentuckyderby.red.iplanet.com \
         -w nss   -i ../tests_pid.4303  &
selfserv started at Mon Nov 19 12:15:04 PST 2001
tstclnt -p 8443 -h kentuckyderby -q -d . <
/share/builds/mccrel/nss/nss322/builds/20010820.1/y2sun2_Solaris8/mozilla/security/nss/tests/ssl/sslreq.txt

strsclnt -p 8443 -d . -w nss -c 1000 -C A  \
         kentuckyderby.red.iplanet.com
strsclnt started at Mon Nov 19 12:15:04 PST 2001
strsclnt: PR_Write returned error -12215:
MD5 digest function failed.
strsclnt: PR_Write returned error -12215:
MD5 digest function failed.
strsclnt: PR_Write returned error -12215:
MD5 digest function failed.




The selfserv disappears, the CPU goes to 100% and on rumraisin the ps looks like
follows:
UID   PID  PPID  C    STIME TTY      TIME CMD
   svbld  4500  4485 12   Nov 16 ?       1271:22 strsclnt -p 8443 -d . -w nss -c
1000 -C A rumraisin.red.iplanet.com
   svbld  9054  9024  0 14:00:50 pts/16   0:00 ps -fu svbld
   svbld  4485  4132  0   Nov 16 ?        0:00 /bin/sh
/share/builds/mccrel/nss/nss332/../nss322/builds/20010820.1/y2sun2_Sola
   svbld  2545   195  0   Nov 16 ?        0:00 sh -c /bin/sh /u/sonmi/bin/nssqa
-cron
   svbld  4009  2546  0   Nov 16 ?        0:00 /bin/sh
/share/builds/mccrel/nss/nss332/../nss322/builds/20010820.1/y2sun2_Sola
   svbld  4132  4009  0   Nov 16 ?        0:00 /bin/sh
/share/builds/mccrel/nss/nss332/../nss322/builds/20010820.1/y2sun2_Sola
   svbld  2546  2545  0   Nov 16 ?        0:00 /bin/sh /u/sonmi/bin/nssqa -cron
   svbld  4492  4485  0                   0:01 <defunct>
   svbld  9024  9022  0 13:54:21 pts/16   0:00 -tcsh

This can only be seen in the backward compatibility tests, since the regular
tests do a certutil coredump, preventing the ssl tests from running at all.
I will attach the output.log, and attempt to attach a stacktrace as well.

Comment 1

[Sonja Mirtitsch]

Attachment: output.log

Comment 2

[Wan-Teh Chang]

Kirk, could you investigate this bug?

Comment 3

[Sonja Mirtitsch]

Since I need the machine, and the hang is reproducible I will kill the
processes. Kirk, let me know when you want to start working on this, I will help
you to set it up.

Comment 4

[Wan-Teh Chang]

Here is the test environment and stack trace I received from Kirk.

kentuckyderby(80) echo $PATH
/usr/dist/pkgs/forte_dev,v6.2/SUNWspro/bin:/tools/ns/workshop/bin:/bin:/usr/bin:/usr/sbin:/tools/ns/bin:/usr/bin/X11:/usr/openwin/bin:/usr/ccs/bin:/usr/ucb/bin://usr/ucb:/usr/local/bin:/share/builds/mccrel/nss/nss332/builds/20011119.1/booboo_Solaris8/mozilla/dist/SunOS5.8_64_OPT.OBJ/bin
kentuckyderby(81) echo $LD_LIBRARY_PATH
/share/builds/mccrel/nss/nss332/builds/20011119.1/booboo_Solaris8/mozilla/dist/SunOS5.8_64_OPT.OBJ/lib
kentuckyderby(82) pwd
/share/builds/mccrel/nss/nss332/builds/20011119.1/booboo_Solaris8/mozilla/tests_results/security/bct/kentuckyderby.4/test

selfserv -p 8443 -d server -n kentuckyderby.red.iplanet.com -w nss
Segmentation fault (core dumped)
kentuckyderby(69) file core
core:           ELF 64-bit MSB core file SPARCV9 Version 1, from 'selfserv'
kentuckyderby(70) ls -l core
-rw-------   1 svbld    staff    1146928 Nov 20 12:21 core

detected a multithreaded program
t@1 (l@1) terminated by signal SEGV (no mapping at the fault address)
0xffffffff7ee2f420: PK11_GenerateKeyPair+0x0838:        ldx     [%i2 + 0x8], %g2
(/usr/dist/pkgs/forte_dev,v6.2/SUNWspro/bin/../WS6U2/bin/sparcv9/dbx) where
current thread: t@1
=>[1] PK11_GenerateKeyPair(0x1, 0x0, 0x0, 0x0, 0x0, 0xffffffff7fffec40), at
0xffffffff7ee2f420
[2] SECKEY_CreateRSAPrivateKey(0x200, 0xffffffff7fffef38, 0x0,
0xffffffff7f218c30, 0x0, 0x0), at 0xffffffff7ee368c8
[3] ssl3_CreateRSAStepDownKeys(0x100151dc0, 0x0, 0x10012be50,
0xffffffff7efa57c8, 0x100135940, 0x0), at 0xffffffff7f2116d0
[4] SSL_ConfigSecureServer(0x100151e90, 0x100151e50, 0x8, 0x100151dc0,
0x10012be50, 0x100151e98), at 0xffffffff7f218cb8
[5] server_main(0x100115c90, 0x0, 0x100118ce0, 0x100113768, 0x0, 0x100118ce0),
at 0x1000060f8
[6] main(0x100113768, 0x100118ce0, 0x100117a24, 0x0, 0x10011b170, 0x0), at
0x100006f4c
(/usr/dist/pkgs/forte_dev,v6.2/SUNWspro/bin/../WS6U2/bin/sparcv9/dbx) qioi
qioi: not found
(/usr/dist/pkgs/forte_dev,v6.2/SUNWspro/bin/../WS6U2/bin/sparcv9/dbx) quit
kentuckyderby(79) ls -l core
-rw-------   1 svbld    staff    1146928 Nov 20 12:21 core

kentuckyderby(74) setenv PATH
"/usr/dist/pkgs/forte_dev,v6.2/SUNWspro/bin:/tools/ns/workshop/bin:/bin:/usr/bin:/usr/sbin:/tools/ns/bin:/usr/bin/X11:/usr/openwin/bin:/usr/ccs/bin:/usr/ucb/bin://usr/ucb:/usr/local/bin:/share/builds/mccrel/nss/nsstip/builds/20011114.1/booboo_Solaris8/mozilla/dist/SunOS5.8_64_OPT.OBJ/bin"
kentuckyderby(75) setenv LD_LIBRARY_PATH
/share/builds/mccrel/nss/nsstip/builds/20011114.1/booboo_Solaris8/mozilla/dist/SunOS5.8_64_OPT.OBJ/lib
kentuckyderby(76) selfserv -p 8443 -d server -n kentuckyderby.red.iplanet.com -w nss
kentuckyderby(77) pwd
/share/builds/mccrel/nss/nss332/builds/20011119.1/booboo_Solaris8/mozilla/tests_results/security/bct/kentuckyderby.4/test

Comment 5

[Kirk Erickson]

A note about the previous attachment:
Sonja and I reproduced the failure by running selfserv alone
(there was no need to kick off tstclnt).  There's something
about 64bit Solaris 9 that breaks selfserv out of the gate.

It segv's in PK11_GenerateKeyPair() given the following args:
0x1, 0x0, 0x0, 0x0, 0x0, 0xffffffff7fffec40

PK11_GenerateKeyPair(PK11SlotInfo *slot,CK_MECHANISM_TYPE type,
   void *param, SECKEYPublicKey **pubKey, PRBool token,
                    PRBool sensitive, void *wincx)

So SECKEY_CreateRSAPrivateKey is calling with a NULL param value.
But in ./mozilla/security/nss/lib/cryptohi/seckey.c, param is 
a local being set before the call:

/* Create an RSA key pair is any slot able to do so.
** The created keys are "session" (temporary), not "token" (permanent),
** and they are "sensitive", which makes them costly to move to another token.
*/
SECKEYPrivateKey *
SECKEY_CreateRSAPrivateKey(int keySizeInBits,SECKEYPublicKey **pubk, void *cx)
{
    SECKEYPrivateKey *privk;
    PK11SlotInfo *slot = PK11_GetBestSlot(CKM_RSA_PKCS_KEY_PAIR_GEN,cx);
    PK11RSAGenParams param;

    param.keySizeInBits = keySizeInBits;
    param.pe = 65537L;

    privk = PK11_GenerateKeyPair(slot,CKM_RSA_PKCS_KEY_PAIR_GEN,&param,pubk,
                    PR_FALSE, PR_TRUE, cx);
    PK11_FreeSlot(slot);
    return(privk);
}

This problem only occurs in the optimized build.

Comment 6

[Robert Relyea]

You really can't trust the parameters in this case, even the first parameter is
clearly wrong. This is either an artifact of the crash (the stack frame is
horked), a problem related to the crach (the stack frame is crashed), or an
artifact of the optimized build (the information just isn't findable in the
optimized stack). Unfortunately experience has should the latter is the most likely.

On a much more distubing note: I can't seem to reproduce this with my builds!

Here are my specifics:
spd04(139) uname -a
SunOS spd04 5.8 Generic_108528-02 sun4u sparc SUNW,Ultra-2
spd04(140) cc -V
cc: Sun WorkShop 6 update 2 C 5.3 2001/05/15
usage: cc [ options] files.  Use 'cc -flags' for details
spd04(141) ld -V
ld: Software Generation Utilities - Solaris-ELF (4.0)

help!

Comment 7

[Sonja Mirtitsch]

sorry, no forte 6, you have to switch to 5.0 we have to build the regular 3.3.x
with workshop 5 for the 64 bit build. 

Comment 8

[Sonja Mirtitsch]

I am not 100% sure that is a duplicate of bug #110498, but I think the
possibility is high enough that we should delay working on this, until #110498
is resolved?

Comment 9

[Sonja Mirtitsch]

duplicate

*** This bug has been marked as a duplicate of 110498 ***