intermittent reference leak in strsclnt caused by race in importing temp cert from server

RESOLVED FIXED in 3.11.3

Status

NSS
Libraries
P1
normal
RESOLVED FIXED
12 years ago
12 years ago

People

(Reporter: Julien Pierre, Assigned: Julien Pierre)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(3 attachments)

(Assignee)

Description

12 years ago
I did a very large number of all.sh runs over the last week. A number of them resulted in core files due to a reference leak assertion in SECMOD_Shutdown() .

Here is the info from the core files about the tests that were run, and the corresponding stacks (the stacks are all the same).

core './monstre.28/client/core' of 25367:	strsclnt -q -p 8443 -d ../client -w nss -c 10 -C :C004 -N monstre.red.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.147/ext_client/core' of 21091:	strsclnt -q -p 8443 -d ../ext_client -B -s -w nss -c 10 -C :C004 -N mo
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.150/ext_client/core' of 22694:	strsclnt -q -p 8443 -d ../ext_client -w nss -c 10 -C :C00E -N monstre.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.381/ext_client/core' of 18823:	strsclnt -q -p 3000 -d ../ext_client -w nss -c 10 -C :C00E -N monstre.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.384/ext_client/core' of 14373:	strsclnt -q -p 3000 -d ../ext_client -B -s -w nss -c 10 -C :C00E -N mo
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.423/client/core' of 9782:	strsclnt -q -p 3000 -d ../client -w nss -c 10 -C :C00E -N monstre.red.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.432/ext_client/core' of 6827:	strsclnt -q -p 3000 -d ../ext_client -w nss -c 10 -C :C004 -N monstre.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.450/client/core' of 26511:	strsclnt -q -p 3000 -d ../client -w nss -c 10 -C :C00E -N monstre.red.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.524/client/core' of 136:	strsclnt -q -p 3000 -d ../client -w nss -c 10 -C :C004 -N monstre.red.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.556/ext_client/core' of 25792:	strsclnt -q -p 3000 -d ../ext_client -w nss -c 10 -C :C004 -N monstre.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.556/client/core' of 24318:	strsclnt -q -p 3000 -d ../client -w nss -c 10 -C :C004 -N monstre.red.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.572/client/core' of 18512:	strsclnt -q -p 3000 -d ../client -w nss -c 10 -C :C004 -N monstre.red.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.575/ext_client/core' of 15518:	strsclnt -q -p 3000 -d ../ext_client -B -s -w nss -c 10 -C :C004 -N mo
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.611/ext_client/core.strsclnt.1136' of 1136:	strsclnt -q -p 3000 -d ../ext_client -w nss -c 10 -C :C00E -N monstre.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.614/ext_client/core.strsclnt.29580' of 29580:	strsclnt -q -p 3000 -d ../ext_client -w nss -c 10 -C :C004 -N monstre.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.615/client/core.strsclnt.14604' of 14604:	strsclnt -q -p 3000 -d ../client -B -s -w nss -c 10 -C :C004 -N monstr
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.620/client/core.strsclnt.5177' of 5177:	strsclnt -q -p 3000 -d ../client -B -s -w nss -c 10 -C :C004 -N monstr
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.621/client/core.strsclnt.24673' of 24673:	strsclnt -q -p 3000 -d ../client -B -s -w nss -c 10 -C :C00E -N monstr
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.637/ext_client/core.strsclnt.11279' of 11279:	strsclnt -q -p 3000 -d ../ext_client -w nss -c 10 -C :C00E -N monstre.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.667/ext_client/core.strsclnt.26473' of 26473:	strsclnt -q -p 3000 -d ../ext_client -w nss -c 10 -C :C00E -N monstre.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.677/ext_client/core.strsclnt.11537' of 11537:	strsclnt -q -p 3000 -d ../ext_client -w nss -c 10 -C :C00E -N monstre.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.694/client/core.strsclnt.8633' of 8633:	strsclnt -q -p 3000 -d ../client -B -s -w nss -c 10 -C :C00E -N monstr
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.772/ext_client/core.strsclnt.4508' of 4508:	strsclnt -q -p 3000 -d ../ext_client -w nss -c 10 -C :C00E -N monstre.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.775/ext_client/core.strsclnt.3041' of 3041:	strsclnt -q -p 3000 -d ../ext_client -w nss -c 10 -C :C00E -N monstre.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.815/client/core.strsclnt.28703' of 28703:	strsclnt -q -p 3000 -d ../client -B -s -w nss -c 10 -C :C00E -N monstr
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.820/ext_client/core.strsclnt.7732' of 7732:	strsclnt -q -p 3000 -d ../ext_client -B -s -w nss -c 10 -C :C004 -N mo
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.833/client/core.strsclnt.5859' of 5859:	strsclnt -q -p 3000 -d ../client -w nss -c 10 -C :C00E -N monstre.red.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.842/client/core.strsclnt.28340' of 28340:	strsclnt -q -p 3000 -d ../client -B -s -w nss -c 10 -C :C00E -N monstr
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.863/client/core.strsclnt.20986' of 20986:	strsclnt -q -p 3000 -d ../client -w nss -c 10 -C :C004 -N monstre.red.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.911/ext_client/core.strsclnt.25706' of 25706:	strsclnt -q -p 3000 -d ../ext_client -B -s -w nss -c 10 -C :C004 -N mo
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.918/client/core.strsclnt.10781' of 10781:	strsclnt -q -p 3000 -d ../client -B -s -w nss -c 10 -C :C004 -N monstr
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.921/ext_client/core.strsclnt.13773' of 13773:	strsclnt -q -p 3000 -d ../ext_client -w nss -c 10 -C :C004 -N monstre.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.948/client/core.strsclnt.28920' of 28920:	strsclnt -q -p 3000 -d ../client -w nss -c 10 -C :C004 -N monstre.red.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.957/client/core.strsclnt.24483' of 24483:	strsclnt -q -p 3000 -d ../client -w nss -c 10 -C :C004 -N monstre.red.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.976/ext_client/core.strsclnt.6593' of 6593:	strsclnt -q -p 3000 -d ../ext_client -w nss -c 10 -C :C004 -N monstre.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.997/client/core.strsclnt.21675' of 21675:	strsclnt -q -p 3000 -d ../client -B -s -w nss -c 10 -C :C004 -N monstr
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.1003/client/core.strsclnt.21745' of 21745:	strsclnt -q -p 3000 -d ../client -w nss -c 10 -C :C004 -N monstre.red.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.1006/ext_client/core.strsclnt.21753' of 21753:	strsclnt -q -p 3000 -d ../ext_client -w nss -c 10 -C :C00E -N monstre.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.1058/client/core.strsclnt.24494' of 24494:	strsclnt -q -p 3000 -d ../client -B -s -w nss -c 10 -C :C004 -N monstr
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.1074/ext_client/core.strsclnt.11130' of 11130:	strsclnt -q -p 3000 -d ../ext_client -w nss -c 10 -C :C004 -N monstre.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.1093/client/core.strsclnt.20224' of 20224:	strsclnt -q -p 3000 -d ../client -w nss -c 10 -C :C00E -N monstre.red.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.1098/client/core.strsclnt.24745' of 24745:	strsclnt -q -p 3000 -d ../client -B -s -w nss -c 10 -C :C004 -N monstr
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.1099/client/core.strsclnt.14246' of 14246:	strsclnt -q -p 3000 -d ../client -B -s -w nss -c 10 -C :C004 -N monstr
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.1152/ext_client/core.strsclnt.2521' of 2521:	strsclnt -q -p 3000 -d ../ext_client -w nss -c 10 -C :C004 -N monstre.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.1158/client/core.strsclnt.28047' of 28047:	strsclnt -q -p 3000 -d ../client -w nss -c 10 -C :C004 -N monstre.red.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.1193/ext_client/core.strsclnt.22476' of 22476:	strsclnt -q -p 3000 -d ../ext_client -w nss -c 10 -C :C00E -N monstre.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.1234/client/core.strsclnt.13758' of 13758:	strsclnt -q -p 3000 -d ../client -B -s -w nss -c 10 -C :C00E -N monstr
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.1248/ext_client/core.strsclnt.4276' of 4276:	strsclnt -q -p 3000 -d ../ext_client -w nss -c 10 -C :C004 -N monstre.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.1253/client/core.strsclnt.10288' of 10288:	strsclnt -q -p 3000 -d ../client -w nss -c 10 -C :C004 -N monstre.red.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.1291/ext_client/core.strsclnt.29815' of 29815:	strsclnt -q -p 3000 -d ../ext_client -B -s -w nss -c 10 -C :C004 -N mo
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.1297/ext_client/core.strsclnt.26843' of 26843:	strsclnt -q -p 3000 -d ../ext_client -B -s -w nss -c 10 -C :C00E -N mo
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.1309/client/core.strsclnt.19371' of 19371:	strsclnt -q -p 3000 -d ../client -B -s -w nss -c 10 -C :C00E -N monstr
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.1317/client/core.strsclnt.25419' of 25419:	strsclnt -q -p 3000 -d ../client -B -s -w nss -c 10 -C :C00E -N monstr
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.1345/client/core.strsclnt.1463' of 1463:	strsclnt -q -p 3000 -d ../client -B -s -w nss -c 10 -C :C00E -N monstr
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.1345/client/core.strsclnt.4448' of 4448:	strsclnt -q -p 3000 -d ../client -w nss -c 10 -C :C004 -N monstre.red.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.1353/ext_client/core.strsclnt.11950' of 11950:	strsclnt -q -p 3000 -d ../ext_client -w nss -c 10 -C :C00E -N monstre.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.1355/ext_client/core.strsclnt.20923' of 20923:	strsclnt -q -p 3000 -d ../ext_client -w nss -c 10 -C :C00E -N monstre.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.1356/ext_client/core.strsclnt.7448' of 7448:	strsclnt -q -p 3000 -d ../ext_client -B -s -w nss -c 10 -C :C00E -N mo
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.1385/ext_client/core.strsclnt.27030' of 27030:	strsclnt -q -p 3000 -d ../ext_client -w nss -c 10 -C :C004 -N monstre.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()
core './monstre.1389/ext_client/core.strsclnt.15029' of 15029:	strsclnt -q -p 3000 -d ../ext_client -w nss -c 10 -C :C004 -N monstre.
 fffffd7ffeeade5a _lwp_kill () + a
 fffffd7ffee573e9 raise () + 19
 fffffd7ffee3a3d0 abort () + 90
 fffffd7fff065b8e PR_Assert () + 7e
 fffffd7fff1d8637 SECMOD_Shutdown () + 217
 fffffd7fff16e4b1 NSS_Shutdown () + 51
 000000000040fb77 main () + a57
 000000000040b73c ???????? ()

From the above, I can tell that these assertions occurred only when the cipher string was -C :C004 or -C :C00E . I'm not sure exactly what cipher suites and/or curves these map to.
Priority: -- → P1
Target Milestone: --- → 3.11.1

Comment 1

12 years ago
(In reply to comment #0)
> From the above, I can tell that these assertions occurred only when the cipher
> string was -C :C004 or -C :C00E . I'm not sure exactly what cipher suites
> and/or curves these map to.

Here are the cipher suite numbers for ECC. The numbers above are
for ECDH-ECDSA-AES128-SHA and ECDH-RSA-AES128-SHA. Were these
the only cipher suites in the tests for their respective key exchange
methods (ECDH-ECDSA and ECDH-RSA) or were there other ciphers
from each group? 

     CipherSuite TLS_ECDH_ECDSA_WITH_NULL_SHA           = { 0xC0, 0x01 }
     CipherSuite TLS_ECDH_ECDSA_WITH_RC4_128_SHA        = { 0xC0, 0x02 }
     CipherSuite TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA   = { 0xC0, 0x03 }
     CipherSuite TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA    = { 0xC0, 0x04 }
     CipherSuite TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA    = { 0xC0, 0x05 }

     CipherSuite TLS_ECDHE_ECDSA_WITH_NULL_SHA          = { 0xC0, 0x06 }
     CipherSuite TLS_ECDHE_ECDSA_WITH_RC4_128_SHA       = { 0xC0, 0x07 }
     CipherSuite TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA  = { 0xC0, 0x08 }
     CipherSuite TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA   = { 0xC0, 0x09 }
     CipherSuite TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA   = { 0xC0, 0x0A }

     CipherSuite TLS_ECDH_RSA_WITH_NULL_SHA             = { 0xC0, 0x0B }
     CipherSuite TLS_ECDH_RSA_WITH_RC4_128_SHA          = { 0xC0, 0x0C }
     CipherSuite TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA     = { 0xC0, 0x0D }
     CipherSuite TLS_ECDH_RSA_WITH_AES_128_CBC_SHA      = { 0xC0, 0x0E }
     CipherSuite TLS_ECDH_RSA_WITH_AES_256_CBC_SHA      = { 0xC0, 0x0F }

     CipherSuite TLS_ECDHE_RSA_WITH_NULL_SHA            = { 0xC0, 0x10 }
     CipherSuite TLS_ECDHE_RSA_WITH_RC4_128_SHA         = { 0xC0, 0x11 }
     CipherSuite TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA    = { 0xC0, 0x12 }
     CipherSuite TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA     = { 0xC0, 0x13 }
     CipherSuite TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA     = { 0xC0, 0x14 }

vipul


Comment 2

12 years ago
I believe these are all ECDHE ciphers. I should have a patch for this problem...

Comment 3

12 years ago
Created attachment 215902 [details] [diff] [review]
CHECKED IN:Free ECDHE Ephemeral key. Fixes server-side leak.

This patch free's the ecdhe ephemeral key on socket close, and 'copies' it on socket copy (copy increments a reference).

NOTE for reviewers: the ecdhe ephmeral key is defined unconditionally (not enclosed in the NSS_ENABLE_ECC), so I did not enclose the copy and free in NSS_ENABLE_ECC either.
Attachment #215902 - Flags: superreview?(alexei.volkov.bugs)
Attachment #215902 - Flags: review?(julien.pierre.bugs)

Comment 4

12 years ago
OK, I'm confused. As vipul pointed out, the ciphers in question are ECDH ciphers, Not ECDHE ciphers. This is a little confusing because it's clear we have a leak in ECDHE. The explanations I can see are:

1) we aren't testing ECDHE in all.sh.
2) we are running ECDHE ciphers before the strsclnt tests.
3) we aren't detecting selfserv leaks (this is a server side leak).
4) 1 and/or 3 and this is a different leak that the patch I attached is for.

The ciphers in question are both ECDH_AES_128 ciphers. If this is a client only problem, then it's unlikely the patch I attached will fix it.

bob
(Assignee)

Comment 5

12 years ago
Bob,

Re: comment 4, I'll examine each of the possibilities .

1) We are running both ECDH and ECDHE tests with tstclnt. However, the multithreaded stress test only runs with ECDH tests, no ECDHE . This makes no sense and is a problem with our test coverage.

3) is definitely true - we aren't detecting server-side leaks ! I put a PORT_Assert(0) assertion in selfserv before the NSS_Shutdown, and it was never hit during all.sh :-( This is a serious problem in our QA .

So, it may be that the leak you fixed is for both client and server-side, but we are only able to detect it on the client side, and the only intermittently .

Re: 2), I'm not sure the order of the tests is relevant. AFAIK, the SSL tests proceed even if a previous one failed. And I have my Solaris machine set to give different core file names (based on executable name and pid), so if more than one test crashed during a single run of all.sh, I would get multiple core files with different names.

Comment 6

12 years ago
I know there is a leak in ECDHE Ephemeral on the server side, which this patch corrects.

After your comments, I think it's most likely that we are dealing with a new leak in this case. The client side does not use the ecdheEphermal member of the ssl socket. It just generates a temporary key on the fly, uses it, then releases it.


Am I correct in assuming that this is a stress related bug (that is tstclnt does not fail), and in addition, this is in intermittent bug (that is it doesn't fail 100% of the time)? If that is the case we are likely looking at a race in the client side of the ECC code.

One other question, are we running other ECC ciphers in the stress scenarios other then C004 and C00E, or can we conclude that the AES_128 interaction is also necessary to trip this bug?

bob
(Assignee)

Comment 7

12 years ago
Bob,

Yes, you are correct, tstclnt does not fail, only strsclnt, and then far less than 100% of the time, it is very intermittent. I got this error about 60 times out of 1400 runs, so it's in 4% of all.sh runs. But each run does many operations of course, so the overall rate of error is much less.

It turns out we do stress test some ECDHE with two cipher suites after all :

- C009 ECDHE-ECDSA AES 128 CBC with SHA

- C013 ECDHE-RSA AES 128 CBC with SHA

We can't conclude anything about AES 128 because each of the 5 ECC stress test uses it. I would say the evidence points to a client-side problem with ECDH so far.
(Assignee)

Comment 8

12 years ago
Taking bug.
Bob, I reviewed the ECDH SSL code for client key exchange. Couldn't find anything wrong with it WRT thread safety. I think the bug is somewhere else, maybe in pk11wrap or softoken.
Assignee: wtchang → julien.pierre.bugs
(Assignee)

Comment 9

12 years ago
Comment on attachment 215902 [details] [diff] [review]
CHECKED IN:Free ECDHE Ephemeral key. Fixes server-side leak.

Using my work in progress in bug 331413, I determined that indeed a reference leak existed in selfserv - I got 8 selfserv core files for each all.sh run, despite all the tests passing.
This patch fixes the problem, and selfserv no longer asserts.
Attachment #215902 - Flags: review?(julien.pierre.bugs) → review+
(Assignee)

Updated

12 years ago
Summary: reference leak results in assertion in SECMOD_Shutdown in strsclnt → intermittent reference leak in strsclnt with ECDH ciphers C004 and C00E

Updated

12 years ago
Attachment #215902 - Flags: superreview?(alexei.volkov.bugs) → superreview+
(Assignee)

Comment 10

12 years ago
Comment on attachment 215902 [details] [diff] [review]
CHECKED IN:Free ECDHE Ephemeral key. Fixes server-side leak.

Bob,

Could you please check in this patch to both the tip and branch ? Thanks.
(Assignee)

Updated

12 years ago
Attachment #215902 - Attachment description: Free ECDHE Ephemeral key. → Free ECDHE Ephemeral key. Fixes server-side leak.

Comment 11

12 years ago
above patch checked in:

tip:
jordan.sfbay.redhat.com(1692) cvs commit sslsock.c
Checking in sslsock.c;
/cvsroot/mozilla/security/nss/lib/ssl/sslsock.c,v  <--  sslsock.c
new revision: 1.47; previous revision: 1.46
done

3.11 branch:
jordan.sfbay.redhat.com(1710) cvs commit sslsock.c
Checking in sslsock.c;
/cvsroot/mozilla/security/nss/lib/ssl/sslsock.c,v  <--  sslsock.c
new revision: 1.44.2.2; previous revision: 1.44.2.1
done

Updated

12 years ago
Attachment #215902 - Attachment description: Free ECDHE Ephemeral key. Fixes server-side leak. → CHECKED IN:Free ECDHE Ephemeral key. Fixes server-side leak.
QA Contact: jason.m.reid → libraries
(Assignee)

Comment 12

12 years ago
*** Bug 333081 has been marked as a duplicate of this bug. ***
(Assignee)

Comment 13

12 years ago
I created a test case that gives me one core file every minute when running in a loop doing 10 connections in strsclnt.

I used our PCKS#11 module logger to try to find out what's wrong. It took a lot longer to get strsclnt to dump core, but it still occurred.

Here is the report in the case of the problem :

Function                     # Calls         Time         Avg.     % Time

C_Initialize                       1         47ms   47380.00us      3.34%
C_GetInfo                          1          0 z       0.00us      0.00%
C_GetSlotList                      2          0 z       0.00us      0.00%
C_GetSlotInfo                      2          0 z       0.00us      0.00%
C_GetTokenInfo                     2         60us      30.00us      0.00%
C_GetMechanismList                 4         20us       5.00us      0.00%
C_GetMechanismInfo                10         10us       1.00us      0.00%
C_OpenSession                     51        300us       5.88us      0.02%
C_CloseSession                    49        590us      12.04us      0.04%
C_CloseAllSessions                 1        200us     200.00us      0.01%
C_GetSessionInfo                   1          0 z       0.00us      0.00%
C_CreateObject                    10       1290us     129.00us      0.09%
C_DestroyObject                   50        590us      11.80us      0.04%
C_GetAttributeValue              164         10ms      58.54us      0.68%
C_SetAttributeValue               10         60us       6.00us      0.00%
C_FindObjectsInit                 76       2810us      36.97us      0.20%
C_FindObjects                     76        540us       7.11us      0.04%
C_FindObjectsFinal                76        120us       1.58us      0.01%
C_DigestInit                      34        350us      10.29us      0.02%
C_DigestUpdate                    34        200us       5.88us      0.01%
C_DigestFinal                     34        110us       3.24us      0.01%
C_VerifyInit                      10         70us       7.00us      0.00%
C_Verify                          10        969ms   96920.00us     68.37%
C_GenerateKeyPair                 10        132ms   13237.00us      9.34%
C_DeriveKey                       20        252ms   12583.00us     17.75%
C_GenerateRandom                  10         70us       7.00us      0.00%

                   Totals        748       1418ms


Maximum number of concurrent open sessions: 18

Here is the report for a good case :

Function                     # Calls         Time         Avg.     % Time

C_Initialize                       1         50ms   49500.00us      3.13%
C_Finalize                         1       1490us    1490.00us      0.09%
C_GetInfo                          1          0 z       0.00us      0.00%
C_GetSlotList                      2          0 z       0.00us      0.00%
C_GetSlotInfo                      2          0 z       0.00us      0.00%
C_GetTokenInfo                     2         60us      30.00us      0.00%
C_GetMechanismList                 4          0 z       0.00us      0.00%
C_GetMechanismInfo                10         20us       2.00us      0.00%
C_OpenSession                     51         74ms    1448.82us      4.67%
C_CloseSession                    49        220us       4.49us      0.01%
C_CloseAllSessions                 2        490us     245.00us      0.03%
C_GetSessionInfo                   1          0 z       0.00us      0.00%
C_CreateObject                    10       1130us     113.00us      0.07%
C_DestroyObject                   50       8210us     164.20us      0.52%
C_GetAttributeValue              154       1390us       9.03us      0.09%
C_SetAttributeValue               10         60us       6.00us      0.00%
C_FindObjectsInit                 72         37ms     517.92us      2.36%
C_FindObjects                     72        890us      12.36us      0.06%
C_FindObjectsFinal                72         20us       0.28us      0.00%
C_DigestInit                      32        180us       5.62us      0.01%
C_DigestUpdate                    32        160us       5.00us      0.01%
C_DigestFinal                     32        100us       3.12us      0.01%
C_VerifyInit                      10         90us       9.00us      0.01%
C_Verify                          10        919ms   91918.00us     58.12%
C_GenerateKeyPair                 10        124ms   12381.00us      7.83%
C_DeriveKey                       20        363ms   18162.50us     22.97%
C_GenerateRandom                  10         80us       8.00us      0.01%

                   Totals        722       1582ms


Maximum number of concurrent open sessions: 18

As you can observe, C_CloseAllSessions is only called once, as if there was only one slot in the program. I find that odd.

I should add that in all my runs, good or bad, not a single call PKCS#11 call returned anything other than 0x0 (except maybe C_Finalize, which return isn't logged).
(Assignee)

Updated

12 years ago
Depends on: 225525

Comment 14

12 years ago
It appears the 20060417 securitytip run hit this problem.

Build: /share/builds/mccrel3/security/securitytip/builds/20060417.1/wozzeck_Solaris8
biarritz.4 failure: Stress TLS ECDH-ECDSA AES 128 CBC with SHA (no reuse)

output.log fragment:
ssl.sh: Stress TLS  ECDH-ECDSA  AES 128 CBC with SHA (no reuse) ----
selfserv -D -p 8444 -d ../ext_server -n biarritz.red.iplanet.com -B -s \
         -e biarritz.red.iplanet.com-ec -w nss -c :C004 -i ../tests_pid.27096  &selfserv started at Mon Apr 17 03:44:54 PDT 2006
tstclnt -p 8444 -h biarritz.red.iplanet.com  -q \
        -d ../ext_client < /share/builds/mccrel3/security/securitytip/builds/20060417.1/wozzeck_Solaris8/mozilla/security/nss/tests/ssl/sslreq.dat
strsclnt -q -p 8444 -d ../ext_client  -w nss -2 -c 100 -C :C004 -N \
          biarritz.red.iplanet.com
strsclnt started at Mon Apr 17 03:44:54 PDT 2006
strsclnt: -- SSL: Server Certificate Validated.
[ Seven lines identical to above deleted. ]
strsclnt: 0 cache hits; 8 cache misses, 0 cache not reusable
strsclnt: -- SSL: Server Certificate Validated.
[ Ninety-one lines identical to above deleted. ]
strsclnt: 0 cache hits; 100 cache misses, 0 cache not reusable
strsclnt: NoReuse - 100 server certificates tested.
strsclnt: NSS_Shutdown() failed.
strsclnt completed at Mon Apr 17 03:44:58 PDT 2006
ssl.sh: Stress TLS  ECDH-ECDSA  AES 128 CBC with SHA (no reuse) produced a returncode of 1, expected is 0.  FAILED
selfserv: normal termination
(Assignee)

Comment 15

12 years ago
This bug did not make 3.11.1 unfortunately.
Target Milestone: 3.11.1 → 3.11.2
According to Slavomir, intermittent crashes persist.
(Assignee)

Comment 17

12 years ago
Not a regression. Changing target to 3.11.3 .
Target Milestone: 3.11.2 → 3.11.3
Julien, since this code is a new feature of 3.11.1, and the leak did not 
exist in older releases, I don't see how you can claim it's not a regression.
It's a bug newly introduced in the immediately preceeding (and most recent)
release.
(Assignee)

Comment 19

12 years ago
Nelson,

The race condition bug at stake here is in the Stan cert code that has been there since NSS 3.4. This is why I'm saying this bug is not a regression.

It only happens to be the case that we witnessed this race condition in action with ECC cipher suites, but the bug has nothing to do with ECC at all and could happen with any cipher suite. The test case I created to reproduce the problem doesn't involve doing ECC or even SSL - all it does is run a modified copy of SSL_AuthCertificate cert handler multithreaded. I will attach the test case shortly. I'm not saying this is not a high priority bug for us to fix; only that it isn't a recent regression.
(Assignee)

Updated

12 years ago
OS: SunOS → All
Hardware: PC → All
Summary: intermittent reference leak in strsclnt with ECDH ciphers C004 and C00E → intermittent reference leak in strsclnt
Version: 3.11 → 3.4
(Assignee)

Comment 20

12 years ago
Created attachment 225364 [details] [diff] [review]
test case for this bug

This is a test program that is basically a modified copy of SSL_AuthCertificate from libssl .
The program takes 3 arguments - duration in seconds, number of threads, and the name of a file containing a DER cert. Any cert can be used.
Run this program with 1 thread and it will never fail.

Run it with at least 2 threads, and it will assert on line 114 because __CERT_NewTempCertificate failed in one of the threads and returned NULL.
(Assignee)

Comment 21

12 years ago
*** Bug 299070 has been marked as a duplicate of this bug. ***
(Assignee)

Comment 22

12 years ago
This is the test case I used in comment 13 .
It involves running selfserv as follows :

[jp96085@monstre]/export/home/nss/tip/mozilla/dist/SunOS5.10_i86pc_64_DBG.OBJ/bin 773 % more serve
#!/bin/tcsh
./selfserv -D -p 8443 -d ../../../tests_results/security/monstre.1/server -n monstre.red.iplanet.com -e monstre.red.iplanet.com-ecmixed -w nss -c :C00E -t 100

And strsclnt like this :

[jp96085@monstre]/export/home/nss/tip/mozilla/dist/SunOS5.10_i86pc_64_DBG.OBJ/bin 772 % more cl
#!/bin/tcsh
while (1)
./strsclnt -N -p 8443 -d ../../../tests_results/security/monstre.1/client -C :C00E -t 100 -c 10 monstre.red.iplanet.com
end

I run both of these on my Solaris Opteron box. I get about 1 to 2 core files per minute with the above, which all result from the reference leak assertion.

The test case in attachment 225364 [details] [diff] [review] does not reproduce the reference leak, although I believe that it exercises the cert code that has the bug, but due to timing the reference leak does not show with that test, even though the __CERT_NewTempCertificate call fails.
(Assignee)

Comment 23

12 years ago
If I run the client as follows :

#!/bin/tcsh
while (1)
./strsclnt -N -p 8443 -d ../../../tests_results/security/monstre.1/client -C :C00E -t 100 -o -o -c 10 monstre.red.iplanet.com
end

I still get the core files, with a slightly higher frequency, even.
The addition -o -o arguments on strsclnt cause it not to call CERT_VerifyCert on the server cert chain. But libssl still calls __CERT_NewTempCertificate in strsclnt when it receives the server cert chain, even though it ends up not getting verified. I believe the race that causes the reference leak is from the Stan code in __CERT_NewTempCertificate, primarily the code in bug 225525 .

I will try to prove this theory by replacing __CERT_NewTempCertificate with CERT_DecodeDERCertificate in libssl, thus eliminating the calls to the Stan code.
(Assignee)

Comment 24

12 years ago
Created attachment 225474 [details] [diff] [review]
just decode the cert in libssl, don't add it to the Stan cert cache

I recompiled libssl with this patch, as an experiment, and ran the test case from comment 23 . This means I'm doing SSL with zero stan code. The library decodes the cert only.

I ran the test case for 10 minutes, and haven't gotten a single core file, ie. no reference leak. I was getting 1 to 2 cores per minute without this patch. This is proof that the reference leak was caused by the call to CERT_NewTempCertificate.
(Assignee)

Comment 25

12 years ago
I backed out my libssl change so I could reproduce the ref leaks again, and immediately could.
Then, I added attachment 225382 [details] [diff] [review] to my tree. That is the fix for bug 341323, which is a race condition in __CERT_NewTempCertificate.
I then reran my test from comment 23. I still got core files, but the frequency seems to have diminished slightly - much closer to 1 core per minute than 2. It is difficult to conclude from that data whether bug 341323 plays a part in this or not. But since that bug is a race in the function that we know caused the reference leak, I'm adding 341323 as a dependency for this bug.
Depends on: 341323
Summary: intermittent reference leak in strsclnt → intermittent reference leak in strsclnt caused by importing temp cert from server
(Assignee)

Updated

12 years ago
Summary: intermittent reference leak in strsclnt caused by importing temp cert from server → intermittent reference leak in strsclnt caused by race in importing temp cert from server
(Assignee)

Comment 26

12 years ago
We are seeing this same reference leak problem in non-ECC test cases, doing client authentication. The previous test case I posted in comment #22 required using the "mixed" certs which aren't currently created, pending resolution of bug 322222 .

The following test case reproduces the problem with SSL3 and RSA :

[jp96085@monstre]/export/home/nss/tip/mozilla/dist/SunOS5.10_i86pc_DBG.OBJ/bin 154 % more cl_331279
#!/bin/tcsh
while (1)
strsclnt -o -o -q -p 8443 -d ../../../tests_results/security/monstre.1/ext_client -B -s -w nss -c 100 -C c -T -N -n ExtendedSSLUser \
          monstre.red.iplanet.com
end

[jp96085@monstre]/export/home/nss/tip/mozilla/dist/SunOS5.10_i86pc_DBG.OBJ/bin 155 % more serve_331279
#!/bin/tcsh
./selfserv -D -p 8443 -d ../../../tests_results/security/monstre.1/ext_server -n monstre.red.iplanet.com  \
         -w nss -r -r

I get about 1 to 2 cores of strsclnt per minute with this test case if NSS_STRICT_SHUTDOWN is set .

I have determined that this problem never happens with just the fix for bug 225525 . So, I'm removing bug 341323 from the dependencies. 225525 and this bug are actually dupliactes, so I'm marking this as a dupe of 225525 .


*** This bug has been marked as a duplicate of 225525 ***
Status: NEW → RESOLVED
Last Resolved: 12 years ago
No longer depends on: 341323
Resolution: --- → DUPLICATE

Comment 27

12 years ago
NISCC tests crashed today (20060911.1) with core file. 

bash-2.00$ /usr/dist/pkgs/sunstudio_i386,v11.0/SUNWspro/prod/bin/dbx strsclnt core.030641
For information about new features see `help changes'
To remove this message, put `dbxenv suppress_startup_message 7.5' in your .dbxrc
Reading strsclnt
core file header read successfully
Reading ld.so.1
Reading libssl3.so
Reading libsmime3.so
Reading libnss3.so
dbx: internal warning: Object file name not available before reading LO syms.
dbx: internal warning: Object file name not available before reading LO syms.
dbx: internal warning: Object file name not available before reading LO syms.
dbx: internal warning: Object file name not available before reading LO syms.
dbx: internal warning: Loadobj::textbound_add_loff_to_objfile_entry
duplicate textbound symbol found: loff=77288,size=0,of=(objname not available yet)
Overlaps: loff=77288,size=5504,of=(objname not available yet)

Reading libplc4.so
Reading libplds4.so
Reading libnspr4.so
Reading libthread.so.1
Reading libnsl.so.1
Reading libsocket.so.1
Reading librt.so.1
Reading libdl.so.1
Reading libc.so.1
Reading libsoftokn3.so
Reading libpthread.so.1
Reading libaio.so.1
Reading libmd5.so.1
Reading libmp.so.2
Reading libscf.so.1
Reading libdoor.so.1
Reading libuutil.so.1
t@null (l@1) program terminated by signal ABRT (Abort)
0xd0c2e875: __lwp_kill+0x0015:  jae      __lwp_kill+0x23        [ 0xd0c2e883, .+0xe ]
Current function is PR_Assert
dbx: warning: can't find file "/niscc/hacked/mozilla/nsprpub/SunOS5.10_i86pc_DBG.OBJ/pr/src/io/../../../../pr/src/io/prlog.c"
dbx: warning: see `help finding-files'
(dbx) where                                                                  
  [1] __lwp_kill(0x1, 0x6), at 0xd0c2e875 
  [2] _thr_kill(0x1, 0x6), at 0xd0c2b71b 
  [3] raise(0x6), at 0xd0bdadbb 
  [4] abort(0xd0f066d0, 0xd0d738a9, 0x8047640, 0xd0e829e9, 0xd0ef0f68, 0xd0ef0f88), at 0xd0bbe909 
=>[5] PR_Assert(s = 0xd0ef0f68 "secmod_PrivateModuleCount == 0", file = 0xd0ef0f88 "pk11util.c", ln = 119), line 538 in "prlog.c"
  [6] SECMOD_Shutdown(), line 119 in "pk11util.c"
  [7] NSS_Shutdown(), line 559 in "nssinit.c"
  [8] main(argc = 16, argv = 0x8047700), line 1342 in "strsclnt.c"

This stack looks like stack from bug 299070 which is marked as duplicate of this bug. I'm reopening this one, please check if it's really duplicate, and if it's not then close this one and reopen 299070. 
Status: RESOLVED → REOPENED
Resolution: DUPLICATE → ---
(Assignee)

Comment 28

12 years ago
Slavo,

The NISCC tests use a special hacked version of NSS built with NISCC_TEST=1 . This build was done once, a while ago within the last year. The hacked version of NSS isn't getting rebuilt at all currently. So, any fixes that are being made on the branch or tip, including this fixes, aren't making it into the hacked build. This needs to be dealt with by adding an extra nightly build for the NISCC version so that it picks up the fixes, and modifying the NISCC test scripts to use it (actually, no script modification may be necessary, it's probably just an environment variable to point to the right build).
Because the intended target milestone of 3.11.3 for this bug really has the fix, I'm closing this bug again as FIXED.
Status: REOPENED → RESOLVED
Last Resolved: 12 years ago12 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.