Closed Bug 125149 Opened 23 years ago Closed 23 years ago

selfserv fails with "MD5 digest function failed"

Categories

(NSS :: Libraries, defect, P1)

defect

Tracking

(Not tracked)

VERIFIED FIXED

People

(Reporter: wtc, Assigned: rrelyea)

Details

Attachments

(2 files, 3 obsolete files)

I checked out the tip of NSS this morning and did a debug build
on Windows 2000.

I am running selfserv (modified to add a PR_SetConcurrency(4) call)
on a 4-CPU Windows 2000 box with this command line:
selfserv -n Server-Cert -p 8880 -w enterprise -t 100 -d . -l -m -D

I am using the cert and key databases from web server 6.0.

I am running the web server 6.0 httptest client (optimized build,
with NSS 3.3.1) on a Solaris box with this command line:
httptest -h area51:8880 -s -H 1 -g COMMON -g SSL -L 5 -p 128 -x "url51" -e 3600 -P

(area51 is the host name of the Windows 2000 box running selfserv.)

After 2-3 hours of smooth running, selfserv suddenly started to
emit a lot of error messages:

selfserv: HDX PR_Read returned error -12215:
MD5 digest function failed.
selfserv: HDX PR_Read returned error -12215:
MD5 digest function failed.
[same error message repeated]

I tried to use a browser to talk to the selfserv [https://area51:8880]
and did not get a response.
Status: NEW → ASSIGNED
Priority: -- → P1
Target Milestone: --- → 3.4
Note that this particular error means that the PK11 call to compute 
the MD5 hash actaully failed.  This error is not saying that the 
value computed was not the one expected, but rather than no value 
was computed.
Assigned the bug to Ian.

Kirk, you said you've seen this error on Solaris before.
Could you help Ian reproduce it?

I've only seen it once on Windows 2000 after running
selfserv for 2-3 hours.  That kind of frequency is difficult
to debug.  If it's much easier to reproduce this error on
Solaris, we should debug it on Solaris.
Assignee: wtc → ian.mcgreer
Status: ASSIGNED → NEW
Tip build from yesterday.
Built and run on soupnazi:
kirke@soupnazi[3] uname -a
SunOS soupnazi 5.8 Generic sun4u sparc SUNW,Ultra-4
kirke@soupnazi[4] 

Selfserv runs for half hour or so hitting from 8 client machines.
100 threads on server and each of the clients.

513.89 ops/second, 100 threads  
 1 0 0 2324848 569568 0   0  0  0  0  0  0  0  0  0  1 4033 14108 5684 59 23 18
 2 0 0 2324840 569560 0   0  0  0  0  0  0  0  0  0  0 4171 14339 5778 62 22 16
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr f0 m0 m1 s0   in   sy   cs us sy id
 3 0 0 2324840 569560 0   0  0  0  0  0  0  0  0  0  0 4202 14470 5694 59 27 14
 2 0 0 2324832 569544 0   0  0  0  0  0  0  0  0  0  0 4291 14858 5955 61 23 16
selfserv: HDX PR_Read returned error -8152:
The key does not support the requested operation.
selfserv: HDX PR_Read returned error -12212:
Failure to create Symmetric Key context.
selfserv: HDX PR_Read returned error -12215:
MD5 digest function failed.
selfserv: HDX PR_Read returned error -12215:
MD5 digest function failed.

Note, this is with my own hacks to NSPR for the zone allocator.
I'll try using the standard NSPR with a DBG build under the debugger.
If that doesn't yield a trace, I'll rebuild OPT and run that under dbx.
I'll try building and running 
Ok, the DBG build failed under dbx in about an hour:

204.46 ops/second, 100 threads
204.23 ops/second, 100 threads
204.83 ops/second, 100 threads
203.57 ops/second, 100 threads
selfserv: HDX PR_Read returned error -12212:
Failure to create Symmetric Key context.
selfserv: HDX PR_Read returned error -12207:
Client failed to generate session keys for SSL session.
selfserv: HDX PR_Read returned error -12201:
Received incorrect handshakes hash values from peer.
selfserv: HDX PR_Read returned error -8190:
security library: received bad data.
selfserv: HDX PR_Read returned error -12215:

Now we see the above repeatedly on the selfserv side.
The strclnt's are all reading EOF:

strsclnt: PR_Write returned error -5938:
Encountered end of file.
strsclnt: PR_Write returned error -5938:
Encountered end of file.

Here's what I could get dbx to tell us about the selfserv side:

Current function is errWarn
  227               funcString, perr, errString);
(/usr/suntools/internal/SUNWspro/bin/../WS6U1/bin/sparcv9/dbx) where
current thread: t@41
  [1] _libc_write(0x2, 0x4d95c, 0xb, 0xff170690, 0xfef3a408, 0xfef3a428), at
0xfef19e98
  [2] _fwrite_unlocked(0x0, 0xfef39b84, 0xb, 0xfef3a274, 0x4d95c, 0x1), at
0xfef0e120
  [3] _dowrite(0x4d95c, 0xb, 0x4d95c, 0xfdf8ee00, 0xfef3a274, 0xff00), at
0xfeeff65c
  [4] _doprnt(0x0, 0x4d967, 0x0, 0xfef39731, 0x4d95c, 0x338e0), at 0xfef01d44
  [5] _fprintf(0xfef3a274, 0x338d4, 0xfef3d99c, 0xfef39c78, 0x361fc,
0xfdf91d78), at 0xfef02d60
=>[6] errWarn(funcString = 0x4d95c "HDX PR_Read"), line 227 in "selfserv.c"
  [7] handle_connection(tcp_sock = 0x259c38, model_sock = 0x58558, requestCert =
0), line 836 in "selfserv.c"
  [8] jobLoop(a = (nil), b = (nil), c = 0), line 455 in "selfserv.c"
  [9] thread_wrapper(arg = 0x90998), line 423 in "selfserv.c"
  [10] _pt_root(arg = 0x929d8), line 214 in "ptthread.c"
(/usr/suntools/internal/SUNWspro/bin/../WS6U1/bin/sparcv9/dbx) threads
      t@1  a l@57 ?()   sleep on 0x59e98        in _lwp_sema_wait()
      t@2  b l@2  ?()   running                 in __signotifywait()
      t@3  b l@3  ?()   running                 in _lwp_sema_wait()
      t@4         ?()   sleep on 0xff172a08     in _reap_wait()

     t@40  a l@95 _pt_root()    sleep on 0xfef39c90     in _lwp_sema_wait()
*>   t@41  a l@78 _pt_root()    signal SIGINT   in _libc_write()
     t@42  a l@92 _pt_root()    sleep on 0xfef39c90     in _lwp_sema_wait()

     t@106  b l@6  _co_timerset()        running                 in
_lwp_sema_wait()

(/usr/suntools/internal/SUNWspro/bin/../WS6U1/bin/sparcv9/dbx) thread t@41
t@41 (l@78) stopped in _libc_write at 0xfef19e98
0xfef19e98: _libc_write+0x0008:	ta      0x8
(/usr/suntools/internal/SUNWspro/bin/../WS6U1/bin/sparcv9/dbx) where
current thread: t@41
  [1] _libc_write(0x2, 0x4d95c, 0xb, 0xff170690, 0xfef3a408, 0xfef3a428), at 0xf
ef19e98
  [2] _fwrite_unlocked(0x0, 0xfef39b84, 0xb, 0xfef3a274, 0x4d95c, 0x1), at 0xfef
0e120
  [3] _dowrite(0x4d95c, 0xb, 0x4d95c, 0xfdf8ee00, 0xfef3a274, 0xff00), at 0xfeef
f65c
  [4] _doprnt(0x0, 0x4d967, 0x0, 0xfef39731, 0x4d95c, 0x338e0), at 0xfef01d44
  [5] _fprintf(0xfef3a274, 0x338d4, 0xfef3d99c, 0xfef39c78, 0x361fc, 0xfdf91d78)
, at 0xfef02d60
=>[6] errWarn(funcString = 0x4d95c "HDX PR_Read"), line 227 in "selfserv.c"
  [7] handle_connection(tcp_sock = 0x259c38, model_sock = 0x58558, requestCert =
 0), line 836 in "selfserv.c"
  [8] jobLoop(a = (nil), b = (nil), c = 0), line 455 in "selfserv.c"
  [9] thread_wrapper(arg = 0x90998), line 423 in "selfserv.c"
  [10] _pt_root(arg = 0x929d8), line 214 in "ptthread.c"
(/usr/suntools/internal/SUNWspro/bin/../WS6U1/bin/sparcv9/dbx) 

(/usr/suntools/internal/SUNWspro/bin/../WS6U1/bin/sparcv9/dbx) thread t@106
t@106 (l@6) stopped in _lwp_sema_wait at 0xfef198bc
0xfef198bc: _lwp_sema_wait+0x0008:	ta      0x8
(/usr/suntools/internal/SUNWspro/bin/../WS6U1/bin/sparcv9/dbx) where
current thread: t@106
=>[1] _lwp_sema_wait(0xff16fa30, 0xff16fa20, 0x0, 0xfd791f10, 0xe, 0xfe909ce0),
at 0xfef198bc
  [2] _co_timerset(0xff16ed30, 0xff16e000, 0x1, 0x3, 0xff16e000, 0x0), at
0xff148a6c
(/usr/suntools/internal/SUNWspro/bin/../WS6U1/bin/sparcv9/dbx) 

Note, I'm hitting with pure NSS 3.3.1 built strsclnts.
Hope this helps.
FYI, my week-end stress test of NSS 3.4 with NES6 also showed this :

[18/Feb/2002:05:15:28] failure ( 2180): Error receiving connection (SSL_ERROR_MD
5_DIGEST_FAILURE - MD5 digest function failed.)

There were a total of 2026103 such MD5 errors in the server's errors log over 
the week-end. This was with the server running under Windows 2000. The test used 
client authentication. I was hitting the server with an NSS 3.3.2 based client.
Overnight on 2/19, I ran a stress test on Solaris. Both the client and server
were using NSS 3.4 this time. I was not using client auth, just regular full SSL
handshakes.
The server logged 148906 MD5 digest errors during the test, as follows :
[19/Feb/2002:08:10:15] failure (12079): Error receiving connection
(SSL_ERROR_MD5_DIGEST_FAILURE - MD5 digest function failed.)

According to the log, there were also 936703 successful HTTP
requests/connections/full SSL handshakes (1:1 mapping in this test) to the server.

I now have an NES server on Solaris running 3.4 in the state where it's doing
the MD5 error on every connection. It seems to have leaked memory (about 20 MB
in a 12 hour run) but it's still up and running debug bits.
After an extensive debug session with Nelson and Wan-Teh, we think there is some
corruption occurring in some of the hash tables, possibly the slot hash table.
We traced down the MD5 digest operation failure to a problem looking up the
PKCS#11 session in the slot. This was puzzling because the session had just been
created. It appears there is a mismatch between the slot in which the PKCS#11
session is getting created and the one in which it is being looked up.
The nscSlotHashTable in lib/softoken/pkcs11.c is not protected
by a lock.

This is okay if we only add entries to nscSlotHashTable during
NSS initialization.  Is this true, Bob?

Assuming after NSS has been initialized, nscSlotHashTable is
static (not changing), we can do lookups in it without a lock.
However, we are using PL_HashTableLookup(), which is problematic
because this function modifies the hashtable.  We should use
PL_HashTableLookupConst().  This patch does that.
Attachment #70420 - Flags: review+
FYI, here is where the error actually happens deep down. The failure is looking
up a dsession from a handle within the NSC_DigestInit of the softoken. I have
printed the slot information below and the stack at the time the failure happens.

(dbx) where
current thread: t@145
=>[1] pk11_SessionFromHandle(handle = 16926137U), line 2438 in "pkcs11u.c"
  [2] NSC_DigestInit(hSession = 16926137U, pMechanism = 0xfbb51570), line 1024
in "pkcs11c.c"
  [3] pk11_context_init(context = 0x27824f0, mech_info = 0xfbb51570), line 3382
in "pk11skey.c"
  [4] pk11_CreateNewContextInSlot(type = 528U, slot = 0x92ad8, operation =
2164260864U, symKey = (nil), param = 0xfbb515e4), line 3467 in "pk11skey.c"
  [5] PK11_CreateDigestContext(hashAlg = SEC_OID_MD5), line 3575 in "pk11skey.c"
  [6] ssl3_InitState(ss = 0x27700e0), line 7645 in "ssl3con.c"
  [7] ssl3_HandleV2ClientHello(ss = 0x27700e0, buffer = 0x263dce8 "^A^C^A",
length = 70), line 5356 in "ssl3con.c"
  [8] ssl2_HandleClientHelloMessage(ss = 0x27700e0), line 3556 in "sslcon.c"
  [9] ssl_Do1stHandshake(ss = 0x27700e0), line 156 in "sslsecur.c"
  [10] ssl_SecureRecv(ss = 0x27700e0, buf = 0xc47d28 "GET /index.html
HTTP/1.0\n\n", len = 8191, flags = 0), line 1038 in "sslsecur.c"
  [11] ssl_Recv(fd = 0x8129a8, buf = 0xc47d28, len = 8191, flags = 0, timeout =
3000000U), line 1191 in "sslsock.c"
  [12] PR_Recv(fd = 0x8129a8, buf = 0xc47d28, amount = 8191, flags = 0, timeout
= 3000000U), line 215 in "priometh.c"
  [13] DaemonSession::GetConnection(this = 0xc3a4b8), line 400 in
"daemonsession.cpp"
  [14] DaemonSession::run(this = 0xc3a4b8), line 462 in "daemonsession.cpp"
  [15] Thread::run_(this = 0xc3a4b8), line 234 in "Thread.cpp"
  [16] ThreadMain(thisObject = 0xc3a4b8), line 226 in "Thread.cpp"
  [17] _pt_root(arg = 0x5761c8), line 214 in "ptthread.c"(dbx) p *slot
*slot = {
    slotID          = 2U
    slotLock        = 0x81840
    sessionLock     = (0x818c0, 0x81920, 0x81980, 0x819e0, 0x81a40, 0x81aa0,
0x81b00, 0x81b60, 0x81bc0, 0x81c20, 0x81c80, 0x81ce0, 0x81d40, 0x81da0, 0x81e00,
0x81e60, 0x81ec0, 0x81f20, 0x81f80, 0x81fe0, 0x82040, 0x820a0, 0x82100, 0x82160,
0x821c0, 0x82220, 0x82280, 0x822e0, 0x82340, 0x823a0, 0x82400, 0x82460, 0x824c0,
0x82520, 0x82580, 0x825e0, 0x82640, 0x826a0, 0x82700, 0x82760, 0x827c0, 0x82820,
0x82880, 0x828e0, 0x82940, 0x829a0, 0x82a00, 0x82a60, 0x82ac0, 0x82b20, 0x82b80,
0x82be0, 0x82c40, 0x82ca0, 0x82d00, 0x82d60, 0x82dc0, 0x82e20, 0x82e80, 0x82ee0,
0x82f40, 0x82fa0, 0x83000, 0x83060, 0x830c0, 0x83120, 0x83180, 0x831e0, 0x83240,
0x832a0, 0x83300, 0x83360, 0x833c0, 0x83420, 0x83480, 0x834e0, 0x83540, 0x835a0,
0x83600, 0x83660, 0x836c0, 0x83720, 0x83780, 0x837e0, 0x83840, 0x838c0, 0x83920,
0x83980, 0x839e0, 0x83a40, 0x83aa0, 0x83b00, 0x83b60, 0x83bc0, 0x83c20, 0x83c80,
0x83ce0, 0x83d40, 0x83da0, 0x83e00, 0x83e60, 0x83ec0, 0x83f20, 0x83f80, 0x83fe0,
0x84040, 0x840a0, 0x84100, 0x84160, 0x841c0, 0x84220, 0x84280, 0x842e0, 0x84340,
0x843a0, 0x84400, 0x84460, 0x844c0, 0x84520, 0x84580, 0x845e0, 0x84640, 0x846a0,
0x84700, 0x84760, 0x847c0, 0x84820, 0x84880, 0x848e0, 0x84940, 0x849a0, 0x84a00,
0x84a60, 0x84ac0, 0x84b20, 0x84b80, 0x84be0, 0x84c40, 0x84ca0, 0x84d00, 0x84d60,
0x84dc0, 0x84e20, 0x84e80, 0x84ee0, 0x84f40, 0x84fa0, 0x85000, 0x85060, 0x850c0,
0x85120, 0x85180, 0x851e0, 0x85240, 0x852a0, 0x85300, 0x85360, 0x853c0, 0x85420,
0x85480, 0x854e0, 0x85540, 0x855a0, 0x85600, 0x85660, 0x856c0, 0x85720, 0x85780,
0x857e0, 0x85840, 0x858c0, 0x85920, 0x85980, 0x859e0, 0x85a40, 0x85aa0, 0x85b00,
0x85b60, 0x85bc0, 0x85c20, 0x85c80, 0x85ce0, 0x85d40, 0x85da0, 0x85e00, 0x85e60,
0x85ec0, 0x85f20, 0x85f80, 0x85fe0, 0x86040, 0x860a0, 0x86100, 0x86160, 0x861c0,
0x86220, 0x86280, 0x862e0, 0x86340, 0x863a0, 0x86400, 0x86460, 0x864c0, 0x86520,
0x86580, 0x865e0, 0x86640, 0x866a0, 0x86700, 0x86760, 0x867c0, 0x86820, 0x86880,
0x868e0, 0x86940, 0x869a0, 0x86a00, 0x86a60, 0x86ac0, 0x86b20, 0x86b80, 0x86be0,
0x86c40, 0x86ca0, 0x86d00, 0x86d60, 0x86dc0, 0x86e20, 0x86e80, 0x86ee0, 0x86f40,
0x86fa0, 0x87000, 0x87060, 0x870c0, 0x87120, 0x87180, 0x871e0, 0x87240, 0x872a0,
0x87300, 0x87360, 0x873c0, 0x87420, 0x87480, 0x874e0, 0x87540, 0x875a0, 0x87600,
0x87660, 0x876c0, 0x87720, 0x87780, 0x877e0, 0x87840, 0x878c0, 0x87920, 0x87980,
0x879e0, 0x87a40, 0x87aa0, 0x87b00, 0x87b60, 0x87bc0, 0x87c20, 0x87c80, 0x87ce0,
0x87d40, 0x87da0, 0x87e00, 0x87e60, 0x87ec0, 0x87f20, 0x87f80, 0x87fe0, 0x88040,
0x880a0, 0x88100, 0x88160, 0x881c0, 0x88220, 0x88280, 0x882e0, 0x88340, 0x883a0,
0x88400, 0x88460, 0x884c0, 0x88520, 0x88580, 0x885e0, 0x88640, 0x886a0, 0x88700,
0x88760, 0x887c0, 0x88820, 0x88880, 0x888e0, 0x88940, 0x889a0, 0x88a00, 0x88a60,
0x88ac0, 0x88b20, 0x88b80, 0x88be0, 0x88c40, 0x88ca0, 0x88d00, 0x88d60, 0x88dc0,
0x88e20, 0x88e80, 0x88ee0, 0x88f40, 0x88fa0, 0x89000, 0x89060, 0x890c0, 0x89120,
0x89180, 0x891e0, 0x89240, 0x892a0, 0x89300, 0x89360, 0x893c0, 0x89420, 0x89480,
0x894e0, 0x89540, 0x895a0, 0x89600, 0x89660, 0x896c0, 0x89720, 0x89780, 0x897e0,
0x89840, 0x898c0, 0x89920, 0x89980, 0x899e0, 0x89a40, 0x89aa0, 0x89b00, 0x89b60,
0x89bc0, 0x89c20, 0x89c80, 0x89ce0, 0x89d40, 0x89da0, 0x89e00, 0x89e60, 0x89ec0,
0x89f20, 0x89f80, 0x89fe0, 0x8a040, 0x8a0a0, 0x8a100, 0x8a160, 0x8a1c0, 0x8a220,
0x8a280, 0x8a2e0, 0x8a340, 0x8a3a0, 0x8a400, 0x8a460, 0x8a4c0, 0x8a520, 0x8a580,
0x8a5e0, 0x8a640, 0x8a6a0, 0x8a700, 0x8a760, 0x8a7c0, 0x8a820, 0x8a880, 0x8a8e0,
0x8a940, 0x8a9a0, 0x8aa00, 0x8aa60, 0x8aac0, 0x8ab20, 0x8ab80, 0x8abe0, 0x8ac40,
0x8aca0, 0x8ad00, 0x8ad60, 0x8adc0, 0x8ae20, 0x8ae80, 0x8aee0, 0x8af40, 0x8afa0,
0x8b000, 0x8b060, 0x8b0c0, 0x8b120, 0x8b180, 0x8b1e0, 0x8b240, 0x8b2a0, 0x8b300,
0x8b360, 0x8b3c0, 0x8b420, 0x8b480, 0x8b4e0, 0x8b540, 0x8b5a0, 0x8b600, 0x8b660,
0x8b6c0, 0x8b720, 0x8b780, 0x8b7e0, 0x8b840, 0x8b8c0, 0x8b920, 0x8b980, 0x8b9e0,
0x8ba40, 0x8baa0, 0x8bb00, 0x8bb60, 0x8bbc0, 0x8bc20, 0x8bc80, 0x8bce0, 0x8bd40,
0x8bda0, 0x8be00, 0x8be60, 0x8bec0, 0x8bf20, 0x8bf80, 0x8bfe0, 0x8c040, 0x8c0a0,
0x8c100, 0x8c160, 0x8c1c0, 0x8c220, 0x8c280, 0x8c2e0, 0x8c340, 0x8c3a0, 0x8c400,
0x8c460, 0x8c4c0, 0x8c520, 0x8c580, 0x8c5e0, 0x8c640, 0x8c6a0, 0x8c700, 0x8c760,
0x8c7c0, 0x8c820, 0x8c880, 0x8c8e0, 0x8c940, 0x8c9a0, 0x8ca00, 0x8ca60, 0x8cac0,
 0x8cb20, 0x8cb80, 0x8cbe0, 0x8cc40, 0x8cca0, 0x8cd00, 0x8cd60, 0x8cdc0,
0x8ce20, 0x8ce80, 0x8cee0, 0x8cf40, 0x8cfa0, 0x8d000, 0x8d060, 0x8d0c0, 0x8d120,
0x8d180, 0x8d1e0, 0x8d240, 0x8d2a0, 0x8d300, 0x8d360, 0x8d3c0, 0x8d420, 0x8d480,
0x8d4e0, 0x8d540, 0x8d5a0, 0x8d600, 0x8d660, 0x8d6c0, 0x8d720, 0x8d780, 0x8d7e0,
0x8d840, 0x8d8c0, 0x8d920)
    objectLock      = 0x8d980
    password        = (nil)
    hasTokens       = 0
    isLoggedIn      = 0
    ssoLoggedIn     = 0
    needLogin       = 1
    DB_loaded       = 0
    readOnly        = 1
    certDB          = 0x32f90
    keyDB           = 0x67c40
    minimumPinLen   = 1
    sessionIDCount  = 941193
    sessionCount    = 0
    rwSessionCount  = 0
    tokenIDCount    = 1
    index           = 1
    tokenHashTable  = 0x69768
    tokObjects      = ((nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (n
il), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil))
    head            = ((nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil))
    tokDescription  = "internal                        "
    slotDescription = "NSS User Private Key and Certificate Services           
      "
}
(dbx) 



OS: Windows 2000 → All
Hardware: PC → All
Just to be clear on that last printout, here is the short code of the function
it was taken in :

/*
 * look up a session structure from a session handle
 * generate a reference to it.
 */
PK11Session *
pk11_SessionFromHandle(CK_SESSION_HANDLE handle)
{
    PK11Slot	*slot = pk11_SlotFromSessionHandle(handle);
    PK11Session *session;

    PK11_USE_THREADS(PZ_Lock(PK11_SESSION_LOCK(slot,handle));)
    pk11queue_find(session,handle,slot->head,SESSION_HASH_SIZE);
    if (session) session->refCount++;
    PK11_USE_THREADS(PZ_Unlock(PK11_SESSION_LOCK(slot,handle));)

    return (session);
}

The slot was returned by pk11_SlotFromSessionHandle . It was slot ID 2 , which
is the internal key database. Previous tracing has shown that that session was
created in slot ID 1, the generic crypto services.

So the bug could be in pk11_SlotFromSessionHandle . That code looks like :

PK11Slot *
pk11_SlotFromSessionHandle(CK_SESSION_HANDLE handle)
{
    int slotIDIndex = (handle >> 24) & 0xff;

    if (slotIDIndex >= nscSlotCount) {
	return NULL;
    }

    return pk11_SlotFromID(nscSlotList[slotIDIndex]);
}

I don't know if that's the right formula to use.

pk11_SlotFromID is as follows :

/* look up a slot structure from the ID (used to be a macro when we only
 * had two slots) */
PK11Slot *
pk11_SlotFromID(CK_SLOT_ID slotID)
{
    return (PK11Slot *)PL_HashTableLookup(nscSlotHashTable, (void *)slotID);
}

I just traced through the pk11_SlotFromSessionHandle listed above .
I printed slotIDIndex . It was 1 .

But the slot returned by this function was slot #2. So there is the problem.
This could be caused by the hash table being corrupted, or an algorithm problem
in the mechanism for looking up slots / buckets.
Sorry, my bad. The slotIDIndex gets looked up in an array. ID 1 maps to slot 2,
which is looked in the hash table. ID 0 maps to slot 1. So it is correctly
returning slot 2. The question is why the code thinks this session is from slot
2. I'll keep working on this and trace where that session got created because
I'm pretty sure it was created in slot 1.
Here is more info. At the level of PK11_CreateDigestContext, the slot returned
by PK11_GetBestSlot for MD5 looks like this :

*slot = {
    functionList    = 0xfe40be24
    module          = 0x721d8
    needTest        = 0
    isPerm          = 1
    isHW            = 0
    isInternal      = 1
    disabled        = 0
    reason          = PK11_DIS_NONE
    readOnly        = 1
    needLogin       = 0
    hasRandom       = 1
    defRWSession    = 0
    isThreadSafe    = 1
    flags           = 32771U
    session         = 1U
    sessionLock     = 0x6f2c8
    slotID          = 1U
    defaultFlags    = 2147499837U
    refCount        = 22
    refLock         = 0x6f6f0
    freeListLock    = 0x8dae8
    freeSymKeysHead = 0xb16cf0
    keyCount        = 800
    maxKeyCount     = 800
    askpw           = 0
    timeout         = 0
    authTransact    = 0
    authTime        = 0
    minPassword     = 0
    maxPassword     = 0
    series          = 1U
    wrapKey         = 0
    wrapMechanism   = 4294967295U
    refKeys         = (0)
    mechanismList   = 0x94300
    mechanismCount  = 86
    cert_array      = (nil)
    array_size      = 0
    cert_count      = 0
    serial          = "0000000000000000"
    slot_name       = "NSS Internal Cryptographic Services Version 3.4         
      "
    token_name      = "NSS Generic Crypto Services"
    hasRootCerts    = 0
    hasRootTrust    = 0
    hasRSAInfo      = 0
    RSAInfoFlags    = 0
    nssToken        = 0x8e8b8

In pk11_CreateNewContextInSlot, there is the following code snippet :

    /* initialize the critical fields of the context */
    context->operation = operation;
    context->key = symKey ? PK11_ReferenceSymKey(symKey) : NULL;
    context->slot = PK11_ReferenceSlot(slot);
    context->session = pk11_GetNewSession(slot,&context->ownSession);
    context->cx = symKey ? symKey->cx : NULL;

The slot is the same that was passed in from above (see previous comment, crypto
slot). As you can see, the slotID in the above structure is listed as being 1.

The same slot structure ends up being passed down to pk11_GetNewSession.

This is where we cross the barrier and go into the NSC_OpenSession of the softoken.

At that point, the slotID is passed to the opensession, and that slot ID is 1 .
See the stack below :

current thread: t@151
=>[1] NSC_OpenSession(slotID = 1U, flags = 4U, pApplication = 0x92ad8, Notify =
0xfe5badb0 = &pk11_notify(), phSession = 0xfba314f0), line 2673 in "pkcs11.c"
  [2] pk11_GetNewSession(slot = 0x92ad8, owner = 0x2782504), line 92 in "pk11skey.c"
  [3] pk11_CreateNewContextInSlot(type = 528U, slot = 0x92ad8, operation =
2164260864U, symKey = (nil), param = 0xfba315e4), line 3447 in "pk11skey.c"
  [4] PK11_CreateDigestContext(hashAlg = SEC_OID_MD5), line 3575 in "pk11skey.c"
  [5] ssl3_InitState(ss = 0x27700e0), line 7645 in "ssl3con.c"
  [6] ssl3_HandleV2ClientHello(ss = 0x27700e0, buffer = 0x263dce8 "^A^C^A",
length = 70), line 5356 in "ssl3con.c"
  [7] ssl2_HandleClientHelloMessage(ss = 0x27700e0), line 3556 in "sslcon.c"
  [8] ssl_Do1stHandshake(ss = 0x27700e0), line 156 in "sslsecur.c"
  [9] ssl_SecureRecv(ss = 0x27700e0, buf = 0xcb14f8 "GET /index.html
HTTP/1.0\n\n", len = 8191, flags = 0), line 1038 in "sslsecur.c"
  [10] ssl_Recv(fd = 0x449da0, buf = 0xcb14f8, len = 8191, flags = 0, timeout =
3000000U), line 1191 in "sslsock.c"
  [11] PR_Recv(fd = 0x449da0, buf = 0xcb14f8, amount = 8191, flags = 0, timeout
= 3000000U), line 215 in "priometh.c"
  [12] DaemonSession::GetConnection(this = 0xc3f118), line 400 in
"daemonsession.cpp"
  [13] DaemonSession::run(this = 0xc3f118), line 462 in "daemonsession.cpp"
  [14] Thread::run_(this = 0xc3f118), line 234 in "Thread.cpp"
  [15] ThreadMain(thisObject = 0xc3f118), line 226 in "Thread.cpp"
  [16] _pt_root(arg = 0xbe18c8), line 214 in "ptthread.c"
(dbx) 

The first thing NSC_OpenSession does is 

    slot = pk11_SlotFromID(slotID);
The slotID passed to it is 1. The slot structure returned is :

*slot = {
    slotID          = 1U
    slotLock        = 0x72008
    sessionLock     = (0x72068, 0x720c8, 0x72128, 0x72e78, 0x72ed8, 0x72f38,
0x72f98, 0x72ff8, 0x73058, 0x730b8, 0x73118, 0x73178, 0x731d8, 0x73238, 0x73298,
0x732f8, 0x73358, 0x733b8, 0x73418, 0x73478, 0x734d8, 0x73538, 0x73598, 0x735f8,
0x73658, 0x736b8, 0x73718, 0x73778, 0x737d8, 0x73838, 0x738c0, 0x73920, 0x73980,
0x739e0, 0x73a40, 0x73aa0, 0x73b00, 0x73b60, 0x73bc0, 0x73c20, 0x73c80, 0x73ce0,
0x73d40, 0x73da0, 0x73e00, 0x73e60, 0x73ec0, 0x73f20, 0x73f80, 0x73fe0, 0x74040,
0x740a0, 0x74100, 0x74160, 0x741c0, 0x74220, 0x74280, 0x742e0, 0x74340, 0x743a0,
0x74400, 0x74460, 0x744c0, 0x74520, 0x74580, 0x745e0, 0x74640, 0x746a0, 0x74700,
0x74760, 0x747c0, 0x74820, 0x74880, 0x748e0, 0x74940, 0x749a0, 0x74a00, 0x74a60,
0x74ac0, 0x74b20, 0x74b80, 0x74be0, 0x74c40, 0x74ca0, 0x74d00, 0x74d60, 0x74dc0,
0x74e20, 0x74e80, 0x74ee0, 0x74f40, 0x74fa0, 0x75000, 0x75060, 0x750c0, 0x75120,
0x75180, 0x751e0, 0x75240, 0x752a0, 0x75300, 0x75360, 0x753c0, 0x75420, 0x75480,
0x754e0, 0x75540, 0x755a0, 0x75600, 0x75660, 0x756c0, 0x75720, 0x75780, 0x757e0,
0x75840, 0x758c0, 0x75920, 0x75980, 0x759e0, 0x75a40, 0x75aa0, 0x75b00, 0x75b60,
0x75bc0, 0x75c20, 0x75c80, 0x75ce0, 0x75d40, 0x75da0, 0x75e00, 0x75e60, 0x75ec0,
0x75f20, 0x75f80, 0x75fe0, 0x76040, 0x760a0, 0x76100, 0x76160, 0x761c0, 0x76220,
0x76280, 0x762e0, 0x76340, 0x763a0, 0x76400, 0x76460, 0x764c0, 0x76520, 0x76580,
0x765e0, 0x76640, 0x766a0, 0x76700, 0x76760, 0x767c0, 0x76820, 0x76880, 0x768e0,
0x76940, 0x769a0, 0x76a00, 0x76a60, 0x76ac0, 0x76b20, 0x76b80, 0x76be0, 0x76c40,
0x76ca0, 0x76d00, 0x76d60, 0x76dc0, 0x76e20, 0x76e80, 0x76ee0, 0x76f40, 0x76fa0,
0x77000, 0x77060, 0x770c0, 0x77120, 0x77180, 0x771e0, 0x77240, 0x772a0, 0x77300,
0x77360, 0x773c0, 0x77420, 0x77480, 0x774e0, 0x77540, 0x775a0, 0x77600, 0x77660,
0x776c0, 0x77720, 0x77780, 0x777e0, 0x77840, 0x778c0, 0x77920, 0x77980, 0x779e0,
0x77a40, 0x77aa0, 0x77b00, 0x77b60, 0x77bc0, 0x77c20, 0x77c80, 0x77ce0, 0x77d40,
0x77da0, 0x77e00, 0x77e60, 0x77ec0, 0x77f20, 0x77f80, 0x77fe0, 0x78040, 0x780a0,
0x78100, 0x78160, 0x781c0, 0x78220, 0x78280, 0x782e0, 0x78340, 0x783a0, 0x78400,
0x78460, 0x784c0, 0x78520, 0x78580, 0x785e0, 0x78640, 0x786a0, 0x78700, 0x78760,
0x787c0, 0x78820, 0x78880, 0x788e0, 0x78940, 0x789a0, 0x78a00, 0x78a60, 0x78ac0,
0x78b20, 0x78b80, 0x78be0, 0x78c40, 0x78ca0, 0x78d00, 0x78d60, 0x78dc0, 0x78e20,
0x78e80, 0x78ee0, 0x78f40, 0x78fa0, 0x79000, 0x79060, 0x790c0, 0x79120, 0x79180,
0x791e0, 0x79240, 0x792a0, 0x79300, 0x79360, 0x793c0, 0x79420, 0x79480, 0x794e0,
0x79540, 0x795a0, 0x79600, 0x79660, 0x796c0, 0x79720, 0x79780, 0x797e0, 0x79840,
0x798c0, 0x79920, 0x79980, 0x799e0, 0x79a40, 0x79aa0, 0x79b00, 0x79b60, 0x79bc0,
0x79c20, 0x79c80, 0x79ce0, 0x79d40, 0x79da0, 0x79e00, 0x79e60, 0x79ec0, 0x79f20,
0x79f80, 0x79fe0, 0x7a040, 0x7a0a0, 0x7a100, 0x7a160, 0x7a1c0, 0x7a220, 0x7a280,
0x7a2e0, 0x7a340, 0x7a3a0, 0x7a400, 0x7a460, 0x7a4c0, 0x7a520, 0x7a580, 0x7a5e0,
0x7a640, 0x7a6a0, 0x7a700, 0x7a760, 0x7a7c0, 0x7a820, 0x7a880, 0x7a8e0, 0x7a940,
0x7a9a0, 0x7aa00, 0x7aa60, 0x7aac0, 0x7ab20, 0x7ab80, 0x7abe0, 0x7ac40, 0x7aca0,
0x7ad00, 0x7ad60, 0x7adc0, 0x7ae20, 0x7ae80, 0x7aee0, 0x7af40, 0x7afa0, 0x7b000,
0x7b060, 0x7b0c0, 0x7b120, 0x7b180, 0x7b1e0, 0x7b240, 0x7b2a0, 0x7b300, 0x7b360,
0x7b3c0, 0x7b420, 0x7b480, 0x7b4e0, 0x7b540, 0x7b5a0, 0x7b600, 0x7b660, 0x7b6c0,
0x7b720, 0x7b780, 0x7b7e0, 0x7b840, 0x7b8c0, 0x7b920, 0x7b980, 0x7b9e0, 0x7ba40,
0x7baa0, 0x7bb00, 0x7bb60, 0x7bbc0, 0x7bc20, 0x7bc80, 0x7bce0, 0x7bd40, 0x7bda0,
0x7be00, 0x7be60, 0x7bec0, 0x7bf20, 0x7bf80, 0x7bfe0, 0x7c040, 0x7c0a0, 0x7c100,
0x7c160, 0x7c1c0, 0x7c220, 0x7c280, 0x7c2e0, 0x7c340, 0x7c3a0, 0x7c400, 0x7c460,
0x7c4c0, 0x7c520, 0x7c580, 0x7c5e0, 0x7c640, 0x7c6a0, 0x7c700, 0x7c760, 0x7c7c0,
0x7c820, 0x7c880, 0x7c8e0, 0x7c940, 0x7c9a0, 0x7ca00, 0x7ca60, 0x7cac0, 0x7cb20,
0x7cb80, 0x7cbe0, 0x7cc40, 0x7cca0, 0x7cd00, 0x7cd60, 0x7cdc0, 0x7ce20, 0x7ce80,
0x7cee0, 0x7cf40, 0x7cfa0, 0x7d000, 0x7d060, 0x7d0c0, 0x7d120, 0x7d180, 0x7d1e0,
0x7d240, 0x7d2a0, 0x7d300, 0x7d360, 0x7d3c0, 0x7d420, 0x7d480, 0x7d4e0, 0x7d540,
0x7d5a0, 0x7d600, 0x7d660, 0x7d6c0, 0x7d720, 0x7d780, 0x7d7e0, 0x7d840, 0x7d8c0,
0x7d920, 0x7d980, 0x7d9e0, 0x7da40, 0x7daa0, 0x7db00, 0x7db60, 0x7dbc0, 0x7dc20,
0x7dc80, 0x7dce0, 0x7dd40, 0x7dda0, 0x7de00, 0x7de60, 0x7dec0, 0x7df20, 0x7df80,
0x7dfe0, 0x7e040, 0x7e0a0, 0x7e100, 0x7e160, 0x7e1c0, 0x7e220, 0x7e280, 0x7e2e0,
0x7e340, 0x7e3a0, 0x7e400, 0x7e460, 0x7e4c0, 0x7e520, 0x7e580, 0x7e5e0, 0x7e640,
0x7e6a0, 0x7e700, 0x7e760, 0x7e7c0, 0x7e820, 0x7e880, 0x7e8e0, 0x7e940, 0x7e9a0,
0x7ea00, 0x7ea60, 0x7eac0, 0x7eb20, 0x7eb80, 0x7ebe0, 0x7ec40, 0x7eca0, 0x7ed00,
0x7ed60, 0x7edc0)
    objectLock      = 0x7ee20
    password        = (nil)
    hasTokens       = 0
    isLoggedIn      = 0
    ssoLoggedIn     = 0
    needLogin       = 0
    DB_loaded       = 0
    readOnly        = 0
    certDB          = (nil)
    keyDB           = (nil)
    minimumPinLen   = 0
    sessionIDCount  = 16926142
    sessionCount    = 149261
    rwSessionCount  = 0
    tokenIDCount    = 5657229
    index           = 0
    tokenHashTable  = 0x69790
    tokObjects      = ((nil), 0x24c218, 0x247418, 0x24f208, 0x252358, 0x261bc8,
0x25b268, (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), 0x44cde8, (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (n
il), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil), (nil),
(nil), (nil), (nil), (nil), (nil), (nil))
    head            = (0x266f428, 0x2755050, 0x2759ed0, 0x273c350, 0x26e7528,
0x27029a8, 0x26fc188, 0x26d64f0, 0x26141c0, 0x276b420, 0x266e910, 0x273f4e8,
0x2786950, 0x27fe668, 0x284cf50, 0x279a7c0, 0x26febe8, 0x276e868, 0x2758af8,
0x2826020, 0x2736a88, 0x2715f98, 0x27686e8, 0x282ecd8, 0x270cdb0, 0x2736a38,
0x276e8b8, 0x266e8c0, 0x2614170, 0x2755000, 0x271c5a8, 0x2613f28, 0x27e2420,
0x27f6590, 0x26ed138, 0x277b698, 0x280ca08, 0x27d4320, 0x27d2588, 0x27976a0,
0x2759f80, 0x27977c0, 0x2819460, 0x2837268, 0x266e9c0, 0x2614348, 0x2758c08,
0x273c7e8, 0x26ea8c0, 0x27e2240, 0x2614270, 0x276b288, 0x2807060, 0x263ee70,
0x281fa78, 0x26d6478, 0x26fc050, 0x263ee10, 0x2702870, 0x276b338, 0x2614450,
0x2614500, 0x2717a80, 0x2717b30, 0x2825b78, 0x274d950, 0x26f5c30, 0x26c1ac0,
0x273c290, 0x27326f0, 0x26fa0e0, 0x26ecd40, 0x27d2708, 0x266f538, 0x27e20f8,
0x263efa8, 0x27c4368, 0x2823638, 0x2732da0, 0x28660b8, 0x261a1c0, 0x27e0400,
0x27e0528, 0x2716158, 0x2716208, 0x2732d50, 0x271ba88, 0x271bb38, 0x2732e50,
0x26ea870, 0x2710298, 0x27770b0, 0x28047a8, 0x2807208, 0x28142e8, 0x27162b8,
0x26fc100, 0x2708de0, 0x2807110, 0x266ea70, 0x266eac0, 0x277b920, 0x26d6418,
0x261a0a0, 0x27dfd00, 0x2696668, 0x2804660, 0x276fdd8, 0x27868e8, 0x2750b70,
0x2786768, 0x2786818, 0x279aa38, 0x26bab28, 0x2759560, 0x2766210, 0x2825fd0,
0x2797f60, 0x273f498, 0x26e98a0, 0x282b098, 0x27e0870, 0x26ed778, 0x2820010,
0x281ffb0, 0x2734460, 0x2766558, 0x281feb0, 0x273f360, 0x273f300, 0x2797ee8,
0x2734520, 0x28208b0, 0x2820900, 0x27e0630, 0x2856f48, 0x276ea08, 0x282adf8,
0x26fabb8, 0x26ed688, 0x278a060, 0x278a110, 0x26ba158, 0x27e55b8, 0x27de0e0,
0x27de190, 0x2776728, 0x27928a8, 0x2792958, 0x279a8d0, 0x279a980, 0x282aeb8,
0x26ba798, 0x282afc8, 0x27e0690, 0x2816a20, 0x27e0740, 0x27e07f0, 0x276b580,
0x2758d08, 0x2820c68, 0x2820d18, 0x27fe778, 0x27fe828, 0x265f1e0, 0x27df590,
0x2825dd8, 0x2789860, 0x27e2550, 0x2825ee8, 0x27559b8, 0x26fe7a0, 0x26fe850,
0x26fe900, 0x2736b38, 0x2736be8, 0x2736c98, 0x2857008, 0x28570b8, 0x28571b8,
0x276eac8, 0x276eb78, 0x276ec28, 0x276ec78, 0x28431b0, 0x2843260, 0x2843310,
0x284d060, 0x2733878, 0x26fee18, 0x276e818, 0x271bf50, 0x276e680, 0x26fed58,
0x2845960, 0x2845828, 0x271c0c0, 0x2848080, 0x284d110, 0x284d1c0, 0x26ba268,
0x26ba318, 0x280f250, 0x26726c8, 0x2672778, 0x28457c8, 0x280f190, 0x2857168,
0x26ba418, 0x26ba3c8, 0x2715e70, 0x28481f8, 0x27e1f20, 0x27a11f8, 0x2797bc8,
0x2797c18, 0x27a1060, 0x2797a90, 0x286a698, 0x26fae10, 0x26fae60, 0x2715ac0,
0x2744080, 0x27c72a0, 0x2709160, 0x27e5618, 0x2709050, 0x27e5898, 0x25aacb8,
0x25aad08, 0x270cb78, 0x276fd88, 0x270c858, 0x26ed4f0, 0x2641dd8, 0x263f648,
0x276e500, 0x263f4c0, 0x2641bf0, 0x2809dd8, 0x2798498, 0x25ab348, 0x2759130,
0x2792858, 0x27766d8, 0x263f470, 0x26a98a8, 0x27c7518, 0x276e4a8, 0x26e9298,
0x26e9220, 0x27157b8, 0x272f8b0, 0x2672408, 0x2786378, 0x27fb6a0, 0x27fbba8,
0x2734340, 0x27b09f0, 0x27b11f0, 0x276d410, 0x276dc10, 0x27a02a8, 0x27a0aa8,
0x271cf00, 0x271d700, 0x2701ef8, 0x27026f8, 0x26f9090, 0x26f9780, 0x27ac090,
0x27ac598, 0x2691b20, 0x2692028, 0x2796738, 0x2641528, 0x2647a10, 0x2647b10, 0


x26d6010, 0x26d60c0, 0x269c938, 0x2780048, 0x2782290, 0x27c3fc0, 0x27c4258,
0x2732588, 0x273f140, 0x26e8a50, 0x2619e58, 0x26e8940, 0x26fe1a0, 0x26fe040,
0x26fe140, 0x2619da8, 0x2799e00, 0x27c6890, 0x27c6990, 0x26d3420, 0x2613af8,
0x25aa7a8, 0x2613850, 0x2796d38, 0x2796c28, 0x2797278, 0x27ec390, 0x27eca48,
0x27f4568, 0x27f4668, 0x27ab1c0, 0x2865d98, 0x2854390, 0x273b7b0, 0x2800c68,
0x279a5b8, 0x27153e0, 0x26e8f50, 0x2800c18, 0x2715430, 0x27155a0, 0x2715650,
0x279a0d0, 0x2752a08, 0x27ecaf8, 0x27a91d8, 0x27a9470, 0x27ec2c8, 0x274d7a0,
0x275a4a0, 0x2752e20, 0x2753190, 0x28356b0, 0x2835bb8, 0x2836380, 0x2813690,
0x2619d58, 0x26fe0f0, 0x2647ac0, 0x2750b00, 0x270cc88, 0x26415d8, 0x26f5af8,
0x28482a8, 0x27a1110, 0x2766120, 0x27e1e00, 0x27e1eb0, 0x2847fa8, 0x27ac648,
0x28045b0, 0x26fac78, 0x26fad28, 0x25aa858, 0x27e0130, 0x27e0180, 0x2807010,
0x2750ab0, 0x270cc28, 0x286a7b8, 0x27343f0, 0x27ac6f8, 0x276fe88, 0x263f5f8,
0x26baab8, 0x2710398, 0x2641d38, 0x2641d88, 0x27ab270, 0x27ab320, 0x275a5b0,
0x275dbf8, 0x275dba8, 0x2825a30, 0x26724b8, 0x2672568, 0x26e73a8, 0x26e7458,
0x2854440, 0x28544f0, 0x269c9e8, 0x269ca98, 0x27593c8, 0x2759478, 0x2613ba8,
0x2702bc8, 0x27ec218, 0x25aa620, 0x27e5788, 0x273f1f0, 0x2758a80, 0x27e5728,
0x25aa730, 0x27ec158, 0x275a2b8, 0x275a368, 0x275a418, 0x27a92e8, 0x27a9398,
0x27767f8, 0x27768a8, 0x2776958, 0x2836e70, 0x2836f20, 0x27323a0, 0x2732450,
0x27f4618, 0x2732500, 0x27965b0, 0x2796660, 0x26f9598, 0x27684c8, 0x270fd80,
0x2847c90, 0x2613960, 0x2847da0, 0x272f960, 0x272fa10, 0x26f9708, 0x270fc48,
0x2768478, 0x2613900, 0x270fcf8, 0x26c27c0, 0x26f96a8, 0x27685d8, 0x27c40d0,
0x27c4180, 0x2782340, 0x27823f0, 0x27824a0, 0x28259e0, 0x27c6940, 0x270cce8,
0x2613fe8, 0x281ff00, 0x274d900, 0x272fb20, 0x26d63b8, 0x271b4b0, 0x2796cd8,
0x270fea0, 0x28237f0, 0x281fb28, 0x27a0fe0, 0x275a150, 0x26e9000, 0x26e9050,
0x280cb18, 0x280f328, 0x280f3d8, 0x280f080, 0x2811cc0, 0x28144d0, 0x2816ff8,
0x2816f48, 0x2819880, 0x281c090, 0x281e8a0, 0x28210b0, 0x28238c0, 0x28260d0,
0x28288e0, 0x282b0f0, 0x2807840, 0x282f018, 0x2831828, 0x2834038, 0x2836848,
0x28368f8, 0x28369a8, 0x28365a0, 0x2641710, 0x263f068, 0x27769e0, 0x2836ca0,
0x26fa130, 0x276b9f0, 0x26ed540, 0x2836ff8, 0x271c138, 0x2744140, 0x2702a08,
0x2837400, 0x2837158, 0x2837388, 0x2732a18, 0x261a2e8, 0x26fe6f0, 0x26feaa8,
0x27c6c38, 0x27c7140, 0x276e570, 0x2797d18, 0x2797c68, 0x2754f50, 0x275a6b0,
0x27fe550, 0x27fbdb0, 0x27fe478, 0x2716058, 0x2715ca0, 0x2708e68, 0x266eb78,
0x271b808, 0x271b5e8, 0x2786b20, 0x266ef20, 0x266ee20, 0x26fa9a0, 0x27708d0,
0x27a1288, 0x266f2c8, 0x266f378, 0x27d8880, 0x269d3e0, 0x26967d0, 0x27496a8,
0x2736f90, 0x27f1f28, 0x26bad78, 0x2696568, 0x27ed0e0, 0x27f1628, 0x273c468,
0x2789618, 0x2789508, 0x266f808, 0x2710650, 0x270d140, 0x2755af8, 0x2759d10,
0x2776b40, 0x2792798, 0x27db5d8, 0x268bfd0, 0x278a2b8, 0x2797fb0, 0x269cb80,
0x2789e50, 0x27802f0, 0x2780298, 0x26c1b10, 0x261f0c8, 0x26ea1c8, 0x26bab78,
0x2800d18, 0x27dfc50, 0x28193b0, 0x25aad58, 0x2804190, 0x27ddfd0, 0x27e0580,
0x2749418, 0x273bdf0, 0x280c8f8, 0x2816e30, 0x2809fd8, 0x26c2650, 0x270c500,
0x26aff48, 0x26b0748, 0x277b6f0, 0x27663f0, 0x283bb30, 0x283e340, 0x2840b50,
0x2843360, 0x2845b70, 0x2848380, 0x284d2b0, 0x284fac0, 0x28522d0, 0x2854ae0,
0x28572f0, 0x2859b00, 0x285c310, 0x285eb20, 0x2861330, 0x2863b40, 0x2866350,
0x281e2e8, 0x28662a0, 0x281e150, 0x28661f0, 0x28666f8, 0x28667a8, 0x2866858,
0x2866908, 0x2820af8, 0x2866648, 0x2823478, 0x28233b8, 0x2866cb0, 0x2866c00,
0x2823170, 0x2866ba0, 0x2820a20, 0x2866a18, 0x2820fd0, 0x2822648, 0x2822460,
0x2822560, 0x2822890, 0x2822940, 0x28229f0, 0x2825c78, 0x28223b0, 0x2822c38,
0x2822ce8, 0x2822d98, 0x2822e48, 0x2822ef8, 0x2822350, 0x2823868, 0x2867d68,
0x2867e18, 0x2867ec8, 0x2867f78, 0x2868028, 0x281bad8, 0x281bb88, 0x286a9a0,
0x286aa50, 0x286ab00, 0x2736f40, 0x27b14f0, 0x2703808, 0x27d2968, 0x266eed0,
0x2822510, 0x27922a8, 0x25f09a8, 0x2823220, 0x268bef8, 0x2647c88, 0x268bc00,
0x2753370, 0x2770980, 0x27709d0, 0x2825d88, 0x273c578, 0x27f1738, 0x2751820,
0x2822b00, 0x268c270, 0x28230c8, 0x2648070, 0x2717e28, 0x27896c8, 0x2737040,
0x27bf530, 0x27ba9f0, 0x25f10f8, 0x2718b70, 0x2801010, 0x275a888, 0x2777258,
0x271b6f8, 0x26faab0, 0x2822818, 0x271b968, 0x27f1e60, 0x261f750, 0x28665f8,
0x27f1578, 0x27ecf80, 0x265eee8, 0x281e5f8, 0x26f5690, 0x26fa8f8, 0x2820ea0,
0x27f1b30, 0x27f1a80, 0x2770858, 0x26fc348, 0x2755aa0, 0x270cf98, 0x270d048,
0x270ce70, 0x26b9fe8, 0x27f1048, 0x27866b8, 0x2786a28, 0x27101e8, 0x2710558,
0x26143a0, 0x2614710, 0x26e93b8, 0x2820950, 0x286dbc8, 0x2648020, 0x27187c8,
0x26fa948, 0x26fab60, 0x271ba18, 0x2768830, 0x266f780, 0x27f1be0, 0x27ed030,
0x271b8b8, 0x2786cc0, 0x27bac98, 0x27f1d40, 0x27f1df0, 0x2822758, 0x27ecde8,
0x26ecb38, 0x2866450, 0x26baf48, 0x26baf98, 0x271b510, 0x26c9848, 0x2777110,
0x2716f50, 0x27ece98, 0x2736548, 0x27365f8, 0x27366a8, 0x261f5e0, 0x261f690,
0x281e6a8, 0x281e848, 0x271bde8, 0x26b1000, 0x271bd28, 0x266edc0, 0x281e758,
0x269d138, 0x269d1e8, 0x26b1110, 0x2770b30, 0x2770be0, 0x27f1838, 0x27f18e8,
0x27f1998, 0x26f5448, 0x26f54f8, 0x26f55a8, 0x265eca0, 0x265ed50, 0x265ee00,
0x27f1330, 0x27f13e0, 0x27f1490, 0x26fa688, 0x26fa738, 0x26fa7e8, 0x26fa898,
0x266f090, 0x266f140, 0x266f1f0, 0x2717a30, 0x2789e00, 0x25aaa10, 0x280c8a8,
0x27c70f0, 0x27ab490, 0x275a660, 0x26bac28, 0x27d2868, 0x276e908, 0x26ea990,
0x2672938, 0x273f570, 0x2744268, 0x27869d8, 0x26bad28, 0x2749540, 0x270ce18,
0x2814478, 0x2709260, 0x2732778, 0x276baa0, 0x2708d68, 0x280a220, 0x26fc238,
0x273c860, 0x2614638, 0x2809f88, 0x277ba98, 0x26b3258, 0x26fc448, 0x270c930,
0x277b988, 0x26b0620, 0x26b04c0, 0x26b05c0, 0x2809ed8, 0x2819770, 0x26b0e58,
0x2820d90, 0x266f5c0, 0x266f6c0, 0x26c25c8, 0x26c2468, 0x28287d0, 0x2717c18,
0x2717d18, 0x2717920, 0x27102e8, 0x2766260, 0x2766360, 0x28221c8, 0x2822278,
0x26c1a00, 0x27d24d8, 0x27c8b00, 0x27e1d50, 0x27e1fe8, 0x281fba0, 0x27f0f38,
0x2797898, 0x27975f0, 0x26d62a8, 0x27df4e0, 0x26d6198, 0x2759180, 0x27dfaf8,
0x27dfbf8, 0x2759908, 0x27595b0, 0x270fb38, 0x270fa28, 0x2710078, 0x2710178,
0x270ffc8, 0x2613e78, 0x2613d68, 0x26140c0, 0x26e76a0, 0x26e7590, 0x26e8be8,
0x26e8b28, 0x26e9310, 0x281b9a0, 0x27ab380, 0x26afe38, 0x273bd58, 0x2825bc8,
0x273bc48, 0x27fe3f0, 0x2865fa8, 0x2865de8, 0x28140b0, 0x2854688, 0x2854788,
0x282ad80, 0x2831430, 0x28314e0, 0x284a798, 0x284a848, 0x2718980, 0x28545d8,
0x2828488, 0x28363f0, 0x2806f60, 0x2816970, 0x2845718, 0x2845a88, 0x28430a0,
0x2840940, 0x283e1f0, 0x2833fc0, 0x28480e8, 0x2863920, 0x2861188, 0x285e9f0,
0x285c258, 0x2856e98, 0x2857208, 0x284cea0, 0x284d210, 0x286a848, 0x277b210,
0x2765fb0, 0x272f370, 0x272fb70, 0x2715078, 0x2715878, 0x2715be8, 0x282e648,
0x282e6f8, 0x282e7a8, 0x282e858, 0x282e908, 0x282e9b8, 0x282ea68, 0x282eb18,
0x2831698, 0x27551a8, 0x266ec88, 0x282ec88, 0x282e5a8, 0x282e5f8, 0x26145e8,
0x2819720, 0x2809e88, 0x26b0570, 0x271c600, 0x26fc3f8, 0x26b3208, 0x27f1158,
0x26fc1e8, 0x2710508, 0x26fea58, 0x2836c50, 0x2837108, 0x273c410, 0x2770808,
0x2758c58, 0x280c780, 0x276e100, 0x26fe2e0, 0x2619f98, 0x2671da8, 0x261a298,
0x2736e40, 0x2782530, 0x268c0f8, 0x26f9b40, 0x27fe908, 0x25ab808, 0x27baf38,
0x273f610, 0x2836a58, 0x26f9ee8, 0x2755388, 0x2755e10, 0x28366b0, 0x28367d0,
0x2641930, 0x27f1208, 0x2819648, 0x27172e0, 0x277b7a0, 0x2717390, 0x2732b28,
0x26b06d0, 0x27c6aa0, 0x276dd28, 0x2836ab8, 0x26d6248, 0x26b0fb0, 0x2809900,
0x27dfba8, 0x2710478, 0x2759660, 0x27597e0, 0x270fad8, 0x2710128, 0x270ff78,
0x2804240, 0x2732940, 0x2828880, 0x266f670, 0x2766310, 0x27178d0, 0x2789f00,
0x2789f60, 0x26baee8, 0x27176d8, 0x2717cc8, 0x26c2418, 0x284d260, 0x279b860,
0x26fbe10, 0x266ec28, 0x275a838, 0x2816ee0, 0x2814160, 0x261acf0, 0x26e9360,
0x26ef038, 0x26f5800, 0x2866500, 0x283e2a0, 0x27e2b60, 0x26ecae8, 0x283e2f0,
0x2800fc0, 0x2718ab0, 0x263f230, 0x271b7b8, 0x2866400, 0x282ad30, 0x2854738,
0x266ed38, 0x2865f58, 0x26c2578, 0x2717738, 0x2759780, 0x25f0eb0, 0x26416a8,
0x26fafc0, 0x272fc20, 0x27108a8, 0x25f0a08, 0x27171a8, 0x2800dc8, 0x26c1bc0,
0x281e200, 0x261f1c8, 0x2861238, 0x28194b0, 0x2732640, 0x27e2188, 0x26ece00,
0x2814358, 0x26f9fc0, 0x275a1d8, 0x26fa290, 0x2814238, 0x2836500, 0x2836550,
0x26fa240, 0x280cac8, 0x2863af0, 0x2840a50, 0x2840b00, 0x26c9280, 0x2614038,
0x27f64e0, 0x27d8200, 0x2759968, 0x2759a18)
    tokDescription  = "NSS Generic Crypto Services     "
    slotDescription = "NSS Internal Cryptographic Services Version 3.4         
      "
}
(dbx) 

However, that slot is not used as part of the session creation, but only to
check if the slot is valid. The slotID of 1 ends up getting passed down to
pk11_NewSession from softoken ..
    /* new session (we only have serial sessions) */
    session = pk11_NewSession(slotID, Notify, pApplication,
						 flags | CKF_SERIAL_SESSION);


In pk11_NewSession, pk11_SlotFromID is called yet again to find the slot from ID
1, and sets that slot into session->slot . This is still pointing to slot 1, the
crypto services.

After creating the session, a session ID is computed as follows :

    sessionID = slot->sessionIDCount++ | (slot->index << 24);

One interesting fact is that the slotID is not a factor in this computation,
even though we are going to try to recover the slot ID later from the session.

I think we are relying on the invariant that
  slot->slotID == nscSlotList[slot->index]
Comment on attachment 70420 [details] [diff] [review]
Use PL_HashTableLookupConst() to look up in nscSlotHashTable

Although the nscSlotHashTable is looked up with
the wrong function, PL_HashTableLookup() won't
modify the table unless the two entries (slots)
are in the same buckets.

Since the key is the slot ID and the slot IDs
are 1 and 2, I doubt they would be hashed to the
same bucket.  Therefore, although this patch is
correct, I don't think it will fix the "MD5 digest
function failed" errors.
The snippets from the debug session have established that this PKCS#11 session
is being created in the NSS crypto services slot (slotID 1), but later on it is
getting looked up in the NSS database (slotID 2).

I finally understand what this code does about the sessionID !!!

It tries to take the slot->index, which is the index into the nscSlotList array
of slots . So index 0 is the correct index when the session gets created, since
nscSlotList[0] is 1.
This index gets shifted to the left by 24 bits. Then the slot's sessionIDCount
gets added to it.

Later on, when the session is looked up, we shift it 24 bits to the right, and
retrieve the index.
The problem occurs when more than 2^24 sessions are in the slot. At that point,
the 24th bit is no longer the slot index, but slot index + 1. It could be more
as the number of PKCS#11 sessions in the slot increases, up to 2^8 for the upper
8 bits.
So when the lookup occurs, the slot index is not zero but 1. nscSlotList[1]
returns 2. So we look up slot 2 in the hash table, and get the 2nd slot. Of
course the session isn't found and the operation fails.

This seems to be a design problem because we are packing the session and slot ID
into one 32-bit session ID. There is nothing to prevent the overflow of session
IDs past 2^24.

I have verified that this was the case in my test :

(dbx) p slot->sessionIDCount
slot->sessionIDCount = 16926143
(dbx) 

The 12 hour run of my SSL stress test was sufficient to create more than 2^24
PKCS#11 sessions .

We may need to do something to reuse lower session IDs. The token shouldn't just
fail when it reaches that number of sessions. Especially since they are "dead"
sessions that should have been long closed.



BTW, there has been some memory growth observed in my server since the beginning
of the test. I don't know if it's due to the error condition or not. I will keep
the server running but continue to hit it with a client to increase the session
count. I will run pmap in the background periodically to see if the server is
growing.
I have confirmed that the memory usage of the process is growing at a rapid rate
in the error case. It doubled from 60MB to 120MB in an hour of stress (all SSL
sessions failed with MD5 digest failures). All those PKCS#11 sessions that were
created in slot 1 couldn't be freed for the same reason they couldn't be looked
up in the digest init. The lookup fails before freeing the session too.
FYI, I had 936703 sucessful full handshake SSL sessions in the server before I
hit the 2^24 limit of PKCS#11 sessions in softoken. That's about 18 PKCS#11
sessions per SSL session. Bob says we should use 4 to 8 PKCS#11 sessions per SSL
session only. Maybe there is something going on there too.

Good catch, Julien.  It all makes sense now.

One quick fix is to change
  sessionID = slot->sessionIDCount++ | (slot->index << 24);
to
  sessionID = (slot->sessionIDCount++ & 0xffffff) | (slot->index << 24);

This caps the slot->sessionIDCount component of sessionID at
0xffffff and does not allow it to overflow to the slot->index
component of sessionID.
Wan-Teh,

I discussed this shortly with Bob after he dropped by from the RSA conference.
There are issues with doing this because it's going to wrap to PKCS#11 sessions
0 or 1 which have special meaning in NSS and stay open. The problem is we could
clash with some old PKCS#11 sessions that are still open. We could work around
it by doing a lookup first to see if there is a clash, but that would come at an
extra cost.

We might need to do something different and possibly more complex to come up
with new session numbers. Bob suggested a fixed-size bitmapped array to check if
a particular session is in use. That would probably mean restricting the session
ID range to something less than 24 bits though, unless we want to allocate some
2^24   bits array - which would be 2 MB. I'm sure the client guys would hate us
forever for doing that, but it would be good for servers :). 
Perhaps the extra hash table lookup to check for a clash wouldn't be so bad? It
would only be needed after the wrap-around occurred. So most likely only in
servers. Or maybe the above-mentioned big bitmap array could be dynamically
allocated at that time ? Presumably a browser would never run into this because
it would crash long before 16M PKCS#11 sessions would be performed, unlike a
server :)

BTW I also suggested to Bob something like
sessionID = slot->index << 24 + 2 + slot->sessioncount % (2^24 -2) 

That would create an exclusive session in the 2 - 2^24 range and would not clash
with the "reserved" 0 and 1 sessions that NSS uses. The problem with that is
that other PKCS#11 implementations might not use 0 and 1 as reserved sessions,
but might use other. And it still doesn't solve the case where the application
explicitly left some old sessions open, while most of them were closed.

I also suggested we decrement the count if possible when the highest session is
closed to keep it from going up too quickly. But that would imply that we would
reuse session IDs fast after a session is closed. At that point an application
might be confused and have a reference to an old session pointing to a new
session. Bob said it also creates problems with token removal if we support that
in the future. There is no 100% safe and 100% efficient fix here as you can see.

We might just evaluate the impact of an extra lookup (enabling it in all cases,
not just on wraparound when the count reaches 2^24).

The other thing is, maybe could we pass a pointer as a session ID and bypass the
whole lookup in the first place ? I guess maybe that would break 64-bit
platforms. Is the PKCS#11 session ID strictly 32-bit on all platforms ?

I use PR_AtomicIncrement to increment slot->sessionIDCount.
This field is no longer protected by slot->slotLock.

I check for duplicate session IDs before I add *any* new
session to the session hash table (slot->head[]).  Although
we only need to do this after slot->sessionIDCount exceeds
2 ^ 24, I think it is better to always do this check, otherwise
we will only be able to test this code path after 2 ^ 24
sessions have been created.

The effect of the extra hash table lookup on the contention
for the session locks is unknown.  Kirk, could you measure
that?

If you have other solutions, please propose them.
Attachment #70475 - Attachment is obsolete: true
> The other thing is, maybe could we pass a pointer as a session ID and bypass the
> whole lookup in the first place ? I guess maybe that would break 64-bit
> platforms. Is the PKCS#11 session ID strictly 32-bit on all platforms ?

CK_SESSION_HANDLE is a CK_ULONG, which itself is defined as unsigned long int. 
Thus it should be large enough for any pointer in the address space, correct?

I like this idea.  Each session has a PK11Session pointer, so why not just
return the pointer address as CK_SESSION_HANDLE?
Kirk,

I suspect that the problem you described in comment #3 and comment #4
is not the same problem.

Next time you reproduce it, could you attach a debugger to selfserv
and print the value of slot->sessionIDCount?  If it is less than
2^24, it is not the same problem.

Thanks.
Assignee: ian.mcgreer → wtc
Ian wrote:
> CK_SESSION_HANDLE is a CK_ULONG, which itself is defined as unsigned long int. 
> Thus it should be large enough for any pointer in the address space, correct?

No, an unsigned long int may not be large enough for a pointer.  A
counter example is 64-bit Windows, where a long is 32-bit and a
pointer is 64-bit.

It appears that PKCS#11 mandates that CK_SESSION_HANDLE be defined
as a CK_ULONG, so most likely we won't be able to change it to an
unsigned integer type that is large enough for any pointer in the
address space.

I found that 0 (CK_INVALID_HANDLE) is an invalid session handle,
so this new patch makes sure the new session ID we generate is
nonzero.
Attachment #70525 - Attachment is obsolete: true
Re: making sessionID a pointer:

The intent of making the session ID be an integer (not a pointer) is to 
avoid crashes caused by errant callers passing in bogus values.  The 
integer necessitates a lookup, which in turn prevents bogus pointer
references.  

Although 0 is a special value, IINM, the token is otherwise free to create 
sessionIDs as it sees fit, subject to the constraint that it be a ULONG.
I generated the failure again under dbx on soupnazi:
selfserv: HDX PR_Read returned error -12215:
MD5 digest function failed.

t@41 (l@76) stopped in NSC_OpenSession at line 2681 in file "pkcs11.c"
 2681       PK11_USE_THREADS(PZ_Lock(slot->slotLock);)
(/usr/suntools/internal/SUNWspro/bin/../WS6U1/bin/sparcv9/dbx) n
:
t@41 (l@76) stopped in NSC_OpenSession at line 2682 in file "pkcs11.c"
 2682       sessionID = slot->sessionIDCount++ | (slot->index << 24);
(/usr/suntools/internal/SUNWspro/bin/../WS6U1/bin/sparcv9/dbx) p
slot->sessionIDCount
slot->sessionIDCount = 16952579
(/usr/suntools/internal/SUNWspro/bin/../WS6U1/bin/sparcv9/dbx)
Thank you, Kirk.  16952579 is 0x102ad03, which is greater than 2^24.
This confirms that you are seeing the same problem.  (You also saw
some other errors that I did not see though.)

I also found that the session IDs in NSS 3.3.2 have similar problems,
which I wrote about in bug 126769.

At this point, I think we have understood the problem and what remains
is a design issue.  So I am reassigning the bug to Bob.  Here is a
summary of the issues and proposed solutions.

1. We can try to fix the problems in the current design.  This requires
an efficient method to generate distinct session ID count.  Every OS
that has pid's must have solved this problem.  If doing a lookup in the
session hash table is inefficient, I suggest we look at the code that
generates the pid's in FreeBSD.

2. We can use the pointer to PK11Session as the session handle.  This
design has two problems.
- NSS will crash if errant callers pass in bogus values.
- CK_SESSION_HANDLE (CK_ULONG) is not large enough for a pointer in
  64-bit Windows.
But it has two advantages.
- We don't need to solve the problem of generating distinct session ID
  count.
- The contention for "sessionLock" is gone because there is no session
  hash table.

Do you think my proposed patch (attachment 70544 [details] [diff] [review]) is good enough for
NSS 3.4?
Assignee: wtc → relyea
This new version has a slot->sessionIDConflict counter to log
the number of times we find the same session ID in the session
hash table and have to regenerate the session ID.
Attachment #70544 - Attachment is obsolete: true
BTW, the pointer approach has another possible problem - with zone allocator and
other memory allocators, the pointers for old/new sessions could get reused so
there is still a possible clash where an app would have an old session ptr
pointing to a new one. I guess that's not really something that could be avoided
though.
I checked in both of my patches (the nscSlotHashTable const lookup
patch and the session ID overflow patch) into the tip of NSS.

Kirk, Julien, please repeat your stress tests with the tip of NSS.
Thanks.
Wan-Teh,

The server has been running on Solaris with your test for about 5 hours. There
haven't been any errors yet, and 389804 HTTP requests/connections/full SSL
handshakes have completed so far. In addition, the memory usage of both the
client and server running the latest 3.4 TIP have remained stable so far, so we
may be seeing the end of this problem. Hopefully tomorrow it will be the same.
Then I can restart the test with single and double-handshake client auth :)

Now up to 637792 full handshakes, still no errors or memory growth.
I've been unable to reproduce this failure with the latest tip build 
on soupnazi.  I ran for over two hours.  I'll plan on leaving a 
run overnight tonight...
After 18 hours and 1434864 full SSL handshakes on my Sparc without client auth,
I still see no errors or memory growth on either side. Looks good to me.

I also have an NT test running since 3am and the server is still up. I can't
check the request count or memory usage from home though.
I consider my Solaris test to have passed, after 23 hours and 1790039 full SSL
handshakes without any error or leak.

My NT test was unsuccessful, but it is a different problem, not MD5 digest
errors, so I will file another bug.
The consensus is that my patch is good enough for NSS 3.4.
I opened bug 127172 for future performance enhancement if
my patch has negative impact on performance.
Status: NEW → RESOLVED
Closed: 23 years ago
Resolution: --- → FIXED
I ran overnight with last nights build.  No failures.
I pulled another tip build this morning, and attached
selfserv results to bug 127172.
Marking verified
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: