170500 - Troubles with Mutexes - PKCS#11 supporting CKF_LOGIN_REQUIRED

Status: RESOLVED INCOMPLETE

(Core Graveyard :: Security: UI, defect, P3)

Description

[Pavel Zazvorka]

We have checked smart card/PKCS#11 library with new Mozilla/Netscape7. We have
more version of the library, and one of them called (supporting
CKF_LOGIN_REQUIRED) stop  to work at all (Netscape 4.7 was working fine ...)

Syndrom:
--------
Mozilla/Netscape 7 using PKCS#11 with CKF_LOGIN_REQUIRED locks. The version
without CKF_LOGIN_REQUIRED is working well.

Diagnosis:
----------
We have traversed an Mozilla SSL and discover that
lock appears during a handling "ServerHello" message as seen on stack:

nssSlot_IsTokenPresent
nssToken_IsPresent
pk11_IsPresentCertLoad
PK11_IsPresent
PK11_TokenExists
ssl3_config_match_init
ssl3_HandleServerHello
ssl3_HandleHandshakeMessage
ssl3_HandleHandshake
ssl3_HandleRecord
ssl3_GatherCompleteHandshake
ssl_GatherRecord1stHandshake
ssl_Do1stHandshake
ssl_SecureSend
ssl_SecureWrite
ssl_Write
nsSSLIOLayerWrite
PR_Write
nsSocketOS::Write

and immediate call causing lock is an attempt to create Mutex in our source

PKCS11api.cpp at line 666
in function C_GetSlotInfo(unsigned long 0x00000000, CK_SLOT_INFO * 0x01bef58c) 
... mutex lock attempt

We have modify our
implementation to force usage of native threads - it is possible according to
the PKCS#11 with rv=CKR_NEED_TO_CREATE_THREADS. The combination of library
supporting CKF_LOGIN_REQUIRED with native mutexes was working fine

We will go on with the research, however it is quite suspicious behavior.
I hope to prove/cancel the bug report soon with further details. We would
apreciate to be told if somebody is using succesfully CKF_LOGIN_REQUIRED with
"external" mutexes.

Comment 1

[Stephane Saux]

cc relyea

Comment 2

[Robert Relyea]

The built-ins use the external mutexes, though it doesn't have LoginRequired
set. The CreateMutex call should map back to a PR_NewLock() call in NSPR. Maybe
setting a breakpoint there can shed some light. It sounds like either data
corruption problem on the pointer passed in to your application, or a calling
sequence mismatch. If PR_NewLock() is south at this point, Netscape 6 isn't
going to get to far.

The critical difference here isn't CKF_LOGIN_REQUIRED, but the removeable flag.

Comment 3

[John Unruh]

Setting to new

Comment 4

[John G. Myers]

Mass reassign ssaux bugs to nobody