354613 - non-NSPR applications will crash after unloading softoken and NSPR

Status: NEW

(NSS :: Libraries, defect, P3)

Description

[Julien Pierre]

Use the Windows build of softokn3.dll with OS_TARGET=WINNT, and then build a program not linked against NSPR that does the following  :

    hModule = LoadLibrary("softokn3.dll");

    pC_GetFunctionList = (CK_C_GetFunctionList)
                             GetProcAddress(hModule, "FC_GetFunctionList")

    crv = pFunctionList->C_Initialize(&initArgs);
    crv = pFunctionList->C_Finalize(NULL);

    FreeLibrary(hModule)

    Sleep(10000);
    exit(0);

This program will crash during the Sleep(10000); when executing code from an NSPR background thread after NSPR has been unloaded from memory .

The problem is that :

1) C_Finalize didn't call PR_Cleanup . PR_Cleanup is only supposed to be called as the end of main(), but this can't be done in programs that don't link with NSPR such as this one .

2) Even if it had been called, PR_Cleanup doesn't stop internal background threads .

This problem is actually common to programs that unload any NSS DLLs, not just softoken . But I think it is more serious with the PKCS#11 softoken, which is explicitly defined to be a dynamically loadable/unloadable mechanism, and is actually used in non-NSPR aplications, such as the JDK's PKCS#11 engine .

I only know of crashes on the Windows OS_TARGET=WINNT platform. There may be other consequences to unloading NSPR on other platforms. I haven't tested this yet. For now I have limited this bug to PC / Windows NT for this reason, but that may change if problems on other platforms are discovered. These could include other internal threads used on other platforms, or any OS callbacks that are registered into NSPR and may be called after the shared library is unloaded (eg. thread termination callbacks).

Comment 1

[Julien Pierre]

This problem can be reproduced on NSS 3.10.x OS_TARGET=WINNT bits, or on NSS 3.11.4 beta with bug 115951 fixed - fixing the leak of the freebl shared lib also causes the NSPR DLL refcount to go to zero if softoken is unloaded .

Comment 2

[Nelson Bolyard (seldom reads bugmail)]

(In reply to comment #0)
> Use the Windows build of softokn3.dll with OS_TARGET=WINNT, and then build a
> program not linked against NSPR that does the following  :
> 
>     hModule = LoadLibrary("softokn3.dll");
> 
>     pC_GetFunctionList = (CK_C_GetFunctionList)
>                              GetProcAddress(hModule, "FC_GetFunctionList")

Seems like there's a step missing here.  The above call returns the 
address of a function that returns pFunctionList.  But you have to call
that function to get pFunctionList.  I don't see that function getting
called here.

>     crv = pFunctionList->C_Initialize(&initArgs);
>     crv = pFunctionList->C_Finalize(NULL);