454508 - NSS_InitReadWrite() returns SECFailure if a alternate sqlite3.dll is found by LoadLibrary()

Status: RESOLVED WONTFIX

(NSS :: Libraries, defect)

Description

[Bryan Ashby]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1

If calling NSS_InitReadWrite() (or any of the NSS_Init* functions) SECFailure will be returned if a alternate, non compatible sqlite3.dll is found by LoadLibrary() before Mozilla/Firefox's version.

Reproducible: Always

Steps to Reproduce:
1. Copy a non-FF compatible sqlite3.dll or create a dummy file in C:\Windows\System32, C:\Windows, C:\Windows\System or anywhere else in %PATH%
2. Outside of Firefox, call NSS_InitReadWrite().

Actual Results:  
NSS_InitReadWrite() eventually attempts to LoadLibrary() on softokn3.dll without path information which appears to be statically linked to sqlite3.dll. If one of the sqlite3.dll modules mentioned in step 1 are located, softokn3.dll will fail to load and NSS_InitReadWrite() will return SECFailure. 

Expected Results:  
NSS_InitReadWrite() should return SECSuccess

This could in theory happen with other NSS DLLs as well. I was able to work around this issue by calling SetDllDirectory() before NSS_InitReadWrite() to alter the LoadLibrary() search path.

Comment 1

[Nelson Bolyard (seldom reads bugmail)]

For what application is this an issue? 

According to http://msdn.microsoft.com/en-us/library/ms682586(VS.85).aspx
in all cases, the first directory searched for a DLL is 
"The directory from which the application loaded." 
For Firefox applications, NSS's shared libraries, including sqlite3, should
be in the directory where firefox.exe is located.

Comment 2

[Wan-Teh Chang]

Bryan, I believe I understand the problem you described.
As Nelson said, we expect that the NSS DLLs are installed
in the same directory where the .exe resides.  This is how
the NSS DLLs are installed for firefox.exe.

Are you installing the NSS DLLs and your .exe in different
directories?

Another way to make this work is for nss3.dll to load
softokn3.dll using LoadLibraryEx with the
LOAD_WITH_ALTERED_SEARCH_PATH flag.  Or it can try
LoadLibrary first.  If that fails, then try LoadLibraryEx
with the LOAD_WITH_ALTERED_SEARCH_PATH flag.

Comment 3

[Bryan Ashby]

As pointed out, DLLs are first searched for in the current processes directory. This is ONLY a issue when using the NSS DLLs from either a) a alternate installation directory (which could point the issue to being a configuration error) or b) using the NSS DLLs that are installed on the users box already by FF from a external application. That said, it would never be a issue with Firefox itself. Depending on how you look at it, this may or may not be considered a bug at all.

Our application is using (b) above: If a Firefox installation is detected, we dynamically load up the NSS modules and utilize their functionality. Obviously not the normal usage pattern.

Comment 4

[Nelson Bolyard (seldom reads bugmail)]

For reference, SetDllDirectory() is documented in
http://msdn.microsoft.com/en-us/library/ms686203(VS.85).aspx

Given that:
a) on Windows, the NSS libraries are application libraries, not system 
libraries, and 
b) your application (mysterious and unknown as it is :) wants to rely on 
the DLLs belonging to another application, 
I think SetDllDirectory is the right solution for you.

Because they are application libraries, and different applications may 
choose to install them in different directories, we cannot link NSS 
libraries with path names (as opposed to simple file names) for other 
NSS shared libraries.  

So, I think this is not an NSS bug.  
Wan-Teh, do you concur?

Comment 5

[Wan-Teh Chang]

We can consider this an NSS or even an NSPR bug.
We can fix this by having PR_LoadLibrary call
LoadLibraryEx without and then with the
LOAD_WITH_ALTERED_SEARCH_PATH flag.  Or we can
add a new PR_LOAD_ALTERED_SEARCH_PATH flag for
PR_LoadLibraryWithFlags and have NSS call it
without and with that flag.  But this will make
PR_LoadLibrary take twice as long to fail, which
may not be acceptable.

Workaround 1:

I worked around this bug by first using LoadLibraryEx
with LOAD_WITH_ALTERED_SEARCH_PATH to load softokn3.dll,
and then using LoadLibraryEx with LOAD_WITH_ALTERED_SEARCH_PATH
to load nss3.dll.  Since softokn3.dll is already loaded
when we call NSS_InitReadWrite, the LoadLibrary call
made by nss3.dll doesn't need to load sofokn3.dll again,
so it doesn't need to worry about how to search for
sqlite3.dll.

Workaround 2:

Use SetDllDirectory, if you don't need to support Windows 2000
and it is fine for you to change the application's DLL
search path (a process-wide property).

Comment 6

[Wan-Teh Chang]

I should add that LoadLibrary (and LoadLibraryEx as well)
uses the same error code ERROR_MOD_NOT_FOUND (126) when it
cannot find the primary DLL and when it cannot find a
dependent DLL.  This is why we would always need to retry
with the LOAD_WITH_ALTERED_SEARCH_PATH flag, which would
make PR_LoadLibrary take twice as long to fail.

Comment 7

[Julien Pierre]

softoken always attempts to first load the sqlite3.dll from the same directory as softoken3.dll first. It shouldn't matter if an sqlite3.dll exists in another location.

Unless you deleted the sqlite3.dll which was in same location as softokn3.dll, I don't see how that could fail.

The code in question is sftkdb_LoadLibrary in lgglue.c .
Is PR_GetLibraryFilePathname not returning the path of the softoken DLL ?

All NSS libraries expect to be in the same location. In particular, softoken and its dependencies (freebl, sqlite3) must be in the same location. If you don't have that in your installation, then don't expect it to work.

Comment 8

[Wan-Teh Chang]

softokn3.dll is linked with sqlite3.dll directly.

softokn3.dll only uses sftkdb_LoadLibrary to load nssdbm3.dll.  The "lg"
in lgglue.c stands for "legacy", referring to the old DBM code.

Comment 9

[Julien Pierre]

Thanks for that Wan-Teh. I missed the part about "non-FF compatible".

Perhaps in this case we should be taking advantage of assemblies to make sure we only load the correct and compatible sqlite3 from the proper location ?
That would be a platforn-specific way to solve the problem, however.

A more portable way would be for softoken to load sqlite3.dll dynamically, just like it does for freebl . It seems we only need 16 external symbols from it.

Comment 10

[Nelson Bolyard (seldom reads bugmail)]

This bug is already resolved WONTFIX, and I believe that is the right 
resolution. (That or INVALID)