240784 - FIPS mode does not work on OS/2

Status: RESOLVED FIXED

(NSS :: Libraries, defect, P2)

Description

[Julien Pierre]

There are two issues.

The first bug is in the generation of the temporary filename that is used to
check the signature on the softokn3.dll file. The check file that NSS tries to
open should be softokn3.chk . But it turns out that on OS/2, NSS tries to open
softokn3.dll.chk . The bug is in mkCheckFilename in freebl . It needs to do a
case-insensitive comparison of SHLIB_PREFIX . The filename reported by OS/2 is
in CAPS, but the SHLIB_PREFIX is lowercase, which is why NSS tries to add .chk
to the full library name ...

This is enough to fix it in NSS. With this fix, and new copies of softokn3.dll
and softokn3.chk, I can now enable FIPS in PSM, which previously did not work.

However, modutil also has a bug. It calls PR_ACCESS and tries to find if the
directory for secmod.db is readable or writable. On OS/2 that's never the case,
so modutil always aborts. That's not technically a FIPS bug, but it got in my
way of testing my fixes.

Comment 1

[Julien Pierre]

Attachment: Do case-insensitive comparison of extension against DLL name

Comment 2

[Julien Pierre]

Attachment: Fix for modutil not to check for writable/readable directories on OS/2

Comment 3

[Julien Pierre]

Comment on attachment 146343 [details] [diff] [review]
Fix for modutil not to check for writable/readable directories on OS/2

Ignore the temporary variable names I added; I will delete them before
check-in. I used them for debugging since I couldn't display the value of
macros in the debugger and the values were passed to functions without debug
info, in the C RTL.

Comment 4

[Wan-Teh Chang]

Comment on attachment 146342 [details] [diff] [review]
Do case-insensitive comparison of extension against DLL name

Case-insensitive comparison should only be
used on file systems that have case-insensitive
file names.

This change should be ifdef'd for OS2 for now.

Comment 5

[Mike Kaply [:mkaply]]

I don't understand your directory change.

You can certainly have directories that are read only on a network drive.

Comment 6

[Mike Kaply [:mkaply]]

Also, why wouldn't you just fix PR_Access on OS/2, not #ifdef NSS?

Comment 7

[Julien Pierre]

Wan-Teh,

Thanks for your feedback. I'm not sure if there is a way to query if the file
system is case-sensitive. Mike, do you know ? All the local file systems that
come with OS/2 (FAT16, HPFS, and JFS) are case-insensitive. FAT16 isn't
case-preserving; only upper-case is supported. Remote file systems could pose a
problem. I will try to setup NFS, which does come with OS/2 also, and see how it
behaves when mounting file systems from my newest computer acquisition, a used
Sun Ultra 60.

Mike,

The directory change is because I found that PR_Access always return SECFailure
when asked about read/write status for directory. I was doing this on a local
drive, without any permissions. The only test that I founed worked with
directories is "exists". PR_Access just calls the C RTL "access" function . I
got the same behavior under both gcc and VACPP runtimes. We should indeed
investigate why that's the case. I had to make this patch to get past the
problem however.

If we want to #ifdef that change for OS/2, PR_Access would have to know that the
name being queried is a directory, as opposed to a file. Currently it doesn't.
Doing that would involve making additional system calls and is a more extensive
patch. It might be something worth considering, and should probably be the
subject of a separate bug about PR_Access .

As far as modutil is concerned, it isn't reading or writing data from/to the
directory. It is merely concerned about reading or writing secmod.db in the
specified directory. I think the tests are simply invalid, and should be removed
altogether rather, than #ifdef'ed for OS/2.

Comment 8

[Mike Kaply [:mkaply]]

There is no way to query if a file system is case sensitive that I have ever
heard of.

If this fix is important, it needs to be done by tomorrow.

Does it break anything important in the client that ever worked?

Comment 9

[Julien Pierre]

Mike,

It breaks the FIPS feature . This has apparently been broken since NSS 3.8 on
OS/2, which would be about 8 months, when the softokn3.chk file was introduced .
To enable FIPS, you go to Edit / Preferences / Privacy & Security / Certificates
Manage security devices / Enable FIPS .

When you enable FIPS, the browser is subject to a much stricter crypto policy
defined by the federal government . Some operations are forbidden, such as key
export. And you are required to re-login to the token every time you use a key.
This feature is be important only to federal government customers, particularly
the military and intelligence agencies.

Whether this is important for OS/2 is your call. I don't think anybody has been
shouting for this fix. I actually discovered the problem by trying to run the
"all.sh" NSS test suite on OS/2. I noticed that the FIPS test failed. Then I
tried it in the browser, and saw that it failed there too.

To fix all the problems I reported in this bug, including the modutil and
PR_Access issues, cannot be done by tomorrow.

To fix the library issue, which would fix the browser issue, could possibly be
done by tomorrow. The simplest fix wouldn't might not involve many code changes
you could ensure that the softokn3.dll gets packaged / zipped as uppercase, ie.
SOFTOKN3.DLL. The CHK file softokn3.dll should also be packaged as SOFTOKN3.CHK .

Then, you would need to ensure in the build that the DLL_SUFFIX macro is passed
to the softokn code as uppercase. All filesystems support uppercase to my
knowledge, so I believe this would work on all local and remote drives.

Comment 10

[Robert Relyea]

I agree with Wan-Teh about making the case insensitive compare based on either 
the OS expects case insensitive file systems. On OS/2 and Windows, even if you 
store file.DLL in a case sensitive file system, the OS will still treat it as a 
shared library.

We should add a flag 'CASEINSENSTIVE_FS' in the .mk file for Windows and OS/2 
(and other case insensitive OS's... is Mac one?) and make the new code 
conditional on that.

bob

Comment 11

[Mike Kaply [:mkaply]]

Do I need to put some version of this fix in for 1.7? Even an #ifdef XP_OS2 one
temporarily?

Comment 12

[Julien Pierre]

Mike,

I would say probably not. I don't think anybody using OS/2 depends on FIPS right
now. It's been broken for a while. You only want the fix short-term if you have
some federal agencies complaining about it not working. You make the call. If
the answer is no,we will fix this later in NSS 3.9.2 once we agree on the best
fix, and the browser will pick it whenever it picks it up that release of NSS.

Currently, the plan is to freeze NSS 3.9.1 on the code that Mozilla 1.7 rtm will
use.

If you decide that you really want the fix right now after all, then you can use
my fix with an ifdef . To fix the browser, you only need the first patch .
modutil isn't built as part of the browser, it's an NSS command-line tool, so
you don't need the second patch.

However, if you do that, please state that you are doing so, because we like to
have the browser pick up an official version of NSS, and not a modified one, so
we would want that temporary fix on NSS_3_9_BRANCH too.

Comment 13

[Julien Pierre]

FYI, I traced the NSPR function PR_GetLibraryFilePathname, which calls the OS/2
API DosQueryModuleName to obtain the filename of the softokn3 DLL that's
currently loaded. It returned a fully-qualified uppercase name (including path,
etc). I was running on JFS, a case-insensitive filesystem. I still need to run
on a case-sensitive FS (ie. NFS). I fear that there is a kernel bug here and
that the OS/2 system function always returns uppercase regardless of the actual
filename and pathname. I was running on JFS, and everything was lowercase on the
disk (path and filename). If that's the case, it would make it very difficult to
support FIPS if softokn3.dll/.chk were stored on case-sensitive file systems,
even we fix the case of the files, because we can't fix the case of the
directories, and therefore we would always fail to open the softokn3.chk file
due to improper case :-(

Comment 14

[Scott Garfinkle]

DosQueryModuleName returns all upper-case by design. All OS/2 IFSes must be
case-insensitive (case-preserving  is optional). If a case-sensitive filesys is
remotely available via NFS (or netwksta or whatever), then the IFS is
responsible for mapping names. If NFS on OS/2 allows two different files to
exist in the same directrory distingished only by capitalization, that's an NFS
bug, and should not really be a concern of the mozilla implementation.

Comment 15

[Mike Kaply [:mkaply]]

Wouldn't the real correct fix for the first part to be to define DLL_SUFFIX to
DLL instead of DLL in OS2.mk?

Comment 16

[Julien Pierre]

Scott, thanks for the info. That simplifies things a lot. I will test with NFS
to see what the result actually is, but that shouldn't be relevant to this
patch. I couldn't get OS/2 to mount NFS drives from my sparc last night.

Mike, given this information, I think indeed the following patch should work for
FIPS on OS/2.

Index: OS2.mk
===================================================================
RCS file: /cvsroot/mozilla/security/coreconf/OS2.mk,v
retrieving revision 1.18
diff -u -3 -r1.18 OS2.mk
--- OS2.mk      11 Sep 2003 20:29:51 -0000      1.18
+++ OS2.mk      27 Apr 2004 17:09:41 -0000
@@ -54,7 +54,7 @@
 
 # Override suffix in suffix.mk
 LIB_SUFFIX  = lib
-DLL_SUFFIX  = dll
+DLL_SUFFIX  = DLL
 PROG_SUFFIX = .exe

Comment 17

[Wan-Teh Chang]

Julien, we should add a comment to coreconf/OS2.mk to
explain why the value of DLL_SUFFIX needs to be
capitalized (DLL).  This is not obvious at all.  Thanks.

Comment 18

[Julien Pierre]

Attachment: make DLL_SUFFIX uppercase, with comment

Comment 19

[Mike Kaply [:mkaply]]

Comment on attachment 146343 [details] [diff] [review]
Fix for modutil not to check for writable/readable directories on OS/2

I still think we should do this by modifying PR_Access on OS/2 to return the
right thing.

http://lxr.mozilla.org/seamonkey/source/nsprpub/pr/src/md/os2/os2io.c#660

Comment 20

[Mike Kaply [:mkaply]]

Someone else will have to check this in, I don't have authority

Comment 21

[Mike Kaply [:mkaply]]

Comment on attachment 147669 [details] [diff] [review]
make DLL_SUFFIX uppercase, with comment

a=mkaply for 1.7

Comment 22

[Julien Pierre]

Checked in to NSS_3_9_BRANCH for 3.9.2 :

Checking in OS2.mk;
/cvsroot/mozilla/security/coreconf/OS2.mk,v  <--  OS2.mk
new revision: 1.18.16.1; previous revision: 1.18

As well as on the tip :
Checking in OS2.mk;
/cvsroot/mozilla/security/coreconf/OS2.mk,v  <--  OS2.mk
new revision: 1.20; previous revision: 1.19

I'm going to close this bug and open a tracking bug for the separate modutil issue .

Comment 23

[Wan-Teh Chang]

Comment on attachment 147669 [details] [diff] [review]
make DLL_SUFFIX uppercase, with comment

mkaply has checked in this patch on the
MOZILLA_1_7_BRANCH for Mozilla 1.7 final.