173369 - Migrating NSS databases from the "CFM" build to the "mach-o" build

Status: VERIFIED FIXED

(Core Graveyard :: Security: UI, defect, P1)

Description

[Wan-Teh Chang]

The Mozilla client development team is planning to
change the binary executable format from CFM to
mach-o on the Mac.  There is more change than the
binary executable format.  The NSPR and NSS
implementation will also change from using the
classic Mac OS API to using the BSD Unix API in
Mac OS X.

We will need to migrate NSS's cert, key, and secmod
databases from the CFM build to the mach-o build.
At least, the names of these databases need to be
changed from "Certificate7", "Key Database3", and
"Security Modules" to "cert7.db", "key3.db", and
"secmod.db".  But more work may be required.
shadow reported in bug 167527 that the mach-o
build of certutil can't read the NSS databases
from the CFM build (after renaming).  Either the
Berkeley DB file format is not cross-platform or
there are some entries in the databases that also
need to be adjusted.

Comment 1

[Wan-Teh Chang]

Bob, could you look into this?

Comment 2

[Julien Pierre]

Certainly in the past I have moved databases between platforms - OS/2, Win32,
Solaris, so the format is cross platform, except for secmod.db of course which
references native PKCS#11 modules. I haven't tried yet on my Mac OS X box though
I did see renaming as an extra step.

Comment 3

[Robert Relyea]

NSS puts no platform dependencies into the databases. If there's a problem, it
would seem to be a problem with the implementation of libdbm on one of the Mac
platforms.

A useful test could be to see which version of the mac can read Win32 databases.
If the Mac CFM can, but the Mac OS X can't, then we have a bug in Mac OS X dbm.
If Mac CFM can't and Mac OS X can, then we have a bug on Mac CFM, and we will
have a pretty major problem. (Unfortunately I suspect the latter is the case).

If they both can read the Win32, then there was probably just some process
problem. In any case the code in question is almost certainly outside of NSS itself.

bob

Comment 4

[Wan-Teh Chang]

I copied the "server" databases generated by our
test script (all.sh) on Windows 2000 to a Mac OS
X machine.

I was able to use the mach-o build of certutil to
list the certs and verify the cert in the cert
database.  I could also run the mach-o build of
selfserv using those databases.  This shows that
the mach-o build of libdbm works.

Could one of you copy your cert and key databases
from Windows to your Mac and see if Netscape 7 (a
CFM app) can read them?  Remember to rename them
to "Certificate7" and "Key Database3". 

Comment 5

[Wan-Teh Chang]

If you have a Mac, could you perform the experiment
described at the end of comment #4?  Thanks.

Comment 6

[Julien Pierre]

I did this experiment yesterday on my Mac at home - I copied the DB from work
(win32). I could list the certs and use the keys for S/MIME as wlel.

Comment 7

[Wan-Teh Chang]

shadow, if you are using Mac as your primary platform,
could you copy your "Certificate7" and "Key Database3"
to a Windows PC, rename them "cert7.db" and "key3.db",
and see if you can use them in Netscape 7 or Mozilla
on Windows?

Julien, could you do some certificate management (such
as deleting and importing certs) on the Mac, copy the
"Certificate7" and "Key Database3" files back to a PC
and rename them, and see if you can still use them?

Comment 8

[Julien Pierre]

Sure. I'll try importing my certs with a brand new DB on the Mac, and see if I
can read that on OS/2.

Comment 9

[Wan-Teh Chang]

I just did the opposite of the experiment described
in comment #4 and verified that the cert and key db's
generated by the mach-o build of NSS on Mac OS X can
be used by certutil and selfserv on Windows.  This
shows again that the mach-o build of libdbm works.

Comment 10

[Julien Pierre]

OK. I created a new DB on the Mac, imported a P12 with about 12 certs, deleted a
CA cert.
Then I ftp'ed the Certificates7 and "Key Database 3" files to my OS/2 system and
created a new Mozilla profile, where they were copied as cert7.db and key3.db .
I could see the certs just fine under OS/2.

Comment 11

[Kai Engert (:KaiE:)]

I copied the database files from Linux to Mac, and Netscape 7 is able to work
with that database correctly.

Comment 12

[Conrad Carlen (not reading bugmail)]

I think we need to stick with the same filenames as the CFM build. The profiles
are the same between a CFM build and a Mach-0 build, so the names and locations
of these files need to be the same between these 2 builds. Granted, the CFM
names are  the oddball ones here, but it's more important that we're compatible
with people's existing Mac CFM profiles.

Comment 13

[Mike Pinkerton (not reading bugmail)]

we should keep the current names. user readable names for all files are a good
thing. let's not take a step backwards here just because it's "easier" for us.
we're not making this software for us, we're making it for our users.

Comment 14

[Wan-Teh Chang]

Mozilla users don't know the names of these NSS databases.
Management of the certificates, keys, and security modules
is done using the GUI under the menu item
Edit:Preferences:Privacy & Security:Certificates.  If
Mozilla users need to know the names of the NSS databases,
we have done something wrong.

If it is necessary to share profiles between the CFM and
mach-o builds, will these symbolic links work?
  
  Certificate7     -> cert7.db
  Key Database3    -> key3.db
  Security Modules -> secmod.db

Comment 15

[Robert Relyea]

1) These are hardly user friendly name ("Certificate7" is actually less
meaningful to a typical user than "cert7.db")
2) At least on my MAC, it thinks they are Microsoft XL files. They are obviously
are not meant for direct user manipulation. 
3) The patch name to get to the files are something like:
User:relyea:Library:Mozilla:Profiles:default:783b4tne.slt:Security -- Obviously
a path meant for a program, not a human, to traverse.  
4) The already contain tons of platform independent names (prefs.js,
panacea.dat, history.dat, etc.) in the profile directory.
5) The Mac is the only platform that does not share common names for the
security databases (in the old Comm 4.7 days there were different names for all
the various platforms. They have long since all converged on a single set of
names -- except the Mac).
6) We are operating on a real operating system at last. Mac OS X has the tools
needed to make this change realitively easy (by using hard or soft links to
equate the 2 file names, we can keep both Mac OS X and Mac OS 9 happy while we
transition).

bob

Comment 16

[Julien Pierre]

I agree with Bob and Wan-Teh, we should use the same filenames as we do on all
other platforms, and use symlinks to the old filenames if they exist to ease the
transition.

Comment 17

[Wan-Teh Chang]

Bob, I think NSS, rather than PSM, should create the
symbolic links I proposed in comment 14 because NSS
clients should not know the names of the NSS databases.

Could you implement this for Mozilla 1.3?

Comment 18

[Wan-Teh Chang]

Moved to target milestone 3.8 because the original
NSS 3.7 release has been renamed 3.8.

Comment 19

[Wan-Teh Chang]

Bob thinks that it is easier for PSM to create the symbolic
links, so I'm changing the product for this bug to PSM.  Kai,
could you talk to Bob about how this should be implemented?

Comment 20

[Kai Engert (:KaiE:)]

This must be fixed by Mozilla 1.3 final, since 1.3 will be released as macho only.
-> me

If we don't fix it, users will be believe they have lost all their certs.

In a chat with Wan-Teh, we agreed on the following strategy, which makes most
sense to us:

- on application start, prior to initializing NSS, call the convert routine
- if any of cert7.db, key3.db or secmod.db exists, we leave the convert routine
- if neither cert7.db, nor key3.db, nor secmod.db exists,
  we *copy*
  Certificate7     -> cert7.db
  Key Database3    -> key3.db
  Security Modules -> secmod.db

Comment 21

[Asa Dotzler [:asa]]

Users of 1.2 CFM builds upgrading to 1.3 Mach-O will lose all of their certs? If
so then this is certainly a blocker.

Comment 22

[Kai Engert (:KaiE:)]

Two more decision, that make sense I think:

- I'll try to copy Certificate8 to cert8.db, too.
  This allows a migration, even if people have used nightly CFM builds
  that already used Certificate8

- I'll only do the conversion, if neither of the 4 new files are available,
  neither key3.db, cert7.db, cert8.db nor secmod.db.
  This avoids a conversion, should users manually try to remove one of 
  those files only.

Comment 23

[Kai Engert (:KaiE:)]

Attachment: Patch

Comment 24

[Kai Engert (:KaiE:)]

Comment on attachment 114344 [details] [diff] [review]
Patch 

Javi, can you please review?

Comment 25

[Conrad Carlen (not reading bugmail)]

It's more efficient for the Mac file impl, and make this code more readable, to
start with machoPath, then for each cert file, use nsIFile::Clone and append to
the cloned file.

Comment 26

[Kai Engert (:KaiE:)]

Comment on attachment 114344 [details] [diff] [review]
Patch 

Thanks for the hint with Clone, I didn't see this method.
New patch will follow.

Comment 27

[Kai Engert (:KaiE:)]

Attachment: Patch v2

Updated patch, addressed Conrad's suggestion.

Comment 28

[Wan-Teh Chang]

Kai,

Are you sure that there are already Mozilla CFM builds
out there that create Certificate8?

If Certificate8 exists, it is only necessary to copy
Certificate8 to cert8.db; we do not need to copy
Certificate7 to cert7.db.

I suggest that you rename TryCFM2MachOConversion as
TryCFM2MachOMigration.

Please check with Bob Relyea again whether you should
copy "Security Modules" to secmod.db, start with
a blank secmod.db, or some file pathname conversion is
needed.  I understand that "Security Modules"
is the only NSS database that contains file pathnames.
Since the pathnames are different between CFM and Mach-O,
it may not make sense to copy them.

Comment 29

[Kai Engert (:KaiE:)]

> Are you sure that there are already Mozilla CFM builds
> out there that create Certificate8?

Yes, nightly Mozilla test builds were/are available in both CFM and macho format.


> If Certificate8 exists, it is only necessary to copy
> Certificate8 to cert8.db; we do not need to copy
> Certificate7 to cert7.db.

Ok, if Certificates8 was found, I will not copy Certificates7.


> I suggest that you rename TryCFM2MachOConversion as
> TryCFM2MachOMigration.

ok

Comment 30

[Kai Engert (:KaiE:)]

Attachment: Patch v3

(pending Bob's confirmation nothing else needs to be done wrt secmod path
names)

Comment 31

[Robert Relyea]

I forgot about the secdbm issue. Not only does the secdbm contain ':' paths (or
can contain ':' paths, the paths point to the .cfm pkcs #11 modules. The mach-O
builds will want to point to mach-O pkcs #11 modules, so the best strategy would
be not to convert secmod.db at all and let NSS create a new blank one.

The things you would loose:
   any installed pkcs #11 modules (of which I know of no vendor which has any
commertial .cfm pkcs #11 modules).
   whether or not you are in fips mode (you will revert back to non-FIPS mode).

bob

Comment 32

[Kai Engert (:KaiE:)]

Attachment: Patch v4

Bob, thanks for the explanation.
This new patch does not make an attempt to copy secmod files, and it does not
use the (non)existence of secmod files as a migration decision.

In a chat with Wan-Teh we came to the conclusion that it doesn't hurt to
migrate Certificates7, even if Certificates8 is already available. But if we do
migrate both in such a scenario, a few users will benefit, namely those who
play with combinations of both macho and cfm builds, with both different
versions between 1.2 and 1.3 (all combinations of 1.2, 1.3, cfm, macho exist).

Comment 33

[Kai Engert (:KaiE:)]

Comment on attachment 114362 [details] [diff] [review]
Patch v4

Javi, I think we are now done with the migration logic, could you please
review?

Comment 34

[Javier Delgadillo]

Comment on attachment 114362 [details] [diff] [review]
Patch v4

r=javi

Comment 35

[Kai Engert (:KaiE:)]

Comment on attachment 114362 [details] [diff] [review]
Patch v4

Peter, can you please review this Mac only patch?

Comment 36

[Peter Van der Beken [:peterv]]

Comment on attachment 114362 [details] [diff] [review]
Patch v4

>Index: mozilla/security/manager/ssl/src/nsNSSComponent.cpp
>===================================================================

>+  macho_key3db->AppendNative(cstr_key3db);

I think you should be using Append here?

>+  if (NS_FAILED(rv))
>+    return rv;
>+
>+  if (bExists)
>+  {
>+    cfm_key3->CopyToFollowingLinksNative(machoPath, cstr_key3db);
>+  }

Be consistent with bracing style. Sometimes you put braces around one line if's
and sometimes you don't. I prefer to have the braces, and that seems to be the
prevalent style in this file. Also put the opening brace on the same line as
the if.

>@@ -952,17 +1063,43 @@
>       return rv;
>     }
> 
>-  #ifdef XP_MAC
>-    // On the Mac we place all NSS DBs in the Security
>+  // XP_MAC == cfm
>+  // XP_MACOSX == macho

I think convention is CFM and MachO.

>+  #if defined(XP_MAC)
>+    // on cfm, cfmSecurityPath and profilePath are simply aliases

s/are simply aliases/point to the same oject/
alias has a double meaning on Mac, it is also a symbolic link.

>+    // ignore return code from conversion, we continue anyway
>+    TryCFM2MachOMigration(cfmSecurityPath, profilePath);

Why make it return nsresult then?

Comment 37

[Kai Engert (:KaiE:)]

Attachment: Patch v5

I have addressed all your change requests, and here is my answer to your
question:


> >+  macho_key3db->AppendNative(cstr_key3db);
> 
> I think you should be using Append here?

I'm not sure.
I'm dealing with hardcoded filenames for security storage information, 
and the filenames are simple 7 byte ASCII strings.

"AppendNative" takes AString, but "Append" takes ACString as a parameter.
		     ^^ 			 ^^^

I decided to go with AppendNative, because I'm trying to migrate some files,
and the old code that deals with the filenames uses the CString/AppendNative.

Is that an argument to use AppendNative for the migration routine, too?

Comment 38

[Kai Engert (:KaiE:)]

Comment on attachment 114826 [details] [diff] [review]
Patch v5

Peter, can you please review again? Please see the previous comment in answer
to your question.

Carrying forward r=javi.

Comment 39

[Peter Van der Beken [:peterv]]

Hmm, actually it's exactly the other way around, which is why I worried about
it. append takes AString while appendNative takes ACString. nsIFile.idl mentions
that for AppendNative the name needs to be in the filesystem's native charset,
but I guess that won't be an issue in this case because all the names are 7-bit
ASCII.

Comment 40

[Kai Engert (:KaiE:)]

Comment on attachment 114826 [details] [diff] [review]
Patch v5

requesting approval for this 1.3 final blocker bug

Comment 41

[Asa Dotzler [:asa]]

Comment on attachment 114826 [details] [diff] [review]
Patch v5

a=asa (on behalf of drivers) for checkin to 1.3.

Comment 42

[Kai Engert (:KaiE:)]

Patch checked in, marking fixed.

Comment 43

[John Unruh]

Verified fixed.