Update Mozilla to NSS 3.12 (was: NSS trunk fails to init with Mozilla trunk)

RESOLVED FIXED in 3.12

Status

defect
RESOLVED FIXED
12 years ago
11 years ago

People

(Reporter: kaie, Assigned: rrelyea)

Tracking

Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments, 1 obsolete attachment)

Reporter

Description

12 years ago
I assume the trunk of NSS is supposed to work with the trunk of Mozilla, at least for development purposes?

That combination does not work for me.
When I try to start up SeaMonkey, I get the error message about being unable to init NSS.

I traced a bit, and NSS fails to load a load (module->loaded == 0).

There seems to be some corruption going on, because after dismissing the error message, I get a crash (in Mozilla, not in NSS).

For testing I deleted the existing *.db files, but I get the same broken behavior.

(I'm currently running a Firefox build and will comment whether this gives the same error. I assume it will.)
Reporter

Comment 1

12 years ago
Same error with Firefox (although there is no follow-up crash).
Reporter

Comment 2

12 years ago
We arrive in SECMOD_LoadModule with the following modulespec:

(gdb) x/5s modulespec
0x17117af0:      "name=\"PSM Internal Crypto Services\" parameters=\"configdir='/home/kaie/.mozilla.org/seamonkey/vq3pgvbl.default' certPrefix='' keyPrefix='' secmod='secmod.db' flags=optimizeSpace  manufacturerID='Mozill"...
0x17117bb8:      "a.org' libraryDescription='PSM Internal Crypto Services' cryptoTokenDescription='Generic Crypto Services' dbTokenDescription='Software Security Device' cryptoSlotDescription='PSM Internal Cryptographi"...
0x17117c80:      "c Services' dbSlotDescription='PSM Private Keys' FIPSSlotDescription='Software Security Device (FIPS)' FIPSTokenDescription='FIPS 140 Cryptographic, Key and Certificate Services' minPS=0\" NSS=\"flags=i"...
0x17117d48:      "nternal,moduleDB,moduleDBOnly,critical\""

(gdb) bt 10
#0  SECMOD_LoadModule (
    modulespec=0x1fdf1ad0 "name=\"PSM Internal Crypto Services\" parameters=\"configdir='/home/kaie/.mozilla.org/seamonkey/vq3pgvbl.default' certPrefix='' keyPrefix='' secmod='secmod.db' flags=optimizeSpace  manufacturerID='Mozill"..., parent=0x0, recurse=1)
    at pk11pars.c:333
#1  0x00002aaab5b92933 in nss_Init (configdir=0x1f79a9c8 "/home/kaie/.mozilla.org/seamonkey/vq3pgvbl.default",
    certPrefix=0x2aaab5e86289 "", keyPrefix=0x2aaab5e86289 "", secmodName=0x2aaab5e86439 "secmod.db", readOnly=0, noCertDB=0,
    noModDB=0, forceOpen=0, noRootInit=0, optimizeSpace=1, noSingleThreadedModules=0, allowAlreadyInitializedModules=0,
    dontFinalizeModules=0) at nssinit.c:486
#2  0x00002aaab5b92b38 in NSS_InitReadWrite (configdir=0x1f79a9c8 "/home/kaie/.mozilla.org/seamonkey/vq3pgvbl.default")
    at nssinit.c:543
#3  0x00002aaab545fc17 in nsNSSComponent::InitializeNSS (this=0x1fdeed70, showWarningBox=1)
    at /home/kaie/moz/head-312/mozilla/security/manager/ssl/src/nsNSSComponent.cpp:1531


SECMOD_LoadModule attempts to do
332             moduleSpecList = SECMOD_GetModuleSpecList(module);

which produces a NULL.

(gdb) print PORT_GetError()
$9 = -5977
Maybe you don't have all of NSS's new shared libs in the product directory?
I'd expect behavior like this if any of NSS's new shared libraries were 
missing from the product directory where they all should reside.
Reporter

Comment 4

12 years ago
(In reply to comment #3)
> Maybe you don't have all of NSS's new shared libs in the product directory?

I don't think so...
My dist/bin directory contains the following files:

-rw-rw-r-- 1 kaie kaie     476 Jul 19 14:13 libfreebl3.chk
-rwxrwxr-x 1 kaie kaie  911713 Jul 19 14:13 libfreebl3.so
-rwxrwxr-x 1 kaie kaie 8906792 Jul 19 14:13 libnss3.so
-rwxrwxr-x 1 kaie kaie  913422 Jul 19 14:13 libnssckbi.so
-rwxrwxr-x 1 kaie kaie 1103261 Jul 19 14:13 libsmime3.so
-rw-rw-r-- 1 kaie kaie     476 Jul 19 14:13 libsoftokn3.chk
-rwxrwxr-x 1 kaie kaie  977814 Jul 19 14:13 libsoftokn3.so
-rwxrwxr-x 1 kaie kaie  930357 Jul 19 14:13 libssl3.so

Looks good I think.

$ LD_LIBRARY_PATH=. certutil -N -d bla

works fine

$ LD_LIBRARY_PATH=. certutil -d bla -S -n bla -s "CN=bla" -x -m 1 -t ,,

works fine

$ LD_LIBRARY_PATH=. certutil -L -d bla
bla                                                          u,u,u

works fine


I temporarily moved the NSS and NSPR files from dist/bin away to a different directory.
This overrides the local search path and will make mozilla use the ones I have installed globally as system libs.
This way I can successfully start up mozilla and connect to an SSL site.


So something must be broken related to the way Mozilla usually inits NSS.
Reporter

Comment 5

12 years ago
Bob, did you recently try this combination while you worked on shared db?

Note this is a 64 bit cpu.
Assignee: nobody → rrelyea
Assignee

Comment 6

12 years ago
Nelson's correct. you are missing libnssdbm3.so
I have a patch for PSM I'm working in now.

bob
Assignee

Comment 7

12 years ago
Attachment #273036 - Flags: review?(kengert)
Reporter

Comment 8

12 years ago
That works, thanks!
I should have been able to figure this out on my own :-/ from the contents of the nss-3.12 alpha that Bob had provided earlier...
Reporter

Comment 9

12 years ago
Comment on attachment 273036 [details] [diff] [review]
Update mozilla to NSS 3.12 [not ready for check in]

r=kengert
Attachment #273036 - Flags: review?(kengert) → review+
Reporter

Comment 10

12 years ago
Comment on attachment 273036 [details] [diff] [review]
Update mozilla to NSS 3.12 [not ready for check in]

While I'm ok with the changes in this patch, this patch alone is not sufficient.

This patch must be combined with additional patches.

We must ensure the new library gets shipped in all variations of mozilla products.

We must produce a patch that adds the new library to all places where the build system produces distribution packages.
Attachment #273036 - Attachment description: Update mozilla to NSS 3.12 → Update mozilla to NSS 3.12 [not ready for check in]

Comment 11

12 years ago
This patch increased our codesize numbers by 1MB. There are only three major changes listed in the 3.12 docs I could find.

 1.) libpkix
 2.) sqlite support
 3.) refactoring

The codesize numbers seem to point at libpkix, is this accurate? If so, why do we need it?
Robert, what sizes are you seeing? and on what platform?
Kaie's linux numbers in comment 4 are 2-x the numbers I get on Windows.
Those are the linux tinderbox numbers (http://tinderbox.mozilla.org/showbuilds.cgi?tree=Firefox)
With percentages of the increase:
Z: 1.14MB  - 8.8% increase (Firefox)
mZ: 0.974MB - 48% increase (Embed)
Nelson, the Firefox tinderbox has a codesize test running on Linux.  See the "Z" number on the fxnewref-linux-tbox tinderbox.  Looking at the full log for builds when this number changes will give you the exact data on which libraries are involved and which functions in those libraries.

In particular, for this checkin we have:

  libnss3.so
  	Total:	   +1038686 (+1046387/-7701)
  	Code:	    +952018 (+965127/-6519)
  	Data:	     +86668 (+81260/-1182)

with following that the exact functions involved and how big they are.  Then there are some other libraries, etc.

The Windows codesize has always been smaller than Linux -- the compiler just generates smaller code.  That said, it's usually smaller by the same factor across the board.  So an 8% increase in total shipping codesize (as here) would still be an 8% increase in total shipping codesize.

Oh, the mZ number for Firefox is meaningless because of the static build stuff.  Just ignore it.
Julien, please see comments 11-14.
Assignee

Comment 16

12 years ago
This patch includes the previous NSS 3.12, plus kai's patch from bug 388942 plus the updates for the package stuff.
Attachment #273036 - Attachment is obsolete: true
Attachment #273178 - Flags: review?(kengert)
For what it's worth, I just looked at the top function in that log (which is 30-some KB of code).  Some things that jumped out at me with a brief skim; these might be wrong observations:

1)  There is a lot of macro use, with all state stored on the stack.  Replacing stack vars with state structs, and the macros with functions, should help a good bit at least in this code (by at least letting the compiler optimization options control the codesize better).

2)  The standard "function start" macros seem to have some logging enabled by default and declare a whole slew of stack vars by default.  I assume these vars are actually used?

3)  PKIX_CHECK should really be a function (see item 1).  Note that there are about 2000 direct uses of this macro; even if you save a few dozen bytes per use you're looking at tens of thousands of kilobytes saved.  There are also some uses in other macros, some with branching factors.  For example, just the PKIX_RETURN macro invokes PKIX_CHECK twice.  There are 900-some PKIX_RETURNs in the code, adding another 1800 uses of PKIX_CHECK.

4)  Some code could hoist identical parts of if/else branches to use less codesize:

169                         if (pkixErrorCode == PKIX_FATAL_ERROR){ \
170                                 pkixErrorMsg = PKIX_ErrorText[descNum]; \
171                                 PKIX_RETURN(FATAL); \
172                         } else { \
173                                 pkixErrorMsg = PKIX_ErrorText[descNum]; \
174                                 goto cleanup; \
175                         } \

(although maybe "cool enough" compilers do that?  Do we use any such?).  That code is in PKIX_CHECK, but I think I saw other examples... and in any case, while PKIX_CHECK is being inlined all over this is certainly a problem.

What are the prospects, if any, of getting some codesize work on this library before it lands on trunk again?
Reporter

Comment 18

12 years ago
Boris, in short, there is desire from Firefox to verify a new class of certificates, providing enhanced identity verfication. This requires verification of certificate policies and verification paths are no longer simple top down trees, but networks with multiple possible paths. In order to support this new kind of complex functionality without having to reinvent the wheel, the NSS team decided to include libPKIX which offers these features.
Reporter

Comment 19

12 years ago
Comment on attachment 273178 [details] [diff] [review]
Combined checkin patch

Patch looks ok, but I'm worried the list of packaging changes is incomplete.
Attachment #273178 - Flags: review?(kengert) → review+
Reporter

Comment 20

12 years ago
Bob, I scanned through the sources. I believe you want these additional changes.

I left out two Mac files, because I don't know how to add the files there:
 mozilla/embedding/tests/cocoaEmbed/CocoaEmbed.pbproj/project.pbxproj
 mozilla/camino/Camino.xcodeproj/project.pbxproj

I hope a Camino person can help out after you land.

I left out mozilla/directory/c-sdk which seems to be incomplete already.

Comment 21

12 years ago
(In reply to comment #20)
> I hope a Camino person can help out after you land.

I'll take care of adding this to Camino once it lands.
Kai, I know what the situation is.  You apparently completely missed the point of my comment.  Please see the discussion on this topic in m.d.planning, ok?
Alexei, please see comments 11-17.
Reporter

Comment 24

12 years ago
(In reply to comment #12)
> Robert, what sizes are you seeing? and on what platform?
> Kaie's linux numbers in comment 4 are 2-x the numbers I get on Windows.

Should I have mentioned my numbers from comment 4 were from a debug build?
Kai: yes, we always need to be clear about debug vs optimized numbers.
IMO, the only numbers that matter are the optimized numbers.

Boris: WHAT discussion in m.d.planning?  I don't see any.
(In reply to comment #26)
> "NSS 3.12 codesize hit (Was: Milestone Scheduling)", starting from
> http://groups.google.com/group/mozilla.dev.planning/msg/14ec6101b26291aa

Why can't Google Groups find this message or thread when I enter NSS code size and the like and then Search / This Group? Anyone know the secret to searching?

/be
Thanks, David,  
I didn't see it because it's buried down inside another larger thread 
with the subject name "Milestone Scheduling", which I was not reading.

Comment 29

12 years ago
Erm, this appears to have landed this morning; is there some reason the bug hasn't been updated to reflect that?
I filed bug 389343 on the codesize issue.
Depends on: 389411

Comment 31

12 years ago
I'm confused.  The summary is "NSS trunk fails to init with Mozilla trunk" but View Bug Activity says the summary was last changed to "Update mozilla to NSS 3.12 [not ready for check in]".
Jesse, it was a patch description that was changed to "Update mozilla ..."

Updated

12 years ago
Summary: NSS trunk fails to init with Mozilla trunk → Update Mozilla to NSS 3.12 (was: NSS trunk fails to init with Mozilla trunk)
Depends on: 390451
It looks like this has caused bug 390451

Updated

11 years ago
Status: NEW → RESOLVED
Last Resolved: 11 years ago
Resolution: --- → FIXED
Target Milestone: --- → 3.12
You need to log in before you can comment on or make changes to this bug.