Closed Bug 504523 Opened 15 years ago Closed 15 years ago

Thunderbird 2 needs NSS 3.12.3.1

Categories

(Firefox Build System :: General, defect)

Product:
Firefox Build System
Component:
General
Version:
1.8 Branch
Type:
defect
Priority:
Not set
Severity:
critical

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dveditz, Assigned: dveditz)

References

Details

(Keywords: fixed1.8.1.23, Whiteboard: [sg:critical])

Attachments

(5 files, 2 obsolete files)

shared sqlite3 builds on windows (PSM/NSS doesn't load)
105.80 KB, patch
Details | Diff | Splinter Review
apply on top of attachment 392939 to get nssdbm3
2.35 KB, patch
Details | Diff | Splinter Review
package list changes for the new NSS libraries
6.99 KB, patch
Details | Diff | Splinter Review
Try to make it build with static rather than shared libs
5.68 KB, patch
Details | Diff | Splinter Review
Backport visibility fix to unbreak Camino
724 bytes, patch
Details | Diff | Splinter Review 
Gecko 1.8 products, Thunderbird 2 and SeaMonkey 1.1 in particular, need patches contained in NSS 3.12.3 to fix several SSL bugs will be revealed at BlackHat at the end of July.

We could backport individual fixes perhaps, but it would be safer to take the entire, tested NSS 3.12.3 release. Either way the fixes will be changing the FIPS-certified parts of NSS, but I'm not sure we really care if Thunderbird or SeaMonkey is FIPS-certified. If we do maybe cherry-picking the patches would be more acceptable since it's a smaller change.

This may also affect Linux distros still shipping Firefox 2, but I believe they already ship using "system NSS" and they'll be able to upgrade NSS when these issues come out.

Bugs this will fix include bug 159483, bug 480509, and bug 504456
Flags: wanted1.8.1.x+
Flags: wanted1.8.0.x?
Flags: blocking1.9.1.1-
Flags: blocking1.8.1.next+
Flags: blocking1.8.0.next?
Whiteboard: [sg:critical]
Version: unspecified → 2.0
also bug 471539 and bug 484111
I'm actually building and running TB2 against 3.12.3 since it's available and haven't noticed any regression. Are there known compat issues with PSM on 1.8.1 and NSS 3.12.3?
Regarding FIPS-140, my main interest is seeing TB3+ use FIPS-140 validated versions of NSS.  TB2 would be nice, but I don't think I can argue that it's critical.
We need the not-yet-created 3.12.3.1 in order to pick up some fixes that went into Firefox 3.5.1
Depends on: 504611
Summary: Thunderbird 2 needs NSS 3.12.3 → Thunderbird 2 needs NSS 3.12.3.1
This upgrades the 1.8 branch to use NSS 3.12.3.1 as used by FF3.5.1, which required undoing the franken-build cruft required to preserve the fips-certified bits. Also required upgrading to NSPR 4.7.4 or higher, so I went with the 4.7.5 as used and tested in Firefox 3.0.x
Attachment #391227 - Flags: review?(kaie)
Attachment #391227 - Flags: approval1.8.1.next?
Does Thunderbird 2 need the fix for bug 506407 ?   (I think it does)
The fix to that is NOT in 3.12.3.1
Comment on attachment 391227 [details] [diff] [review]
Upgrade NSS to the FF3.5.1 version

This has build issues, working on it.
Attachment #391227 - Attachment is obsolete: true
Attachment #391227 - Flags: review?(kaie)
Attachment #391227 - Flags: approval1.8.1.next?
It would be very nice to pick up the fix for bug 506407, but it seemed pretty clear that the NSS team doesn't have the resources to produce another 3.12.3-based tag. We'll pick up 3.12.4 when the active Firefox branches do, that'll have to be soon enough.
Assignee: nobody → dveditz

        Attachment #392619 -
        Flags: approval1.8.1.next?

    
    

    
  
Checked in and everything went up in flames.

Mac:
/builds/tinderbox/Tb-Mozilla1.8-Nightly/Darwin_8.7.0_Depend/mozilla/nsprpub/pr/src/md/unix/unix.c: In function `_MD_unix_get_nonblocking_connect_error':
/builds/tinderbox/Tb-Mozilla1.8-Nightly/Darwin_8.7.0_Depend/mozilla/nsprpub/pr/src/md/unix/unix.c:3295: error: `socklen_t' undeclared (first use in this function)
/builds/tinderbox/Tb-Mozilla1.8-Nightly/Darwin_8.7.0_Depend/mozilla/nsprpub/pr/src/md/unix/unix.c:3295: error: (Each undeclared identifier is reported only once
/builds/tinderbox/Tb-Mozilla1.8-Nightly/Darwin_8.7.0_Depend/mozilla/nsprpub/pr/src/md/unix/unix.c:3295: error: for each function it appears in.)
/builds/tinderbox/Tb-Mozilla1.8-Nightly/Darwin_8.7.0_Depend/mozilla/nsprpub/pr/src/md/unix/unix.c:3295: error: parse error before "optlen"
/builds/tinderbox/Tb-Mozilla1.8-Nightly/Darwin_8.7.0_Depend/mozilla/nsprpub/pr/src/md/unix/unix.c:3296: error: `optlen' undeclared (first use in this function)

Linux:
/usr/bin/gmake -j1: *** No rule to make target /builds/tinderbox/Tb-Mozilla1.8-Nightly/Linux_2.4.18-14_Depend/mozilla/dist/lib/libsoftokn3.chk.  Stop.

Windows:
nsNSSIOLayer.cpp
e:/builds/tinderbox/Tb-Mozilla1.8-Nightly/WINNT_5.0_Depend/mozilla/security/manager/ssl/src/nsNSSIOLayer.cpp(1817) : error C2664: 'DER_Lengths' : cannot convert parameter 3 from 'unsigned long *' to 'unsigned int *'
        Types pointed to are unrelated; conversion requires reinterpret_cast, C-style cast or function-style cast

    
    

    
  
Maybe wtc has some insight into the build bustage?

    
    

    
  
What version of NSPR is in the TB2 tree? 
Maybe it's not new enough.

    
    

    
  
Dan bumped it

    
    

    
  
... from 4.6.8 to 4.7.5 already.

    
    

    
  
On Mac, what version of XCode are you using?

On Linux:  need to see a little more context for that error.  
In what directory what make running when it output that error?
This looks like a dependency problem.  I'm guessing some parts of NSS didn't
get rebuilt.  Need to do FULL rebuild of NSS.  

On Windows, compare that source file with 
http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/security/manager/ssl/src/nsNSSIOLayer.cpp&rev=1.165&mark=2008#2002
Please make that change to the 1.8 tree, too.

    
    

    
  
OK, made the nsNSSIOLayer.cpp change.

The linux build error is here
http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla1.8/1249438440.1249438752.7006.gz

Apparently I grabbed the wrong (second) error, the first problem is in nss's copy of sqlite

gmake[5]: Entering directory `/builds/tinderbox/Tb-Mozilla1.8-Nightly/Linux_2.4.18-14_Depend/mozilla/security/nss/lib/softoken'
gcc -o Linux2.4_x86_glibc_PTH_OPT.OBJ/sdb.o -c -O2 -fPIC -DLINUX1_2 -Di386 -D_XOPEN_SOURCE -DLINUX2_1  -ansi -Wall -Werror-implicit-function-declaration -Wno-switch -pipe -DLINUX -Dlinux -D_POSIX_SOURCE -D_BSD_SOURCE -DHAVE_STRERROR -DXP_UNIX -DSHLIB_SUFFIX=\"so\" -DSHLIB_PREFIX=\"lib\" -DSOFTOKEN_LIB_NAME=\"libsoftokn3.so\" -DSHLIB_VERSION=\"3\" -UDEBUG -DNDEBUG -D_REENTRANT -DNSS_ENABLE_ECC -DUSE_UTIL_DIRECTLY -I/builds/tinderbox/Tb-Mozilla1.8-Nightly/Linux_2.4.18-14_Depend/mozilla/dist/include/sqlite3 -I/builds/tinderbox/Tb-Mozilla1.8-Nightly/Linux_2.4.18-14_Depend/mozilla/dist/include/nspr -I/builds/tinderbox/Tb-Mozilla1.8-Nightly/Linux_2.4.18-14_Depend/mozilla/dist/include -I../../../../dist/public/nss -I../../../../dist/private/nss -I../../../../dist/include -I/builds/tinderbox/Tb-Mozilla1.8-Nightly/Linux_2.4.18-14_Depend/mozilla/dist/include/dbm -I../../../../dist/public/dbm  sdb.c
sdb.c: In function `sdb_FindObjectsInit':
sdb.c:707: error: implicit declaration of function `sqlite3_prepare_v2'
NEXT ERROR gmake[5]: *** [Linux2.4_x86_glibc_PTH_OPT.OBJ/sdb.o] Error 1
gmake[5]: Leaving directory `/builds/tinderbox/Tb-Mozilla1.8-Nightly/Linux_2.4.18-14_Depend/mozilla/security/nss/lib/softoken'

I don't know why that's "implicit", sqlite3_prepare_v2 is in sqlite3.h which is included by that file.

    
    

    
  
I don't want to muddy the water too much, given that there are three different errors already, but Camino's generating a different error than Mac Tb (and Sm).

We're using Xcode 2.5 on 10.4.10 Intel (also note we have higher system requirements than the stock 1_8 branch, namely 10.3.9 SDK and gcc4.0 for PPC); this is a clobber build: http://tinderbox.mozilla.org/showlog.cgi?log=Camino/1249436100.1249436273.12020.gz&fulltext=1

Note that we get past where the Tb and Sm Mac builds are failing (unix.o) and fail when compiling prsystem.o.

When did NSPR and NSS drop support for 10.2/10.3 and those SDKs? I have a vague, maybe incorrect, memory of seeing comments to that effect in a couple of bugs over the past year or so.

    
    

    
  
when MOZILLA_CLIENT is defined (it is) nss uses the sqlite from the mozilla client rather than its own copy. The nss copy is version 3.3.17 and has sqlite_prepare_v2(), the 1.8-branch client copy is version 3.3.5 and does not.

    
    

    
  
the nsNSSIOLayer.cpp change was from bug 397296 -- making this bug depend on that one.

The windows failure is because it's trying to use sqlite3 as a shared lib, but on 1.8 it was built as a static lib. That was changed in bug 306907
Depends on: 397296, 306907

    
    

    
  
I'm getting a build failure for Firefox 2 on Mac at least:

/usr/bin/make -j1: *** No rule to make target /work/mozilla/builds/1.8.1/mozilla/firefox-opt/dist/lib/libsoftokn3.chk.


I know Firefox 2 isn't supported by I use it to test the JS engine on 1.8.

    
    

    
  
The compilation errors of NSPR's unix.c and prsystem.c on the Mac could
be caused by old versions of Mac SDKs.  Unfortunately I don't remember
when NSPR dropped support for old Mac SDKs.  These compilation errors
may have crept in without our dropping support for old Mac SDKs intentionally.

    
    

    
  
re sqlite_prepare_v2().

Executive summary: It should be OK to map sqlite_prepare_v2 to the old sqlite_perpare. The older interface will not return as specific of error codes on failure, and will not automatically recompile the sqlite request on database change. The signatures are exactly the same.

NOTE: the 1.8 branch will not use the sqlite database by default, so only early adopters are likely to run into problems with any issues with the sqlite NSS databases. Those people are likely already on some from of Thunderbird 3.

Here's the documenation from the sqlite website:

The sqlite3_prepare_v2() and sqlite3_prepare16_v2() interfaces are recommended for all new programs. The two older interfaces are retained for backwards compatibility, but their use is discouraged. In the "v2" interfaces, the prepared statement that is returned (the sqlite3_stmt object) contains a copy of the original SQL text. This causes the sqlite3_step() interface to behave a differently in two ways:

   1. If the database schema changes, instead of returning SQLITE_SCHEMA as it always used to do, sqlite3_step() will automatically recompile the SQL statement and try to run it again. If the schema has changed in a way that makes the statement no longer valid, sqlite3_step() will still return SQLITE_SCHEMA. But unlike the legacy behavior, SQLITE_SCHEMA is now a fatal error. Calling sqlite3_prepare_v2() again will not make the error go away. Note: use sqlite3_errmsg() to find the text of the parsing error that results in an SQLITE_SCHEMA return.

   2. When an error occurs, sqlite3_step() will return one of the detailed error codes or extended error codes. The legacy behavior was that sqlite3_step() would only return a generic SQLITE_ERROR result code and you would have to make a second call to sqlite3_reset() in order to find the underlying cause of the problem. With the "v2" prepare interfaces, the underlying reason for the error is returned immediately.

    
    

    
  
(In reply to comment #22)
> The compilation errors of NSPR's unix.c and prsystem.c on the Mac could
> be caused by old versions of Mac SDKs.  Unfortunately I don't remember
> when NSPR dropped support for old Mac SDKs.  These compilation errors
> may have crept in without our dropping support for old Mac SDKs intentionally.

The prsystem.c failure on the Camino tinderbox, PPC building with 10.3.9 SDK and gcc4.0 (http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/nsprpub/pr/src/misc/prsystem.c&rev=NSPR_4_7_5_RTM&mark=310#271 in NSPR_4_7_5_RTM, or 304 in cvs HEAD) points back to bug 454878; see bug 454878 comment 12 and bug 454878 comment 13.  That 'max_mem' item is clearly not in the 10.3.9 SDK version of the host_basic_info structure.

I don't understand the error in unix.c on the Tb/Sm boxen (PPC builds with gcc3.3 and 10.2.8 SDK), though; the function where the error is occurring is unchanged since 1999: http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/nsprpub/pr/src/md/unix/unix.c&rev=NSPR_4_7_5_RTM&mark=3295#3295  The actual source of the problem must be "upstream" from that yet, but I don't know where to go looking.

My Intel-only 1_8 branch Camino build (10.4u SDK, gcc4.0) does get all the way to NSS before failing with sqlite symbol errors mentioned above, which seems to confirm the Mac tinderbox failures we've seen so far are problems with NSPR and the PPC half's older SDKs.

    
    

    
  
(In reply to comment #24)
> My Intel-only 1_8 branch Camino build (10.4u SDK, gcc4.0) does get all the way
> to NSS before failing with sqlite symbol errors mentioned above, which seems to
> confirm the Mac tinderbox failures we've seen so far are problems with NSPR and
> the PPC half's older SDKs.

That implies that we can't upgrade NSPR on the 1.8 branch, which likely means we can't upgrade NSS on the 1.8 branch.

Dare I ask how much work it'd be to backport the necessary fixes from one NSS branch to another? ...

    
    

    
  
You can't upgrade the PPC SDKs?

    
    

    
  
I declare failure. NSS code says to use client's sqlite, but requires a function not in the version of sqlite on the 1.8 branch. We could fix nss to use its own version, but that's not using the tagged version anymore (though it's a build change rather than a code change) and doesn't solve the NSPR/Mac issues.

The other tack is to upgrade sqlite on the 1.8 branch. First, gotta switch it from a static lib to a shared lib. Then have to replace it with the version used by NSS (or later, but hopefully the NSS version is tested to work with NSS). That's introducing some change there. Then we have to reapply the patches we've made to sqlite because mozStorage calls functions that aren't in the NSS version of sqlite.

And it's still not building -- what's the next roadblock? And when we're done will we trust that Thunderbird or SeaMonkey or Camino is going to work?

How hard would it be to backport the NSS fixes to NSS 3.11.x ? at least changes in those areas are fresh in nelson's mind compared to trying to get info on 3 year old mozStorage issues.

    
    

    
  
(In reply to comment #26)
> You can't upgrade the PPC SDKs?

That will make Thunderbird no longer run on 10.2 -- not many people, to be sure, but security fixes shouldn't require folks to upgrade their OS.

    
    

    
  
backed out, someone else's turn to untoast our Tbird2 users

Checking in Makefile.in;
/cvsroot/mozilla/Makefile.in,v  <--  Makefile.in
new revision: 1.299.2.23; previous revision: 1.299.2.22
done
Checking in client.mk;
/cvsroot/mozilla/client.mk,v  <--  client.mk
new revision: 1.245.2.44; previous revision: 1.245.2.43
done
Checking in configure;
/cvsroot/mozilla/configure,v  <--  configure
new revision: 1.1492.2.133; previous revision: 1.1492.2.132
done
Checking in configure.in;
/cvsroot/mozilla/configure.in,v  <--  configure.in
new revision: 1.1503.2.114; previous revision: 1.1503.2.113
done
Checking in config/Makefile.in;
/cvsroot/mozilla/config/Makefile.in,v  <--  Makefile.in
new revision: 3.113.2.5; previous revision: 3.113.2.4
done
Checking in security/manager/Makefile.in;
/cvsroot/mozilla/security/manager/Makefile.in,v  <--  Makefile.in
new revision: 1.57.4.8; previous revision: 1.57.4.7
done
Checking in security/manager/ssl/src/nsNSSIOLayer.cpp;
/cvsroot/mozilla/security/manager/ssl/src/nsNSSIOLayer.cpp,v  <--  nsNSSIOLayer.cpp
new revision: 1.97.2.23; previous revision: 1.97.2.22
done
Assignee: dveditz → nobody

    
    

    
  
The Mac SDKs aren't backward compatible?  That's surprising.

NSS claims to be drop-in backward binary compatible. That means you can
take NSS 3.12.3 shared libs and drop them in as replacements for the older
3.11.x libs without recompiling, and they work (provided their dependencies
are new enough, such as sqlite and NSPR).  

I suggest you try it.  Take a TB2 installation, replace NSS, NSPR and sqlite3 
shared libs with those from a TB3 build, and try it.  Use Mozilla's sqlite3,
not NSS's because, as you've noted, Mozilla has some customizations of sqlite3
in its copy that are not in NSS's copy.  They don't bother NSS any, but 
Mozilla won't run with NSS's copy as-is.  

If it works, it's all software that you (Mozilla) built and for which you have sources.

    
    

    
  
(In reply to comment #24)
Smokey, the unix.c socklen_t compilation error is most likely caused by
this change to mozilla/nsprpub/configure.in in rev. 1.240:
http://bonsai.mozilla.org/cvsblame.cgi?file=mozilla/nsprpub/configure.in&rev=1.261&mark=986#982

    
    

    
  
(In reply to comment #30)
> I suggest you try it.  Take a TB2 installation, replace NSS, NSPR and sqlite3 
> shared libs with those from a TB3 build, and try it.  Use Mozilla's sqlite3,
> not NSS's because, as you've noted, Mozilla has some customizations of sqlite3
> in its copy that are not in NSS's copy.  They don't bother NSS any, but 
> Mozilla won't run with NSS's copy as-is.  
We used to change SQLite, but we don't anymore, and haven't since before Firefox 3.0 shipped (1.9.0).

    
    

    
  
> We used to change SQLite, but we don't anymore

OK, then I don't know why FF won't run with NSS's copy.  But it won't.

    
    

    
  
does TB 2 use mozStorage? Iirc, we ship with it but that may have been for extensions to use. Though we don't want to break Lightning users, for example...

How complicated are the NSS fixes? Nelson, is it realistic to backport them?

    
    

    
  
> How complicated are the NSS fixes? Nelson, is it realistic to backport them?

certainly more complicated than including a #define for sdb.c (or even better, in your sqlite3.h) to remap back to using the old prepare interface.:)

> We used to change SQLite, but we don't anymore, and haven't since before
> Firefox 3.0 shipped (1.9.0).

1.9 is on a new enough version. I'm not sure, however, that not updating is a good idea. sqlite continues to develop, and it's been my experience that they are *very* careful with the api (creating a new _v2 api which has improved semantics that could break old apps is an example, the old prepare still exists so old apps continue to work). That being said, for this release I would suggest simply updating your sqlite3 header to map the two functions to the same value.

bob

    
    

    
  
(In reply to comment #33)
> > We used to change SQLite, but we don't anymore
> OK, then I don't know why FF won't run with NSS's copy.  But it won't.
Because we used to change SQLite (in Firefox 2).

(In reply to comment #34)
> does TB 2 use mozStorage? Iirc, we ship with it but that may have been for
> extensions to use. Though we don't want to break Lightning users, for
> example...
I'm pretty sure satchel uses it.

    
    

    
  
(In reply to comment #36)

> I'm pretty sure satchel uses it.
I think TB 2 was still using wallet internally. But we probably shipped with satchel as well...

    
    

    
  


  
I tried making the 1.8-branch sqlite a shared library and faking the missing sqlite3_prepare_v2 entry point (hoping NSS didn't actually depend on the one-bit difference).

It builds on windows, but the resulting Thunderbird can't load NSS. Tried with a fresh profile, so it's not some migration issue -- I'm guessing NSS just can't use the older sqlite.

I should also note that I built using VC7.1 (msvs 2003) and results might be different using the official VC6 (Learned my lesson after relying on my personal Mac build results).

This patch also adds a new file to the build--sqlite3.dll/libsqlite3.so--that would have to be added to the packaging/installer files. But I'm back to thinking we need to backport the NSS patches to 3.11, even if just on the MOZILLA_1_8_BRANCH

        Attachment #392619 -
        Attachment is obsolete: true

        Attachment #392619 -
        Flags: approval1.8.1.next?

    
    

    
  
> It builds on windows, but the resulting Thunderbird can't load NSS. Tried with
> a fresh profile, so it's not some migration issue -- I'm guessing NSS just
> can't use the older sqlite.

More likely some sort of dll load issue on windows. Unless you explicitly ask for it, NSS will not even try to use the sqlite database (and 1.8 doesn't ask for it).

So if you didn't set the NSS_DEFAULT_DB_TYPE environment variable, there must be some windows DLL/OS issue that's preventing NSS for loading softoken. Is there an equivalent of ldd for windows which you can run on softoken?

bob

    
    

    
  
You can try
  dumpbin /dependents softokn3.dll

Dan, NSS 3.12.x has three new DLLs.  You discovered nssutil3.dll, whose
import library nssutil3.lib is needed at build time.  You already know
about sqlite3.dll.  The third one is nssdbm3.dll, which is only used
at run time.  Make sure you update mozilla/security/manager/Makefile.in
(and perhaps some other makefiles) to add nssdbm3.dll.  Please diff
mozilla/security/manager/Makefile.in against the revision on the CVS
trunk (for Mozilla 1.9.0) to see the required changes.

nssdbm3.dll is loaded using LoadLibrary() at run time, so dumpbin
/dependents won't report it.

    
    

    
  


  
adding nssdbm3 to the build got TB2 on windows running. Extremely minimal testing shows it can talk SSL to mail.mozilla.com

    
    

    
  
To maintain FIPS compatibility you'll also need a nssdbm3.chk file (from a comment in bug 489961 and email with Nelson). On mozilla-central (with NSS 3.2.4.4) it doesn't seem to be copied to dist/bin like the other check files. Still trying to figure that out.

    
    

    
  
Has anyone else tried this? Would love a sanity check. I'm also supposed to have left on vacation so someone else is probably going to have to land this.

    
    

    
  
We met this morning and discussed what we're going to do here.

  * Dan's going to checkin what he has here.
  ** That doesn't solve Mac (and maybe Linux), which will remain red while we 
     work on the next item
  ** Need *extra* verification that nothing SSL-related is broken on any
     platform.
  * We'll need to modify NSPR (probably on a new branch) to remove the changes 
    that cause Mac to not build / run on 10.2/10.3.
  * Mozilla Messaging is going to put some effort into getting everything
    working and ultimately tested since it's unclear what these changes will
    break.
  * QA (between MoCo and MeMo) will test this release, but more focused testing
    should happen by developers/QA which know Thunderbird best

I think that about sums it up, but let me know if I missed anything.

Wan-Teh: Do you have any idea of what NSPR changes we took in the latest NSPR that cause it to not be 10.2/10.3 compatible?

    
    

    
  


  
I've checked in again, and included these packaging list changes.

    
    

    
  
Note that the checkins didn't address comment 42, the reasoning being that since Firefox has shipped a couple of major versions without this and noone's screamed, it's not worth slowing down the effort to defuse this ticking bomb for.

    
    

    
  
Oh, right. FIPS. Somehow I missed comment 42. If we want Thunderbird users in the military (and likely other places) to update to this version of Thunderbird, we'd have to preserve FIPS. Otherwise they can't and are potentially vulnerable. Not sure how many users we're talking about though...

    
    

    
  
http://mxr.mozilla.org/mozilla-central/search?string=.chk&tree=mozilla-central

An examination of mozilla-central shows that there are MANY files outside of 
NSS that have lists of file names including .chk files.  I'll bet that one of 
them needs to be changed in order for the new .chk file to get propagated properly.

    
    

    
  
Samuel, the NSPR changes in question can be found in comment 24
and comment 31.  I will be happy to produce an official
NSPR_4_7_6_RTM release for you if you come up with the fixes.
If you have a Mac with the right build environment in your
Mountain View office, I can come over to fix these two bugs
for you.

    
    

    
  
Sam, if that were true, wouldn't we have heard about them refusing to upgrade to Firefox 3 and Firefox 3.5, since it would appear to have the same problem?

    
    

    
  
Dan: We've already heard of that problem. It's been reported quite a few times to us and it's why the NSS team is working on FIPS validation for a newer NSS (the branch in 3.0.x and 3.5.x, aiui).

Wan-Teh: If we don't figure it out over the weekend, I might take you up on that. I'm already home for the weekend though. Thanks for the offer. :)

    
    

    
  
Sam: I was under the impression that while they weren't excited about the lack of FIPS certification, they had not refused to upgrade, despite the new .chk files not being packaged.

One thing that's not clear to me is what the exact implications of the lack of the .chk file are.  Do they just mean we can't claim that it's "certified"?  Or is there specific functionality that will break?  If so, what functionality is it?

    
    

    
  
If the .chk file is missing, you can't put NSS in FIPS
mode.  Whether the specific version of NSS is FIPS
validated is another question.

    
    

    
  
Yeah, if the .chk files are missing, NSS won't start in FIPS mode. 
It will fail until you take it out of FIPS mode.

    
    

    
  
As far as I can tell, Firefox 3.0 contains nssdbm3 and nssutil and has never contained .chk files for those. Has anyone filed a bug that Firefox 3 doesn't work in FIPS mode, or have people simply not tried since it wasn't certified anyway?

Do we need a .chk file for sqlite3 as well?

    
    

    
  


  
One cause of build bustage on Firefox Windows is that default dev builds use shared libs, so I got it working for that (which make seamonkey go green) but static builds were still busted. I think this fixes it.

    
    

    
  
Could the Mac problems be connected with prebinding? I just by chance stumbled over the patches that removed prebinding support from mozilla-central, but on 1.8 I think we routinely enabled this for the PPC platform at least.

    
    

    
  
Dan, please try NSPR_4_7_1_RTM.  It doesn't have the changes
that break Mac OS X 10.2 and 10.3.

We should add all the necessary .chk files.  If nobody reports
missing .chk files, it means nobody is using the product in
FIPS mode.

We don't need a .chk file for sqlite3.

    
    

    
  
(In reply to comment #58)
> Dan, please try NSPR_4_7_1_RTM.  It doesn't have the changes
> that break Mac OS X 10.2 and 10.3.

Will that "work"?  Dan said in comment 5

> Also required upgrading to NSPR 4.7.4 or higher, so I went
> with the 4.7.5 as used and tested in Firefox 3.0.x

I didn't see any obvious dependencies between NSPR upgrades and NSS upgrades in the bugs for each on the 1.9.0 branch (but I also don't pretend to understand how NSS build-config tracks its dependencies).

    
    

    
  
(In reply to comment #58)
> Dan, please try NSPR_4_7_1_RTM.  It doesn't have the changes
> that break Mac OS X 10.2 and 10.3.

I tried that on the SeaMonkey tinderbox, and it compiles and the Ts/Txul/Tdhtml tests run fine, but none of them uses NSS, AFAIK.

Could someone with a Mac try http://ftp.mozilla.org/pub/mozilla.org/seamonkey/nightly/2009-08-09-04-mozilla1.8/seamonkey-1.1.18pre.en-US.mac.dmg and tell if it works with https and whatever NSS checks can be done on a casual test run?

    
    

    
  
Smokey:

Your question in comment 59 prompted me to look into the
NSPR upgrade more closely.  Here are my findings.

1. The upgrade from NSPR_4_6_8_RTM to NSPR_4_7_RTM is a
strict improvement -- you won't lose any bug fixes.  Note:
I was worried that we continued to patch NSPR 4.6.x after
NSPR 4.7 was released, but that wasn't the case.

2. NSS 3.12.x requires a new function added NSPR_4_7_RTM.
So NSPR_4_7_RTM is the absolute minimum requirement.  NSS
release notes are more conservative and document the version
of NSPR that was used during QA certification.

3. NSPR_4_7_1_RTM doesn't have the _darwin.cfg change that
broke JavaScript's (incorrect) x86<->ppc cross-compilation.
See bug 466531.  So NSPR_4_7_1_RTM doesn't require the JS
patch in bug 466531.  If we upgrade to NSPR_4_7_2_RTM or
later, we need to apply the JS patch in bug 466531.

4. I still think it's better to fix the Mac OS X 10.2/10.3
breakage that was inadvertently introduced.  But let's use
NSPR_4_7_1_RTM as a backup plan.  I'll run the NSS test
suite on the NSS 3.12.3.1/NSPR 4.7.1 combination on Linux,
Mac, and Windows.  I will also review the diffs between
NSPR 4.7.1 and NSPR 4.7.4 to see if there are any bug fixes
that NSS 3.12.3.1 requires.

    
    

    
  
Wan-Teh:
So you confirm that using NSPR_4_7_1_RTM for the moment should be OK and my build from comment #60 should be good? In that case, I think we should check in the switch to that version at least for now so that we can get testing of this whole NSS upgrade on Thunderbird builds as well, which should go green with that.

We still can investigate if we should or need move up to a fixed variant of NSPR 4.7.4, but I think having nightlies for people to test is better in the mean time than not having those.

    
    

    
  
Checked in a new client.mk that uses NSPR_4_7_1_RTM instead of 4.7.5

If Mac builds we need to test the heck out of the PPC version, and make sure the various NSS functionality works on all versions.  IMAP over SSL (e.g. mail.mozilla.com) and S/MIME ought to be enough of a sanity check, although I'm sure this is already all covered in the QA testplan.

    
    

    
  
(In reply to comment #63)
> Checked in a new client.mk that uses NSPR_4_7_1_RTM instead of 4.7.5

How does that mesh with this?

(In reply to comment #61)
> 4. I still think it's better to fix the Mac OS X 10.2/10.3
> breakage that was inadvertently introduced.  But let's use
> NSPR_4_7_1_RTM as a backup plan.

    
    

    
  
At this point, I expected all I'd need to do for Camino was fix packaging (and maybe linking the app); unfortunately, instead builds are (still) failing with missing sqlite3 symbols:

In static builds on the official release tinderbox (10.4, Xcode 2.5), linking libsoftokn3.dylib fails: 

http://tinderbox.mozilla.org/showlog.cgi?log=Camino/1249845540.1249846341.23237.gz

(note: this is fresh build; both the srcdir and objdir were deleted before this build began, after the previous build failed the same way)

In static builds on my own Mac (10.5, Xcode 3), the error is different: http://pastebin.mozilla.org/666327 (same error on the i386 half as well)

In (Intel-only) dev builds on my own Mac, we fail much earlier, building libstoragecomps: http://pastebin.mozilla.org/666326 (after a full distclean)

Do all of these symbols have different (wrong) visibilities in a shared lib as opposed to the former static lib, or are they getting stripped now, or?

Ah, yes, it looks like there were a host of visibility-related changes in bug 306907 that didn't get ported here.  I'm starting a build with the "VISIBILITY_FLAGS= " change in the sqlite Makefile, and the 2 headers in system-headers, but since sqlite on 1.9 is "just" one file, I suspect more changes will be required :-(

    
    

    
  


  
Turns out that the "VISIBILITY_FLAGS= " change in the sqlite Makefile was the only change required to unbreak libstoragecomps and NSS in the Camino build (sqlite{3|3file}.h aren't system headers on the Mac; I misread that part of bug 306907).

Since only Camino is building with hidden-visibility on 1_8, I believe this change is a no-op for everyone else (and we won't need to port any of the rest of the visibility fixes), but I don't have any way of building other platforms for any sort of sanity-check.

This patch, plus my project changes in bug 509342, should make Camino go green.

    
    

    
  
Mac now builds and is green, but it's not clear if it's okay to take NSPR 4.7.1 or if we need to upgrade to make everything work.

Wan-Teh: Thoughts on the above?

Regardless of what we do, we need to test this "setup" really well.

Smokey, if that has no affect on anything but Camino, feel free to check it in at any time.

    
    

    
  
kaie walked me through the steps of figuring out whether FIPS is actually working (thanks Kai!), and I tried it with Firefox 3.5.2 on OS X.  It only ships with libfreebl3.chk and libsoftokn3.chk.  Nonetheless, it allowed itself to be configured into FIPS mode and make SSL connections after that.  This would seem to suggest that there's an NSS bug around the .chk mechanism, and that Thunderbird 2.0.0.23 should benefit from the existence of that bug in the sense that we won't have to get the packaging mechanism working before we ship.

    
    

    
  
We only need .chk files for freebl3, softoken3, and in certain cases nssdbm3 (we never need the .chk file for nssutil).

The checking of nssdbm3 was a late addition to NSS 3.12.4 and may not be in the particular release candidate you have. It is also only checked if you are opening an old NSS database (though that is usually the case in FF).

The .chk file should be needed when we got to the final NSS 3.12.4 RTM, which is the actual FIPS validated version (we are holding RTM release for the lab filing, once the lab files, we will release the RTM). Since this bug is about NSS 3.12.3, we may be OK for now.

bob

    
    

    
  
Comment on attachment 393471 [details] [diff] [review]
Backport visibility fix to unbreak Camino

I've checked this in and will keep an eye on the trees to make sure Camino goes green and no-one else notices a thing ;)

    
    

    
  
Samuel: it is fine to use NSPR 4.7.1.  Yesterday I inspected the
diffs between NSPR 4.7.1 and NSPR 4.7.4 and went through all the
bug reports.  I didn't find any bug fix that NSS 3.12.x requires.
Since bugzilla.mozilla.org was down last night, I could post a
status update until today.  I still want to run the NSS test suite
on the NSS 3.12.3.1/NSPR 4.7.1 combination.

If you have the Mac OS X 10.2/10.3 build environment in your
Mountain View office, I can come over this afternoon around 4 pm
to fix the Mac problems with NSPR 4.7.5.  But it is perfectly fine
for you to use NSPR 4.7.1, unless I discover some problem when
running the NSS test suite.

    
    

    
  
(In reply to comment #70)
> (From update of attachment 393471 [details] [diff] [review])
> I've checked this in and will keep an eye on the trees to make sure Camino goes
> green and no-one else notices a thing ;)

Other than the unhappy coincidence of KaiRo's Windows box losing the ability to talk to cvs-mirror, all of the boxen are green.

    
    

    
  
Wan-Teh: I'll get a 10.3 build environment setup. If you have preferences on static/shared debug/opt, let me know so I can have it ready for you. My extension is 284 when you get here.

    
    

    
  
Question: Is there still concern at this point that customers who run TB2 in
FIPS-mode will have trouble with this upgrade?  

We're not likely to hear from people who run into trouble directly since their
normal internal escalation paths probably don't include contacting us. Also,
it's not practical to find out how many people are using TB2 (let alone in
FIPS-mode) because we're talking about very large departments. It's a common
problem with large, decentralized organizations.

[Off topic: My main ask is that TB3 ship with a FIPS-validated crypto module
that works in FIPS-mode.]

    
    

    
  
Samuel: I will build NSPR and NSS standalone, so I just need
the 10.3 SDK installed.  Thanks.

    
    

    
  
(In reply to comment #69)
> The .chk file should be needed when we got to the final NSS 3.12.4 RTM, which
> is the actual FIPS validated version (we are holding RTM release for the lab
> filing, once the lab files, we will release the RTM). Since this bug is about
> NSS 3.12.3, we may be OK for now.

Filed bug 509558 for packaging nssdbm3.chk for 3.12.4.

    
    

    
  
I now have patches to restore Mac OS X 10.2 and 10.3 support
in NSPR 4.7.x.  Thanks to Samuel for his help.  But the patches
require code review and more testing, so I suggest that we use
NSPR_4_7_1_RTM for now.

I'd also like to correct myself -- I found comments in two bug
reports (bug 454878 comment 13 and bug 469744 comment 0) that
showed I assumed 10.4 was the minimum Mac OS X version.  So it
was wrong for me to say I broke 10.2 and 10.3 inadvertently.

    
    

    
  
I have certified NSPR_4_7_1_RTM for use with NSS_3_12_3_1_RTM
on Linux, Mac, and Windows.

    
    

    
  
This bug is now fixed for 1.8.1.23.
Assignee: nobody → dveditz
Status: NEW → RESOLVED
Closed: 15 years ago
Keywords: fixed1.8.1.23
Resolution: --- → FIXED

    
    

    
  
Will this bug stay closed or will it be opened now that TB 2.0.0.23 is released?

    
  
Depends on: 512187

    
  
Product: Thunderbird → Core
QA Contact: build-config → build-config
Version: 2.0 → 1.8 Branch

    
  
Group: core-security

    
  
Product: Core → Firefox Build System

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: