Closed Bug 500495 Opened 15 years ago Closed 15 years ago

Upgrade Firefox 3.0.x to NSS version used by Firefox 3.5.1 (NSS 3.12.3.1)

Categories

(Firefox Build System :: General, defect)

3.0 Branch
defect
Not set
normal

Tracking

(status1.9.1 unaffected)

VERIFIED FIXED
Tracking Status
status1.9.1 --- unaffected

People

(Reporter: dveditz, Assigned: wtc)

References

Details

(Keywords: verified1.9.0.13, verified1.9.0.14, Whiteboard: [sg:critical])

Attachments

(3 files)

Firefox 3.0.x needs to upgrade to the version of NSS used and tested by Firefox 3.5 (nss 3.12.3) in order to pick up some security fixes for issues that will be discussed at this year's BlackHat.

In addition to the simple NSS tag change there may be PSM patches required to adopt the new version or to fix related bugs -- will have to investigate what went into Firefox 3.5

It would be nice to fix this in 3.0.x before the BlackHat talk (meaning 1.9.0.12), but maybe having a fixed 3.5 to point at is good enough and we can get this change in 1.9.0.13 around the end of August and still beat any active exploits of those problems. Maybe?

Drawbacks to fixing this in 3.0 include some regressions, and being able to point at the BlackHat talk will make it easier to explain them.

1) We fixed our non-standard wildcard behavior (bug 159483). This led to several duplicate bugs although everyone acknowledges we were the only browser to support that. Mostly an issue for linux-only shops where a mozilla-based browser is standard. bug 495339, bug 495602, bug 499122, bug 499647 and possibly more once we ship.

2) Bug 474606 -- EV sites with CRL rather than OSCP no longer green. The fix for that isn't until NSS 3.12.4.1 which 3.0 should eventually upgrade to in order to get FIPS certification, but isn't even in FF3.5 yet. This is probably too visible a thing to break in order to fix what looks like a non-problem before the blackhat talk. We'd at least need to take bug 485052

Since this is affecting user experience around EV I think Johnathan makes the call when we take these fixes.
Flags: wanted1.9.1.x-
Flags: wanted1.9.0.x+
Flags: blocking1.9.0.13?
Flags: blocking1.9.0.12?
Whiteboard: [sg:high]
Summary: Upgrade Firefox 3.0.x to NSS 3.12.3 → Upgrade Firefox 3.0.x to NSS version used by Firefox 3.5 (NSS 3.12.3)
I think that 3.12.3's had pretty good exposure, being the base for the 3.5 release, and present for some time in that release's beta channel.  But I think it's also potentially a Big Amount of Churn for a branch release - I don't know what kind of PSM changes rely on changes in 3.12.3 for instance, so I don't know how much code risk this patch would represent.

I don't want to put our users at risk, either, but having fixed this in 3.5 will go a long way there.

As for whether 3.0.x picks up 3.12.4 in order to get FIPS certification, I agree with what I think I hear Dan saying - that whether we do that or not, it's down the road and really needs more release exposure on 3.5 and trunk first.

If we take 3.12.3 but not 3.12.4, then yes, we'll be back in a situation where we regress some EV sites that use CRL-based certs, since we won't have support for checking the revocation there, but will start to insist on it for EV status. That's a crappy thing to do in a point release, and makes me wonder if we should really just get 3.12.4 into FF3.5.1, and then move straight to the 3.12.4 discussion based on the results of that. After all, for people who care about FIPS, 3.12.3 is not better for them than the tag we currently have.

So my questions, in no particular order:

1) How much code risk is this change, if we make it as described (move 3.0.x to a 3.12.3 NSS tag.) How much PSM churn will this require?  I guess that's a question for kaie or rrelyea.

2) Is it worth waiting until we can confidently move directly to 3.12.4? Larger churn that way, but we eliminate a known behavioural regression around EV, and also give our 3.0.x users actual CRLDP support. My mind's not at all made up, just asking.
That matches my gut feel: no way we can decide all this and cram it into 1.9.0.12. Setting blocking 1.9.0.13 so we come to a decision, but that decision might be to wait for 3.12.4

Switching NSS is easy, figuring out the corresponding PSM changes required might not be.
Flags: blocking1.9.0.13?
Flags: blocking1.9.0.13+
Flags: blocking1.9.0.12?
And now we've got buffer-overflow bug 504456 -- I think we've got to take 3.12.3 and forget waiting for 3.12.4
Depends on: 474606
Whiteboard: [sg:high] → [sg:critical]
What about Thunderbird 2, Seamonkey 1.1, and any linux distro still shipping Firefox 2? I don't think we can just drop 3.12 in in place of 3.11 can we? And no doubt this fix is in the FIPS-certified parts of NSS meaning we'll have to undo the double-pull build cruft.

Hopefully Thunderbird/SeaMonkey users don't care about FIPS certification anyway. I'll create a separate tracking bug for the Gecko 1.8 branch
Blocks: 484111
Blocks: 471539
> I don't think we can just drop 3.12 in in place of 3.11 can we? 

Yes.  

> no doubt this fix is in the FIPS-certified parts of NSS meaning we'll 
> have to undo the double-pull build cruft.

I'm not sure what this means.  
I know Red Hat is doing "double-pull build cruft" (a.k.a. Franken-NSS)
but I wasn't aware that Mozilla is doing that for its builds.  Is it?
Please elaborate.
(In reply to comment #4)
> Hopefully Thunderbird/SeaMonkey users don't care about FIPS certification
> anyway. 

They only care about FIPS 140 if they are in the US Federal government, in foreign governments that give preferential treatment to FIPS, or are in any of the US and non-US financial institutions that require it or give preferential treatment...

In other words, it's a large enough number of potential users that I hope they/we care. :-)
What's status of bug 474606? If dependencies are ok then bug 481968 (merging 1.9.1 with NSS 3.12.3) showed there were just a problem with random generator initiation on win ce.
(In reply to comment #5)
>  
> I know Red Hat is doing "double-pull build cruft" (a.k.a. Franken-NSS)
> but I wasn't aware that Mozilla is doing that for its builds.  Is it?
> Please elaborate.

Mozilla is building Franken-NSS in Firefox 2.0.0.x.  See bug 419030.
Adding Elio to CC list, who works on core NSS and is also working on Red Hat packaging of NSS.
No longer blocks: CVE-2009-2404
Depends on: CVE-2009-2404
Ok, just tested we can build mozilla cvs trunk with NSS_3_12_3_RTM tag (no build problems). Confirming there is still present bug 474606.
(In reply to comment #1)
> 1) How much code risk is this change, if we make it as described (move 3.0.x to
> a 3.12.3 NSS tag.) How much PSM churn will this require?  I guess that's a
> question for kaie or rrelyea.

There haven't been a lot of changes to security/manager recently.

cd mozilla-central/mozilla/security/manager
hg log .

I don't see anything specific to 3.12.3


If you want to reduce the effects of the PSM+EV regression that comes with 3.12.3, one could backport the OCSP-hack from bug 485052 to FF 3.0.x
(which would be unnecessary if we could go straight to something based on 3.12.4)
I think I would recommend picking up the latest version of 
security/manager/ssl/src/nsNSSCallbacks.{h,cpp} rather than trying to 
back port individual patches from them.
Firefox 3.0.x needs to use NSS version used by Firefox 3.5.1,
otherwise it will suffer from bug 501605 and also lose some
new root CA certs.  That version of NSS isn't an official NSS
release, so I proposed in bug 504611 that we do some release
engineering work to officially bless that version of NSS as
NSS 3.12.3.1.
Blocks: BH-2009
wait, why are we back to releasing Firefox from unofficial variants of NSS? Dammit, I knew that would happen when we put a copy of the NSS source in the mozilla repo -- right back to the bad old days of MOZILLA_CLIENT_BRANCH.

It's one thing to argue "convenience" for having it all in one repo, but it is not "convenient" to have bugs or regressions because we shipped something that hadn't gone through the NSS team's test suite and QA process. And once 3.12.4.x is FIPS certified shipping something that deviates from that could invalidate that certification.
Dan, Usually I share this concern, very much.  However, in this case, the NSS 
team gave a special dispensation of grace to FF due to the urgency of FF 3.5.1.  
Wan-Teh wants us to tag that exact version in the CVS tree, and I've agreed.  
I believe the plan is that 3.5.2 will ship NSS_3_12_4_RTM.  That was 
essentially the agreement we had when we agreed to let 3.5.1 ship a "special" 
NSS.
(In reply to comment #15)
> Dan, Usually I share this concern, very much.  However, in this case, the NSS 
> team gave a special dispensation of grace to FF due to the urgency of FF 3.5.1. 
> Wan-Teh wants us to tag that exact version in the CVS tree, and I've agreed.  
> I believe the plan is that 3.5.2 will ship NSS_3_12_4_RTM.  That was 
> essentially the agreement we had when we agreed to let 3.5.1 ship a "special" 
> NSS.

I do think the plan is to take 3_12_4 on the 3.5.x branch, but I think the churnings on 3.0.x have occupied a lot of our thinking lately, such that the questions I asked in bug 504080 about test visibility and crashes don't yet have answers in the bug. But I guess that is really more a topic for that bug than this one, so, to the matter at hand:

I agree that we should stay on known NSS tags wherever possible, and I agree with Wan-Teh's suggestion in comment 13 that we should pick up the fixes we've made to 3_12_3 post-RTM (the file i.o on windows, the new root certs).  In that vein, then, I think getting Wan-Teh to land the mini branch discussed in bug 504611 makes sense and that we should take that tag for 3.0.next (does it include the new roots? He mentions them here, but not in that bug).

As Kai mentions, the 3_12_3 release makes our revocation checking for EV more strict, and the result is that some sites will see downgraded status (from EV->DV). I think we should take the same fix we took on 3.5.x for this - as kai mentions in comment 11 (and the additional metadata added in other bugs - basically the current version of this block - http://mxr.mozilla.org/mozilla-central/source/security/manager/ssl/src/nsNSSCallbacks.cpp#1067 )
Depends on: 504611
Summary: Upgrade Firefox 3.0.x to NSS version used by Firefox 3.5 (NSS 3.12.3) → Upgrade Firefox 3.0.x to NSS version used by Firefox 3.5.1 (NSS 3.12.3.1)
Wan-Teh suggested a 3.12.3.x minibranch, and I reluctantly agreed to let him 
do it if he was motivated and had time, but there are problems with it.
#1 The employers of NSS's developers do not want it, and will not take releases 
from it. 
#2 NSS team doesn't have resources to maintain two or more branches in parallel.
#3 No-one who works on NSS apparently has enough time or motivation to do it.  
#4 We really don't want it to be a full branch from which numerous subsequent
releases are made, but (I fear) that is what MoCo wants.

As you know, MoCo anticipates doing a "fire-drill" release within the next 
10 days, to deal with things revealed at black hat.  The NSS team very much 
wants that release to include 3.12.4, which is where all the vulnerability 
fixes have been committed.  We're prepared to create a 3.12.4.5 RC tag for 
this purpose.  If MoCo now requests (as I expect it will) the NSS team to 
back-port all the recent vulnerability fixes to this proposed 3.12.3.x 
branch, it's not clear how that work would get done.  It's just not in the 
NSS team's plans.
Nelson: where's the testing matrix that was used for the 3.12.4 NSS branch? As we've mentioned before, we're interested in taking it, but it's hard to do so when (to my knowledge) there's been no beta release with the change.

In terms of backporting, it is distressing to hear (this late in the game) that it's not in the NSS team's plans. Is there any way we can assist here? Can we get a list of the patches so we can apply them to our tree specifically?
Also, it seems like bug 504080 should be resolved before this one, and that bug requires the issues in bug 504080 comment 1 to be addressed (or at least answered!). Otherwise we'd be, AIUI, regressing users as they upgrade from Firefox 3.0.x to Firefox 3.5.x, wouldn't we?

(note that bug 504080 has been on file for more than a week without any sort of response from the NSS team, which I find surprising in light of how they don't want to support multiple branches)
(In reply to comment #19)
> Also, it seems like bug 504080 should be resolved before this one, and that bug
> requires the issues in bug 504080 comment 1 to be addressed (or at least
> answered!). Otherwise we'd be, AIUI, regressing users as they upgrade from
> Firefox 3.0.x to Firefox 3.5.x, wouldn't we?

I believe beltzner meant bug 504080, comment #0.
(In reply to comment #17)
> #4 We really don't want it to be a full branch from which numerous subsequent
> releases are made, but (I fear) that is what MoCo wants.

Oh HELL no, "MoCo" does not want it's own private NSS version. I would think comment 14 and comment 16 are fairly clear on that.

> to deal with things revealed at black hat.  The NSS team very much 
> wants that release to include 3.12.4, which is where all the vulnerability 
> fixes have been committed.

I thought 3.12.3 contained all the fixes? This whole bug is predicated on thinking that the already-released Firefox 3.5.1 is in good shape (NSS-wise) so taking the known-shipped/tested NSS from that release was the safest thing to do. Taking a new NSS in a browser release that hasn't got any kind of a testing period, not even nightlies, isn't something we want to do unless there's no other choice.

Which of the bugs does Firefox 3.5.1 still suffer from?

> If MoCo now requests (as I expect it will) the NSS team to back-port
> all the recent vulnerability fixes to this proposed 3.12.3.x  branch,

Not a branch, a one-time release tag is all we would need. Is there more than what's already in bug 504611?
If you all agree, I will create the NSS 3.12.3.1 release as
I specified in bug 504611 comment 0.  It will have the new
roots.
I all agree. Please go ahead.
> "MoCo" does not want it's own private NSS version. 
Mike's comment 18 above strongly suggests otherwise to me.

Mike, suppose you got the private branch you want.  
Who would maintain it?
See comment 17, point 2.
Nelson: We intend to get back onto the actual NSS releases, but not this release. We need to ship something that we know is tested well and since "NSS 3.12.3.1" shipped with Firefox 3.5.1, we're reasonably sure it's safe. We do *not* want our own private branch (that's been said a lot), but rather our own tag for this release based on what we know is safe.

Can you also answer Dan's questions in comment 21?
Nelson, I think you're misreading my comment 18 pretty drastically. I'm interested in getting the patches we need into a pressing release vehicle in a way that doesn't compromise the quality or add untested (and thus risky code) to that release. I'm also interested in synchronizing with NSS, but have yet to get a straight answer about what if any testing NSS 3.12.4 has gone through.
Attached patch Proposed patchSplinter Review
Upgrade NSS from NSS_3_12_2_WITH_CKBI_1_75_RTM to NSS_3_12_3_1_RTM.
Attachment #391126 - Flags: review?(kaie)
Attachment #391126 - Flags: approval1.9.0.13?
Comment on attachment 391126 [details] [diff] [review]
Proposed patch

NSS 3.12.3 requires NSPR 4.7.4 or later:
http://www.mozilla.org/projects/security/pki/nss/nss-3.12.3/nss-3.12.3-release-notes.html

So the current NSPR_4_7_5_RTM tag is fine.
Comment on attachment 391126 [details] [diff] [review]
Proposed patch

Assuming that we have blessing for this tag, I'm fine with landing it:

r=kaie

We shall perform a sanity test build before landing it.
Attachment #391126 - Flags: review?(kaie) → review+
Comment on attachment 391126 [details] [diff] [review]
Proposed patch

I'm pretty sure Dan's working on landing this along with the PSM changes required. But in case he says he's not and this needs to land soon...

Approved for 1.9.0.13. a=ss for release-drivers
Attachment #391126 - Flags: approval1.9.0.13? → approval1.9.0.13+
I checked in the patch on the Mozilla 1.9.0 branch (CVS trunk).

Checking in client.mk;
/cvsroot/mozilla/client.mk,v  <--  client.mk
new revision: 1.394; previous revision: 1.393
done
Assignee: nobody → wtc
Status: NEW → RESOLVED
Closed: 15 years ago
Keywords: fixed1.9.0.13
Resolution: --- → FIXED
Wan-Teh, did your checkin include the OCSP fixes from bug 485052 and kin, as mentioned in comment 12 and comment 16?  I see that Dan has marked bug 485052 as blocking 1.9.0.13, so maybe he means for that to be tackled over there?

We definitely want to get that in - moving to this NSS tag will break EV UI in 3.0.x for several CAs that have already provided fixes in 3.5.x. That patch would be precisely the aggregate of bug 485052, bug 497184, and bug 497172. It will still break EV UI in 3.0 for SECOM trust, but they are broken in 3.5 as well, since they don't supply an OCSP responder and we don't support CRL fetch until NSS 3.12.4.
Johnathan: no, my checkin didn't include those PSM changes.  I can
back out my change.  Sorry.
(In reply to comment #33)
> Johnathan: no, my checkin didn't include those PSM changes.  I can
> back out my change.  Sorry.

I don't know if that's necessary - as long as we get the follow-up fix in to this release.
OK. I'll backport the OCSP fixes from bug 485052, bug 497184, and bug 497172
for you.
Flags: blocking1.9.0.13+
Attached file diff after merge
Because of the way the release automation works we need to merge the NSS change on to the GECKO190_20090706_RELBRANCH (we checkout using client.mk but then cvs up -r <relbranch>). Here are the steps I've used to set that up

mkdir GECKO190_20090706_RELBRANCH_merged
cd GECKO190_20090706_RELBRANCH
cvs co -r GECKO190_20090706_RELBRANCH mozilla/client.mk   mozilla/dbm mozilla/security/nss mozilla/security/coreconf mozilla/security/dbm
cvs co -jNSS_3_12_2_WITH_CKBI_1_75_RTM -jNSS_3_12_3_1_RTM mozilla/dbm mozilla/security/nss mozilla/security/coreconf mozilla/security/dbm

There's a bunch of
cvs checkout: scheduling mozilla/dbm/include/watcomfx.h for removal
in the middle of the merge output, so it's handling the removals and additions properly.

The diff I'm attaching here is generated with 
cvs -q diff -uN mozilla/dbm mozilla/security/nss mozilla/security/coreconf mozilla/security/dbm > attempted-merge.diff

If I do an rdiff like this
cvs -q rdiff -u -r NSS_3_12_2_WITH_CKBI_1_75_RTM -r NSS_3_12_3_1_RTM mozilla/dbm mozilla/security/nss mozilla/security/coreconf mozilla/security/dbm
then there is some output like this
cvs rdiff: failed to read diff file header /tmp/cvsJZSynf for BrAirWaysBadSig.cert,v: end of file
cvs rdiff: failed to read diff file header /tmp/cvsVFNIOe for OCSPCA1.cert,v: end of file
and so on for OCSPCA1.p12,v OCSPCA2.cert,v OCSPCA3.cert,v OCSPEE11.cert,v OCSPEE12.cert,v OCSPEE13.cert,v OCSPEE14.cert,v OCSPEE15.cert,v OCSPEE21.cert,v OCSPEE31.cert,v OCSPRoot.cert,v PayPalEE.cert,v. I'm guessing these are all binary files and rdiff is being dumb but need to double check.

Anyway, interdiff doesn't find any differences between the two diff methods, and a straight diff returns only change in the formatting of the header block for each file.

wtc, gavin, would you please take a look and verify this as you expect ? Then I'll go ahead and land on the relbranch.
Comment on attachment 391750 [details]
diff after merge

This is a bz2 file.
Attachment #391750 - Attachment is patch: false
Attachment #391750 - Attachment mime type: text/plain → application/octet-stream
Attachment #391750 - Attachment is patch: true
Attachment #391750 - Attachment mime type: application/octet-stream → text/plain
Attachment #391750 - Attachment is patch: false
Attachment #391750 - Attachment mime type: text/plain → application/octet-stream
I locally pulled NSS from both tags, using:

cvs co -r NSS_3_12_2_WITH_CKBI_1_75_RTM mozilla/dbm mozilla/security/nss mozilla/security/coreconf mozilla/security/dbm
cvs co -r NSS_3_12_3_1_RTM mozilla/dbm mozilla/security/nss mozilla/security/coreconf mozilla/security/dbm

and then diffed them with |diff -urN -xCVS NSS_3_12_2_WITH_CKBI_1_75_RTM/ NSS_3_12_3_1_RTM/|.

After taking into account minor differences in the diff format using a few different techniques (comparing sorted diffs, comparing their lsdiff output, etc.), I've concluded that my local diff and attachment 391750 [details] are identical.

Looks good to me!
Comment on attachment 391750 [details]
diff after merge

Landed on GECKO190_20090706_RELBRANCH. Thanks Gavin.
Keywords: fixed1.9.0.13
Gavin, Nick, could help us verify this for 3.0.13? Individual verifications for the bugs mentioned in comment #32 are in-progress for 3.0.13 / 3.5.2
The NSS guys would be better to advise you on that - perhaps they can point out a couple of easily tested bugs that were fixed between NSS_3_12_2_WITH_CKBI_1_75_RTM and NSS_3_12_3_1_RTM. You could also check the version on NSS dlls, like nss3.dll, as an indicator.
Yes, just check the version of NSS DLLs.

nss3.dll, softokn3.dll, and nssutil3.dll should be 3.12.3.1.

nssckbi.dll should be 1.75.
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.13) Gecko/2009073022 Firefox/3.0.13 (.NET CLR 3.5.30729)
nss3.dll -> 3.12.3.0
nssckbi.dll -> 1.75.0.0
nssutil3.dll -> 3.12.3.0
softokn3.dll -> 3.12.3.0

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729)
nss3.dll -> 3.12.3.0
nssckbi.dll -> 1.75.0.0
nssutil3.dll -> 3.12.3.0
softokn3.dll -> 3.12.3.0

Based on previous comments, I'm guessing this is unexpected...?
You need to check the version as follows.

1. Right-click over the DLL icon.  Select "Properties" in the drop-down menu.

2. In the "xxx.dll Properties" dialog, click the "Version" tab.

3. Select "File Version".  You should see "3.12.3.1 ...".
Verified on Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.0.13) Gecko/2009073022 Firefox/3.0.13 (.NET CLR 3.5.30729)
Also verified the version in the files mentioned in comment #42
Group: core-security
Flags: wanted1.9.1.x-
Verified for 1.9.0.14 using the nss3.dll file in Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.14pre) Gecko/2009081305 GranParadiso/3.0.14pre (.NET CLR 3.5.30729).
Status: RESOLVED → VERIFIED
Component: Build Config → General
Product: Firefox → Firefox Build System
You need to log in before you can comment on or make changes to this bug.