504080 - Upgrade 1.9.1 to an NSS 3.12.4

Status: RESOLVED FIXED

(Core :: Security: PSM, defect)

Description

[Johnathan Nightingale [:johnath]]

NSS 3.12.4 has some features, most notably CRLDP support, that we'd like to take on bch.  To do that, we'll need to know which tag to use, and then actually land it, but we're also trying to get better about understanding the impact of new tags before taking them, since much of the testing happens outside the regular gecko/firefox test infrastructure.

I think before we land this, we should:

a) Check the top crashers on central (where 3.12.4 has already landed), and ensure that any NSS crashers are well understood

b) Get a clearer picture of the unit test story, so that we know how covered the new code is (though this may spawn into its own bug around integrating/copying those tests and results more deeply into gecko's own testing structures)

not flagging for blocking1.9.1.x until we have answers there, but I thought I'd get the bug going anyhow.

Comment 1

[Peter Weilbacher]

Just for an additional (minor) incentive: taking NSS 3.12.4 would also finally fix a typo in NSS that has for a long time made an extra OS/2 patch necessary on branch (bug 487567).

Comment 2

[Wan-Teh Chang]

Attachment: Update NSS to NSS_3_12_4_FIPS4 in mozilla-central

We should start by pushing the latest NSS_3_12_4_FIPS4 CVS tag to
mozilla-central for some baking on the trunk.

Comment 3

[Nelson Bolyard (seldom reads bugmail)]

Isn't MC already running NSS_3_12_4_FIPSn  for some n < 4 ?

Comment 4

[Wan-Teh Chang]

Attachment: Diffs between NSS_3_12_4_FIPS1_WITH_CKBI_1_75 and NSS_3_12_4_FIPS4

mozilla-central is using NSS_3_12_4_FIPS1_WITH_CKBI_1_75.  That
tag was pushed to mozilla-central on 2009-05-22 (see bug 494236
comment 8).

Comment 5

[Nelson Bolyard (seldom reads bugmail)]

That tag is very similar to the current tag.  It's got all the CRLDP changes
IINM.  And MC has been running it for 60 days now without apparent grief.

Comment 6

[Wan-Teh Chang]

Attachment: Diffs between NSS_3_12_4_FIPS1_WITH_CKBI_1_75 and NSS_3_12_4_FIPS4 (omitting cmd, pkg, and tests for brevity)

I don't understand your point.  What's wrong with updating
NSS to the latest NSS_3_12_4_FIPS4 tag in mozilla-central?

Comment 7

[Kai Engert (:KaiE:)]

NSS_3_12_4_FIPS4 should be documented on https://wiki.mozilla.org/NSS:Tags

Comment 8

[Kai Engert (:KaiE:)]

Comment on attachment 391137 [details]
Update NSS to NSS_3_12_4_FIPS4 in mozilla-central

I think it's fine to give a newer NSS snapshot to Mozilla's development tree.

Assuming that the proposed tag is newer than what's currently used by mozilla-central:

r=kaie

Comment 9

[Kai Engert (:KaiE:)]

(Note that this bug asks for something newer for the stable 1.9.1, while Wan-Teh's current proposal is to update Mozilla's experimental trunk, only)

Comment 10

[Nelson Bolyard (seldom reads bugmail)]

Wan-Teh, in comment 2 you wrote:
> We should *start* by pushing the latest NSS_3_12_4_FIPS4 CVS tag to
> mozilla-central for some baking on the trunk.

as if we had not already started baking over 60 days ago.  
Please do not write things that seem to confirm the widely held but mistaken
belief that CRLDP is not already on Mozilla central and is not already baking.

Comment 11

[Wan-Teh Chang]

I just meant that pushing any NSS tag to mozilla-1.9.1 should
be preceded by pushing that tag to mozilla-central for "baking
on the trunk".  This is Mozilla's standard development procedure.
I didn't imply anything about CRLDP.  This bug's summary is
"Upgrade 1.9.1 to an NSS 3.12.4" and my comment 2 should be
interpreted as follows:

  We should start [Upgrade 1.9.1 to an NSS 3.12.4] by pushing
  the latest NSS_3_12_4_FIPS4 CVS tag to mozilla-central for
  some baking on the trunk.

Comment 12

[Wan-Teh Chang]

Comment on attachment 391137 [details]
Update NSS to NSS_3_12_4_FIPS4 in mozilla-central

I pushed NSS_3_12_4_FIPS4 to mozilla-central in changeset aa1d4674a5cc:
http://hg.mozilla.org/mozilla-central/rev/aa1d4674a5cc

Comment 13

[Johnathan Nightingale [:johnath]]

(In reply to comment #5)
> That tag is very similar to the current tag.  It's got all the CRLDP changes
> IINM.  And MC has been running it for 60 days now without apparent grief.

(In reply to comment #10)
> as if we had not already started baking over 60 days ago.  
> Please do not write things that seem to confirm the widely held but mistaken
> belief that CRLDP is not already on Mozilla central and is not already baking.

Yes, absolutely people who labour under the impression that we haven't had CRLDP on trunk for some time are mistaken.

But the question of apparent grief is really the heart of this bug.  If one were to phrase things negatively, my questions in comment 0 are meant entirely to establish the magnitude of grief we can expect - the risk of taking the change, and the understanding we have that helps us contain that risk.

I still don't know whether the mozilla-central top crashers that mention NSS are all assessed, understood, and known not to be a result of moving to 3_12_4, for instance (http://crash-stats.mozilla.com/query/query?do_query=1&product=Firefox&version=Firefox%3A3.6a1pre&date=&range_value=4&range_unit=weeks&query_search=signature ).

I also still don't know what the testing regimen for new NSS versions looks like, or where to go to see results, or how to run those tests myself.

I appreciate that historically we haven't done things this way - we've taken new NSS tags without a particularly formal approval process. But I think you'd agree, Nelson, that that "process" - such as it is - makes it very difficult for anyone involved to predict when things will happen, or what complications may exist. Any help you or other members of the NSS team can offer in understanding the answers to these questions will help Firefox product drivers decide when 3.12.4 should be landed for the FF3.5.x (or, indeed, FF3.0.x) product.

Comment 14

[Nelson Bolyard (seldom reads bugmail)]

Automated CRL fetching via CRLDP is a new feature in NSS 3.12.4, and like
all new NSS features, it is not enabled by default.  The application must
enable it.  So, it may be reasonable for MoCo to ask "what risk do we take
by enabling this new feature?", but the answer to that question is not the 
same as the answer for the question "what risk do we take by upgrading to 
NSS 3.12.4?".  IMO, questions of CRLDP risk ought not to stand in the way
of MoCo taking 3.12.4.

Comment 18

[Reed Loden [:reed]]

(In reply to comment #14)
> Automated CRL fetching via CRLDP is a new feature in NSS 3.12.4, and like
> all new NSS features, it is not enabled by default.  The application must
> enable it.  So, it may be reasonable for MoCo to ask "what risk do we take
> by enabling this new feature?", but the answer to that question is not the 
> same as the answer for the question "what risk do we take by upgrading to 
> NSS 3.12.4?".  IMO, questions of CRLDP risk ought not to stand in the way
> of MoCo taking 3.12.4.

We do have it enabled on current mozilla-central trunk, though, right?

Comment 19

[Johnathan Nightingale [:johnath]]

(In reply to comment #18)

> We do have it enabled on current mozilla-central trunk, though, right?

Correct.

Comment 20

[Kai Engert (:KaiE:)]

On July-28 Johnathan said: (In reply to comment #13)
> I still don't know whether the mozilla-central top crashers that mention NSS
> are all assessed, understood, and known not to be a result of moving to 3_12_4,
> for instance
> (http://crash-stats.mozilla.com/query/query?do_query=1&product=Firefox&version=Firefox%3A3.6a1pre&date=&range_value=4&range_unit=weeks&query_search=signature
> ).

In that list I see only one mentioning of NSS, which is NSSRWLock_LockRead_Util

According to bug https://bugzilla.mozilla.org/show_bug.cgi?id=427715#c103
this has been fixed, and has been ported to mozilla-1.9.0 and mozilla-1.9.1 already.

The bug was not related to NSS 3.12.4

Comment 21

[Kai Engert (:KaiE:)]

(In reply to comment #13)
> I also still don't know what the testing regimen for new NSS versions looks
> like, or where to go to see results, or how to run those tests myself.

Johnathan:

NSS has a test suite. It's being run automatically on NSS tinderbox at http://tinderbox.mozilla.org/showbuilds.cgi?tree=NSS

You may run the NSS test suite manually, by building NSS locally and then run a script. I build with
  export BUILD_OPT=1
and then change to the output tree
  cd mozilla/security/nss/tests/
and run
  HOST=localhost DOMSUF=localdomain ./all.sh

This will take some time to complete.
After your test you will have an output file with results and you can grep for FAILED to see failures, e.g.:
  grep FAILED ./mozilla/tests_results/security/localhost.1/output.log

Note that localhost.1 will increment when running the script multiple times in the same tree.

Also see http://www.mozilla.org/projects/security/pki/nss/testnss_32.html

Is this what you were asking for?

Besides that, no specific "mozilla psm readyness" tests are performed as part of the release process. Not by the NSS team.

Before we drop releases into Mozilla we usually perform a local test build.
When we are aware of bigger changes that aren't yet being built by the Mozilla tree, then we perform a TryServ build.

Comment 22

[Wan-Teh Chang]

Attachment: Update NSS to NSS_3_12_4_RTM in mozilla-central

Comment 23

[Kai Engert (:KaiE:)]

Comment on attachment 395206 [details]
Update NSS to NSS_3_12_4_RTM in mozilla-central

r=kaie

Comment 24

[Wan-Teh Chang]

Attachment: Diffs between NSS_3_12_4_FIPS4 and NSS_3_12_4_RTM (omitting most of cmd, tests, and RCS keywords for brevity)

Comment 25

[Wan-Teh Chang]

Comment on attachment 395206 [details]
Update NSS to NSS_3_12_4_RTM in mozilla-central

I pushed NSS_3_12_4_RTM to mozilla-central in changeset 5d447d9abfdf:
http://hg.mozilla.org/mozilla-central/rev/5d447d9abfdf

Comment 26

[Wan-Teh Chang]

Comment on attachment 395206 [details]
Update NSS to NSS_3_12_4_RTM in mozilla-central

I request permission to push NSS_3_12_4_RTM to mozilla-1.9.2,
which is using NSS_3_12_4_FIPS4 right now.  See attachment
395260 [review] for the diffs between these two CVS tags.

Comment 27

[Wan-Teh Chang]

This bug should block 1.9.2 because mozilla-1.9.2 should not
use a pre-release NSS tag (NSS_3_12_4_FIPS4).

Comment 28

[Johnathan Nightingale [:johnath]]

We've gotten sort of messy here - this bug was originally about what to do with the Gecko1.9.1/Firefox3.5 branch (currently running 3.12.3, and looking to understand the risk involved in taking 3.12.4). Thank you, Kai, for your comments 20 and 21 on that question.

But now we are talking about this bug blocking 1.9.2, because the 1.9.2 branch is already using 3.12.4, but a pre-RTM version.  I agree that 1.9.2 should get updated to the RTM tag, but I think that question is orthogonal to the one being discussed in this bug, no?

I hope you won't think it's unnecessary bureaucracy if I suggest that we file the "Move 1.9.2 branch to NSS 3_12_4_RTM" request as its own bug, which should then be nominated to block 1.9.2 and, I think, be relatively uncontentious?

Comment 29

[Wan-Teh Chang]

Another reason this bug (intended for 1.9.1) needs to block
1.9.2 is that it is Mozilla's standard development process:
any patch for 1.9.1 should also be checked in on the trunk
and 1.9.2 branch.

I filed bug 511413 for moving the 1.9.2 branch to NSS_3_12_4_RTM.

Comment 30

[Wan-Teh Chang]

I just pushed NSS_3_12_4_RTM to mozilla-1.9.2.

Comment 31

[Wan-Teh Chang]

Johnathan, what else do I need to do to fix this bug?
Would you like more baking on the 1.9.2 branch?

My experience with the upgrade from NSS 3.12.3.1 to
3.12.4 in the Chromium browser is that suddenly
Chromium's HTTP client functions (originally written
to support OCSP) are being used by NSS to download
CRLs.  Because CRLs are much larger than OCSP responses,
that exposed a bug in Chromium's HTTP client functions
(http://crbug.com/18559).

So a targeted testing of that code in Firefox 3.5.x
might be helpful.  The HTTP client functions are in
http://mxr.mozilla.org/mozilla1.9.1/source/security/manager/ssl/src/nsNSSCallbacks.cpp

Comment 32

[Johnathan Nightingale [:johnath]]

Wan-Teh - based on the recent information about the crashers being reduced to zero, and the continuing work on getting the NSS tests running in our codebase, I've been having conversations with Dan Veditz and Sam Sidler about how I think we should take this on 3.5.x. The final call is theirs, though.

Nominating as blocking so that they can provide a clear timeline.

Comment 33

[Daniel Veditz [:dveditz]]

Better to get this in early in a cycle (we're just starting 1.9.1.4) than late, so blocking -- but not yet approving the actual landing.

qawanted: please test trunk (not 1.9.2) with the URL from the chrome bug in comment 31 and make sure we're handling the CRL OK.

wtc/kaie/johnath: comment 14 mentions _client_ changes necessary to pick up the new CRL behavior. Have we made that change in 1.9.2 (probably, have on trunk and that was before the branch right?). What change would we need to pick that up in 1.9.1? I guess that should be filed as a separate bug, or if one exists for that change nominated for the 1.9.1 branch.

Unless there's a huge clamor I think we won't take this back on the 1.9.0 branch since that's EOL in a couple of months anyway.

Comment 34

[Wan-Teh Chang]

dveditz: Nelson is the best person to answer that question.
Firefox should automatically start to download CRLs for
EV certificate verification when you drop in NSS 3.12.4.
That's what I observed with Chromium.  But Firefox uses
the old NSS certificate verification API for normal
certificate verification.  I don't know if the old NSS
certificate verification API supports CRL downloads
from CRLDP.  (Chromium uses the new NSS certificate
verification API for both normal and EV certificate
verifications.)

Comment 35

[Nelson Bolyard (seldom reads bugmail)]

In reply to comments 33 and 34,
The use of CRL DP fetching is enabled on trunk and in 1.9.2 and is seen to 
work in present FF 3.6 alpha/nightlies.  I don't know about 1.9.1 though.
But my point in comment 14 was that whether CRLDP is enabled or not should
not be seen as a requirement, nor as a deterrent, for taking NSS 3.12.4 
on 1.9.1.

Comment 36

[Johnathan Nightingale [:johnath]]

Kai - can you confirm that there are no ride-along PSM patches that depend, expect, perform better with, or otherwise would prefer to attach themselves to this new NSS tag, when we land it?

Comment 37

[Kai Engert (:KaiE:)]

(In reply to comment #36)
> Kai - can you confirm that there are no ride-along PSM patches that depend,
> expect, perform better with, or otherwise would prefer to attach themselves to
> this new NSS tag, when we land it?

Shipping the nssdbm3.chk file is the only requirement I'm aware of.
Given that bug 509558 is a dependency of this one, I wonder, should bug 509319 land, too?

(Bug 509319 adds nssdbm3 to additional places for consistency, that may or may not be needed for correct operation/packaging, I haven't tested. But especially that patch adds another shlibsign step for nssdbm3 that may be necessary for FIPS operation. In addition it removes libs from the linker command line, which will be opened at runtime.)

I think there are no PSM-ride-along patches necessary for the new automatic-crl-fetching mechanism provided by the new NSS.

Comment 38

[Johnathan Nightingale [:johnath]]

Kai/Wan-Teh - as people who have recently landed new NSS tags on various branches for us, can one of you handle the landing here on mozilla-1.9.1, pursuant to kai's comment 37, and given dan's approval in comment 33?

Comment 39

[Johnathan Nightingale [:johnath]]

Reed points out that I jumped the gun on dveditz' comment - he is marking it blocking, but hasn't approved the actual landing. Dan - does Kaie's comment 37 do it, in terms of defining overall scope? Or are there other details we need here?

Comment 40

[Daniel Veditz [:dveditz]]

Comment on attachment 395206 [details]
Update NSS to NSS_3_12_4_RTM in mozilla-central

Sorry to have hung things up, branch approvals are normally driven by approval requests. Let's do this.

Approved for 1.9.1.4, a=dveditz

Comment 41

[Wan-Teh Chang]

I pushed NSS_3_12_4_RTM to mozilla-1.9.1 in changeset 42e6ea64b789:
http://hg.mozilla.org/releases/mozilla-1.9.1/rev/42e6ea64b789

Comment 42

[Al Billings [:abillings - ex-MoCo]]

Verified that this is checked in for 1.9.1. That's all for QA to do here.