Closed Bug 1290999 Opened 3 years ago Closed 3 years ago

August 2016 batch of root CA changes


(NSS :: CA Certificates Code, task)

Not set


(Not tracked)



(Reporter: KaiE, Assigned: KaiE)




(2 files)

August 2016 batch of root CA changes
Kai pointed out to me that in this August batch of root changes, all but one of them are removals. In regards to stability, root removals are more risky than root additions. I would like to have the root removals happen in Firefox 51, so they don't get removed in a release in the November/December time frame.  However, there is no reason to delay the root addition to Firefox 51. So, we will separate out the one root addition, and do that change first. Then we will work on the rest of the August batch of root changes (the root removals).
Assignee: nobody → kaie
(In reply to Kai Engert (:kaie) from comment #2)
> Created attachment 8776898 [details] [diff] [review]
> Part A - additions - v1

A Firefox test build with this patch (that adds the ISRG root) has been started.
( )

Once the build is completed (in a few hours), the test build can be accessed here:
Attachment #8776902 - Attachment description: Part B - removals - v1 → Part B - removals - v1 (incremental patch)
> Part A - additions - v1
> Part B - removals - v1 (incremental patch)

An additional Firefox test build with BOTH patches (additions AND removals) has been started.
( )

Once the build is completed (in a few hours), the test build can be accessed here:
Please test in two steps:

(1) use the build from comment 3 to confirm the ISRG root has been correctly added,
    confirming the correctness of Patch A (attachment 8776898 [details] [diff] [review]).

(2) use the build from comment 5 to confirm the removals are correct,
    confirming the correctness of Patch B (attachment 8776902 [details] [diff] [review]).
I just want to ensure that the Buypass Class 2 CA 1 root certificate (Bugzilla ID = 1266574) NOT is included in this set of root certificates scheduled for removal. 

There are still valid certificates chaining to this root and it should not be removed until it expires at 13th October 2016.
Mads, only the addition will be done earlier.

All removals will target firefox 51 final release, currently scheduled for Jan 24 according to

(The removals will happen earlier only in the beta and developer edition builds.)
The Let's Encrypt team has tested the build with our root added on multiple platforms (Linux, OS X, Windows) and everything seems to work well. Thanks!
Comment on attachment 8776898 [details] [diff] [review]
Part A - additions - v1

Richard, could you please review this patch, which adds the ISRG root to NSS?
Attachment #8776898 - Flags: review?(rlb)
(In reply to Kai Engert (:kaie) from comment #6)
> Please test in two steps:
> (1) use the build from comment 3 to confirm the ISRG root has been correctly
> added,
>     confirming the correctness of Patch A (attachment 8776898 [details] [diff] [review]

Tested. ISRG root added with correct trust bit.
CA has also tested.

> (2) use the build from comment 5 to confirm the removals are correct,
>     confirming the correctness of Patch B (attachment 8776902 [details] [diff] [review]

Tested. Confirmed correct removals.

Comment on attachment 8776898 [details] [diff] [review]
Part A - additions - v1

Review of attachment 8776898 [details] [diff] [review]:

Downloaded ISRG root from:

* SHA-1 fingerprint
* SHA-256 fingerprint
Attachment #8776898 - Flags: review?(rlb) → review+
Comment on attachment 8776898 [details] [diff] [review]
Part A - additions - v1

checked in to trunk for NSS 3.27

and also to NSS_3_26_BRANCH for NSS 3.26
Attachment #8776898 - Flags: checked-in+
I've marked bug 1289889 as fixed for NSS 3.26

This bug and the remaining root changes target NSS 3.27
Target Milestone: --- → 3.27
See Also: → 1294150
If we must delay these changes until January 2017, then I think, we shouldn't check them in now, because they shouldn't become part of NSS 3.27 which will be released around Sep 12, and consumers of the NSS library would pick these changes up too early.

I think these CA removals should be checked in to NSS after Nov 7, to target NSS 3.29 (Firefox 53) as the first NSS release.

Around the same time, we should uplift these CA removals into:
- an NSS 3.28.x release that targets Firefox 52 and will be added into
  the Firefox-Aurora branch after Nov 7
- an NSS 3.27.x release that targets Firefox 51 and will be added into
  the Firefox-Beta branch after Nov 7

This would ensure that nobody would pick up the CA changes into production before January 2017.
We cannot release NSS 3.29 earlier than 2017-01-23.

We need a NSS 3.27.1 release for FF51, that will be officially released, because Linux distributors must be able to pick up the correct version of NSS, that comes with the correct set of root CA certificates.

However, we don't want anyone to consume the CA removals before January 2017. This means, we must not publish the NSS 3.27.1 release earlier than 2017-01-01, to ensure that no consumers of NSS 3.27.1 will experience breakage.

On the other hand, around 2017-01-01 the FF51-beta is really supposed to be very stable already, and late changes are discouraged. Firefox release engineers will need to prepare their final builds as early as possible.

We shouldn't create the NSS 3.27.1 release tag at an earlier time. Although it would allow landing into the FF51 branch at an earlier time, we still couldn't upload the NSS 3.27.1 source archive, and couldn't announce it until 2017-01-01.  While it would simplify it for Mozilla, other consumers of Firefox which build NSS separately (as a system package) would run into a failure at build time, because the build configuration of the Firefox code (where we enforce the minimum expected NSS version) would refuse to work with an older NSS release, and require the non-released NSS version. This would block those consumers from doing early packaging preparations of FF51.

We also shouldn't create it earlier and tell people "don't use it yet". This would complicate the NSS release process, if an emergency NSS security fix needs to be published in the waiting period.

I'm afraid this means, we shouldn't publish any NSS release with this CA update earlier than 2017-01-01.
According to the Firefox rapid release calendar, and because we usually freeze NSS soon after Firefox moves into the aurora phase, I anticipate the following NSS releases:

NSS 3.27 for FF51 by 2016-09-12 or soon afterwards.

NSS 3.28 for FF52 by 2016-11-07 or soon afterwards.

NSS 3.29 for FF53 by 2017-01-23 or soon afterwards.

Because of comment 16, we need to create these additional releases:

NSS 3.27.+1 for FF51 on 2017-01-02

NSS 3.28.+1 for FF52 on 2017-01-02

We should coordinate with Firefox release engineering ahead of time, to ensure that we don't need to hunt for approvals in early January. We should get a statement (in december) from people with approval power, that they approve the January 2nd landing ahead of time.

The idea is to ensure that the NSS releases created on 2017-01-02 can ideally be landed into the FF51-beta and FF52-aurora branches on the same day.

(If for any reasons this tight timing isn't acceptable to FF51 drivers, the alternative is to delay the relevant CA removals to Firefox 52.)
Kathleen, could you please check with Firefox release drivers, if the suggested timing from comment 17 is acceptable to both you and them, and confirmation that we will get ahead approval for 2017-01-02 - or if there's preference for the alternative FF52 target?
Kathleen, also, I'd like to make an additional offer. You could decide which of these CAs removals can be done immediately. We could remove those immediately, they'd target NSS 3.27, and they'd be removed from FF51 for sure.

For the remaining CA removals (that need to be delayed), we could still use the plan described in the previous comments, either targetting FF51 or FF52.

While we're at it, we should also talk about the near term future.

You probably plan another batch of CA changes in about 2-3 months?

If yes, my preference would be to do them after the removals that need to be delayed, even if it means we'll have a larger number of individual CA releases.


We do a CA update within the next 4 weeks, with a subset of removals from this bug, the ones that don't need to be delayed, and release it with NSS 3.27, for FF 51. This would be CA set version 2.10.

If you have any additional CA changes almost ready, that shouldn't wait very long, you could get them ready by 2016-09-12, and we could include them in this batch for NSS 3.27 / FF 51, too.

No other CA changes until January.

The CA update with the delayed removals, only, CA set version 2.11, will target 2017-01-02.
This is the one that the previous comments refer to.
It either targets FF51 or FF52, based on the feedback you get from Firefox drivers (landing 2017-01-02 into FF51-beta ok or not).

In late january, we'd work on the next batch, which can target NSS 3.29 and FF 53.
No longer blocks: 1250699
No longer blocks: 1251025
No longer blocks: 1283326
No longer blocks: 1286696
No longer blocks: 1288250
No longer blocks: 1266574
No longer blocks: 1272156
No longer blocks: 1272158
I've moved all individual bugs to the dependency list of new suggested batch updates for September and 2017-01-02.

Closing this as incomplete, as the ISRG update has been handled individually, and as all remaining changes will be delayed, at least to september.
Closed: 3 years ago
Resolution: --- → INCOMPLETE
See Also: → 1296697, 1296689
You need to log in before you can comment on or make changes to this bug.