1669594 - IdenTrust: Issuance of Subordinate CA’s Without EKU

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[IdenTrust]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36

Steps to reproduce:

We have discovered that 2 Subordinate CA’s issued on 9/30/2020 from our DST Root CA X3 are missing required EKU’s extension as required by BRs 1.7.2, section 7.1.2.2g. We are planning and executing the necessary remediation. A formal incident report will follow.

Comment 1

[Andrew Ayer]

Today I stumbled on the following intermediate certificate issued by IdenTrust which lacks an EKU extension:

https://crt.sh/?sha256=FEE765DA4CACF53C71AF202F89F3612420FD930D804E204FEEEFC9D78084BB7B

Is this one of the two certificates referenced in this bug? Why has IdenTrust not yet provided an incident report for this?

Comment 2

[IdenTrust]

(In reply to Andrew Ayer from comment #1)
Yes, that is one of the intermediate CA's without EKU and the other one is:
https://crt.sh/?id=3470671160
We will be supplying an incident report no later than tomorrow October 16, 2016.

Comment 3

[IdenTrust]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
   IdenTrust:
   On October 6, 2020 as part of the final review, before production deployment, we discovered that two subordinate CAs issued from our DST Root CA X3 were missing EKU extensions as required by B.R. 1.7.2, section 7.1.2.2g.
2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
   IdenTrust:
   September 30, 2020 13:52MT: Issued the 2 Subordinate CA’s
   October 6, 2020 9:00MT: Discovered the that 2 Subordinated were missing the EKU extension
   October 6, 2020 13:01MT: obtained approval to revoke/re-issue
   October 6, 2020 10:01MT: Logged initial Bugzilla bug.
   October 7, 2020 13:21MT: New subordinates were created including EKU extension
   October 7, 2020 15:27MT: Revoked the two mis-issued subordinate CA’s
   October 7, 2020 16:00MT: Updated Mozilla’s CCADB records accordingly
   October 15, 2020, 13:01MT: Confirmed inquiry to comment 1 in the bug.
   October 16, 2020 16:300MT: Added this incident report
3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
   IdenTrust:
   Yes, we have stopped. Zero production end-entity certificates were issued from either of those two subordinate CA’s.
4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
   IdenTrust:
   Two subordinate CA certificates issued on September 30, 2020
5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
   IdenTrust:
   https://crt.sh/?id=3470671161; https://crt.sh/?id=3470671160
6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
   IdenTrust:
   On October 19, 2015 we issued two subordinate CAs to ISRG (providers of Let Encrypt service) as “Let’s Encrypt Authority X1” and “Let’s Encrypt Authority X2” from the DST Root CA X3 which will expire on October 19, 2020. As part of our ongoing partnership with ISRG, we agreed to provide two new subordinate CA’s as replacements with a similar profile to the original subordinate CA’s issued in 2015. For issuance of the new ISRG subordinate CAs, we used the same subordinate certificate profiles as previously issued subordinate CAs, with the exception of changes in common name and removal of OCSP from the AIA extension. These subordinate CA profiles configured for DST Root CA X3 did not have the EKU extension, as this was not required by B.R. back in 2015.

We were aware of the B.R. 1.7.2, section 7.1.2.2g update requiring EKU for subordinate CA’s that are intended to issue TLS Certificates and had implemented the requirement on subordinate CA certificates from our other Root CA (IdenTrust Commercial Root CA 1); unfortunately, we overlooked the update on the certificate profiles for the DST Root CA X3 which is due to expire on September 30, 2021, as we were not expecting to issue any more subordinate CA certificates from that root. In addition, during our compliance review prior to issuance, there was error in incorrectly classifying the yet to be created certificates as Cross Certificates. These two mis-steps and errors caused mis-issuance of the certificates listed in this report.

As noted above, the detection of the mis-issuance occurred as part of the final review prior to any TLS or other end-entity certificate issuance.
7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
IdenTrust:
As a result of this incident, we have updated the certificate profiles in the DST Root CA X3 to include the EKU extension and have revoked the 2 subordinate CA’s identified in this report. Also, we have modified procedural controls for the DST Root CA X3 to conduct a thorough review of subordinate CA certificate profiles prior to issuance, including quality review by our Risk Management Committee.

We have confirmed that the subordinate CA certificate profiles for our other root, IdenTrust Commercial Root CA 1, the EKU extension is asserted by default.

Comment 4

[Ryan Sleevi]

Also, we have modified procedural controls for the DST Root CA X3 to conduct a thorough review of subordinate CA certificate profiles prior to issuance, including quality review by our Risk Management Committee.

Can you please be more specific here?

This is called out in Responding to an Incident, and seems very much on the side of "training has been improved".

Please detail all of the procedural controls that existed prior to this incident with respect to issuing a subordinate CA. Please then detail all of the current procedural controls, who performs them, how they're performed and verified, how they're maintained as the relevant standards evolve, and how frequently the Risk Management Committee reviews them.

For example, see Bug 1654967, and how it's evolved and the level of detail in the responses, as a model to approach this response with.

Comment 5

[IdenTrust]

The controls we have in place for validating compliance with CA/B Forum requirements are:

1. Reviewing the subordinate CA certificate profile with the latest version of these documents:
   a. IdenTrust published CP/CPS
   b. B.R. SSL guidelines
   c. B.R. Extended Validation Certificate guidelines
   d. Root Store CA program requirements.
2. Reviewing if the subordinate CA is capable of issuing:
   a. SSL/TLS certificates
   b. S/MIME certificates
3. Reviewing if the subordinate CA requires public disclosure or not:
   a. Yes, if either 2a or 2b or both above are asserted AND the subordinate CA is NOT fully technically constrained
   b. Not, otherwise
4. When yes, the PEM of the subordinate CA is supplied to the Program Manager within 7 days of issuance and prior to start issuing end-entity certificates, for further disclosure into Mozilla’s CCADB.
5. The Program Manager will ensure that the new subordinate is added to policy documents and annual audit documents.

Prior to this incident, the referenced controls were enforced on new subordinate CA’s issued from of our public roots but not as strict on subordinate CA’s certificates being renewed or replaced.

The updates to the procedural controls now call out that regardless, if the subordinate CA is new, renewed or replaced, the certificate profile must be revisited and adjusted accordingly based on the latest CA/B Forum compliance guidelines before the key ceremony takes place.

The Risk Management Committee reviews certificate profiles as part of policy document updates as often as policy documents are submitted for policy management review/approval.

Comment 6

[Ben Wilson]

Thanks. I don't have any other questions at this time.

Comment 7

[Ben Wilson]

"The Program Manager will ensure that the new subordinate is added to policy documents and annual audit documents" should read something like this, "The Program Manager will ensure that the CCADB record for a new internally operated subordinate CA certificate is marked 'same as parent' and that the certificate record for any externally operated CA is populated with the appropriate audit information and policy documents for that external CA." Also, as a side note, any new external entity operating the corresponding private CA key must undergo a notice-and-approval process if it has not previously been approved by Mozilla. See https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/#8-ca-operational-changes

Comment 8

[IdenTrust]

Yes, our process not only covers your suggested language but also it requires a 'second' pair of eyes reviewing CCADB updates to ensure accuracy. The CCADB update validation is handled via a screen-sharing session.

Comment 9

[Ben Wilson]

I think that this matter can be closed and will do so next Wed. 27-Jan-2021 unless there are other comments or areas to explore.