Open Bug 1895006 Opened 19 days ago Updated 15 hours ago

IdenTrust: unintended creation of a Root CA certificate

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
ASSIGNED

People

(Reporter: roots, Assigned: roots)

Details

(Whiteboard: [ca-compliance] [ca-misissuance])

IdenTrust unintended creation of a Root CA certificate

Summary

On April 30, 2024, during a key ceremony for a new Subordinate Certification Authority (CA), an unintended event occurred. Instead of creating the intended Subordinate CA, the execution of a wrong command resulted in the generation of a new Self-Signed Root CA.
The new Root CA Certificate does not conform to the Server Certificate Baseline Requirements expected for Root CA Certificate profile definition.

Impact

The newly self-signed Root CA certificate has been disclosed into the Common CA Database (CCADB) and we are requesting placement on the OneCRL.

Remediation and Reporting

A full incident report detailing the root cause analysis, corrective actions, and preventive measures will be supplied by May 17, 2027

Impact

We foresee no impact at this time. The newly self-signed Root CA certificate has been disclosed into the Common CA Database (CCADB) and we are requesting placement on the OneCRL.

Summary: IdenTrust: Temporary Errors in Test Web Pages → IdenTrust: IdenTrust unintended creation of a Root CA certificate
Summary: IdenTrust: IdenTrust unintended creation of a Root CA certificate → IdenTrust: unintended creation of a Root CA certificate

Does the public key in this certificate appear in any certificates that are directly or transitively trusted by root programs?

Flags: needinfo?(roots)

(In reply to Andrew Ayer from comment #2)

Does the public key in this certificate appear in any certificates that are directly or transitively trusted by root programs?

Yes, the public key in this certificate is directly trusted by root programs as it shares the same key pair as the "IdenTrust Commercial Root CA 1" (https://crt.sh/?CAID=1587)

The newly self-signed Root CA certificate has been disclosed into the Common CA Database (CCADB)

Please could you provide the CCADB record ID? (I can't find any CCADB certificate record that matches the description you've provided).

(In reply to Rob Stradling from comment #4)

The newly self-signed Root CA certificate has been disclosed into the Common CA Database (CCADB)

Please could you provide the CCADB record ID? (I can't find any CCADB certificate record that matches the description you've provided).

Here is the CCADB Unique Record ID: A011607

Thanks. CCADB does now list that certificate when I search for the SHA-256(SPKI) value, and the certificate also just showed up at https://crt.sh/mozilla-disclosures#unknown.

Since it's unlikely that any CT logs will accept this certificate due to the nature of the misissuance, I've just added it to crt.sh manually:

https://crt.sh/?id=12943609985

I hope that in the full incident report, Identrust explains how their ceremony generation tool made this possible. Considering how long Identrust has been a WebPKI CA, this is extremely concerning and disappointing. Please confirm that something similar to this has not happened in the past.

Screenshots of the ceremony tooling, and the safeguards and options it displays will be extremely helpful in understanding how this happened, and how Identrust & other CAs can prevent this in the future.

There are existing tools that safeguard against this exact problem (https://github.com/letsencrypt/boulder/tree/main/cmd/ceremony), so I'd be curious if you were 1) aware of these tools, 2) have a justification for not using them.

(In reply to amir from comment #7)
We understand your concerns. We will provide a detailed explanation of how this happened, including the specific mechanisms involved in the key generation ceremony within the full incident report.
We confirm that something similar to this has not happened in the past. IdenTrust is committed to maintaining the highest standards of security and processes and we are actively reviewing our procedures to ensure such incidents are not repeated. Thank you for your patience and understanding.

Assignee: nobody → roots
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] [ca-misissuance]

Can someone explain why creating a new self-signed certificate using an existing key is forbidden? If this certificate -technically an Intermediate CA Certificate because it chains to an included Root CA Certificate- is properly disclosed to CCADB, included in the audit report, why is this an issue, besides it being an internal Identrust incident because it intended to create an intermediate CA and accidentally created a self-signed one?

Mozilla root store policy:

When a CA operator fails to comply with any requirement of this policy - whether it be a misissuance, a procedural or operational issue, or any other variety of non-compliance - the event is classified as an incident and MUST be reported to Mozilla as soon as the CA operator is made aware

Thanks but is there any specific requirement of the MRSP that the CA has failed to comply with? This part of the policy explains that a CA must file an incident (publicly) if something is in violation of the MRSP.

Dimitris, if this indeed technically is a Subordinate CA, then it would be a misissued intermediate CA, as it does not match the requirements placed on the issuer extension in TLS BR section 7.1.2.6: "MUST be byte-for-byte identical to the subject field of the Issuing CA."

In the other case, where it's considered a Root CA, its issuance would be a violation of

  • TLS BR Section 6.1.7 (Key usage purposes), as "Self-signed Certificates to represent the Root CA itself;" are allowed, but "the Root CA", singular, should be considered to be the already previously existing Root CA.
  • TLS BR Section 7.1.2.1.2 Root CA Extensions, as it contains the EKU extension

Adding also the certificate contains the User Notice policy qualifier, which is not allowed per the BRs. It seems this was included as well in the eventually issued Subordinate CA with the same Subject DN found at https://crt.sh/?id=12942971617&opt=zlint

On a side note, I believe we should work on adding clearer language to the TLS BRs to show that signing two Root CAs with the same keys is not an allowable practice.

All in all I believe this is a very interesting case. Is it a Root, or a Subordinate and I am looking forward to Identrust's full report and their take on this.

(In reply to Martijn Katerbarg from comment #12)

Dimitris, if this indeed technically is a Subordinate CA, then it would be a misissued intermediate CA, as it does not match the requirements placed on the issuer extension in TLS BR section 7.1.2.6: "MUST be byte-for-byte identical to the subject field of the Issuing CA."

I think my description was not accurate. A self-signed/self-issued (per RFC 5280) certificate technically chains to the Trust Anchor CA Certificate included in the Mozilla Root Store. That doesn't mean that it is a Subordinate CA Certificate thus it is not a violation of BR section 7.1.2.6. I remember a case of a "doppelganger" CA in https://bugzilla.mozilla.org/show_bug.cgi?id=1610000. This seems to be a similar case.

In the other case, where it's considered a Root CA, its issuance would be a violation of

  • TLS BR Section 6.1.7 (Key usage purposes), as "Self-signed Certificates to represent the Root CA itself;" are allowed, but "the Root CA", singular, should be considered to be the already previously existing Root CA.

I do not fully agree with this interpretation. BR section 6.1.7 allows the creation of self-signed Certificates that can be used as Root CAs. A CA can issue as many of those as it likes to distribute as Trust Anchors in various Root Stores. However, because of the chaining effect of a Mozilla-trusted Root CA Certificate, it must be audited and properly disclosed per MRSP.

  • TLS BR Section 7.1.2.1.2 Root CA Extensions, as it contains the EKU extension

Adding also the certificate contains the User Notice policy qualifier, which is not allowed per the BRs. It seems this was included as well in the eventually issued Subordinate CA with the same Subject DN found at https://crt.sh/?id=12942971617&opt=zlint

Even that is debatable because this new Root CA Certificate could be considered out-of-scope of the BRs. Perhaps it is meant for another hierarchy that is not related to TLS Certificates. To the best of my knowledge, the BRs do not require that CA keys must not be re-used. If that was the case, cross-certificates that share the same key as the Root CA would not be allowed.

On a side note, I believe we should work on adding clearer language to the TLS BRs to show that signing two Root CAs with the same keys is not an allowable practice.

This is allowed by RFC 5280 and I'm sure some CAs had to sign multiple Root CA Certificates to fix extensions or other information before submitting a Root CA Certificate to Mozilla for Root Store inclusion.

Also, in practice, I can't think of a security risk to Relying Parties as these Certificates are ignored when included in a CA Certificate chain.

All in all I believe this is a very interesting case. Is it a Root, or a Subordinate and I am looking forward to Identrust's full report and their take on this.

If my reading is correct, at this time it is a self-signed or self-issued certificate as described in RFC 5280. I'm also interested to hear other opinions and of course the follow-up from Identrust, even if this bug is ultimately deemed as INVALID.

Even that is debatable because this new Root CA Certificate could be considered out-of-scope of the BRs. Perhaps it is meant for another hierarchy that is not related to TLS Certificates. To the best of my knowledge, the BRs do not require that CA keys must not be re-used. If that was the case, cross-certificates that share the same key as the Root CA would not be allowed.

Section 7.1.2 of TLS BR says:

"7.1.2 Certificate Content and Extensions
If the CA asserts compliance with these Baseline Requirements, all certificates that it issues MUST
comply with one of the following certificate profiles, which incorporate, and are derived from RFC
5280."

I interpret this to mean that all certificates signed by a given CA Private Key must conform to one of the listed profiles in section 7. The intent of SC-62 was to restrict the profile of certificates signed by CA Private Key material to a defined set of profiles.

If there is disagreement on the interpretation of the quoted sentence above then it should be amended to make the intent clear. In this particular case, the certificate in question is not usable and does not represent a concrete security risk. Nonetheless, it is not a positive for the ecosystem to allow CA Private Key material to sign certificates that cannot be revoked.

Thanks Corey. My understanding of "if the CA asserts compliance" is the existence of a CA/B Forum Policy OID after the path building algorithm produces a valid policy tree as described in section 6.1.2 of RFC 5280. This particular Root Certificate does not participate in any path building.

If you recall SC62, there was a lot of discussion around the profiles of mainly Intermediate CA Certificates, which includes Cross-Certificates, Technically Constrained non-TLS, Technically Constrained TLS, Unconstrained TLS CA Certificates, OCSP responder, and also Subscriber TLS Certificates. I'm not even sure what language would need to be added to the TLS BRs to prevent this, or why.

The use of a Key associated with a Root CA Certificate is already described in section 6.1.7 and as mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=1895006#c13 it allows the use for the creation of a self-signed certificate, which is necessary for cases where the CA needs to update or extend the validity of its self-signed Root Certificate. Do you agree that this needs to be allowed?

(In reply to Martijn Katerbarg from comment #12)
We acknowledge and confirm that the intended subordinate CA certificate is also mis issued due to the presence of the User Notice in the certificate profile extension; this Subordinate CA has been revoked and we will be disclosing a separate incident report for issue by May 17, 2024.

Flags: needinfo?(roots)

Incident Report

Summary

On April 29, 2024, during a key ceremony for an intermediate CA certificate (ICA), an unintended event occurred. Instead of creating the intended ICA, the execution of a wrong command resulted in the generation of a new malformed self-signed Root CA certificate.
The malformed self-signed Root CA Certificate does not conform to the Server Certificate Baseline Requirements of the CA/Browser Forum (BRs) expected for Root CA certificate profile definition.

Impact

A malformed self-signed Root CA certificate was inadvertently created but was not placed into service.
This malformed, self-signed Root CA certificate did not comply with the technical requirements outlined in BRs.

Remediation and Reporting

To rectify the situation the following actions were taken:
• Revoke the malformed, non-compliant self-signed Root CA certificate rendering it invalid and unusable.
• Disclosed the issue ensuring transparency and accountability.

Timeline

2024-04-16:
The PKI operator created the ICA in a pre-production environment; no issues were flagged when the ICA was checked with the Linter tool and the certificate was approved for production issuance.

2024-04-29:

• 13:45 MDT: PKI operations started key-ceremony process to create the ICA in the production environment.
• 15:00 MDT: The PKI operator inadvertently created a self-signed malformed Root CA certificate when they had intended to generate an ICA. This error occurred due to the operator using an incorrect command parameter in the in-house developed application for issuance of ICA and Root CA certificates. Although there was a witness present for the key ceremony process, the mistake went unnoticed initially. However, the PKI operator promptly realized their error after the malformed root CA certificate had been created.
• 15:05 MT: The PKI operator started a second key-ceremony, this time invoking the correct command parameter and confirmed proper creation of the expected ICA.
• 16:06 MDT: PKI Operations made internal compliance team aware of the malformed self-signed Root CA certificate.
• Compliance team began to review the issue.

2024-05-03:

The compliance team agreed with other stakeholders to disclose a preliminary incident report while investigating further details.

• 13:00 MDT: Both certificates were uploaded into CCADB.
• 14:19 MDT: We disclosed a preliminary incident and addressed initial community feedback.

2024-05-06:

• 14:30 MDT: PKI Operations successfully tested revocation of the malformed self-signed Root CA certificate in the pre-production environment.

2024-05-07:

• 10:32 MDT: PKI Operations completed the revocation/CRL process key-ceremony of the malformed self-signed Root CA certificate in the production environment.

• 12:30 MDT: The revocation status for the malformed self-signed Root CA certificate was updated in CCADB.

Root Cause Analysis

The unintended creation of the self-signed Root CA certificate was the result of two human errors:
• A PKI operator entered the wrong command parameter using an in-house developed application for issuance of ICA and Root CA certificates.
• The ceremony witness missed noticing the invocation of the wrong command until just after the certificate was created.

Lessons Learned

• Regular reminder training for ceremony witnesses will help to catch such errors more promptly.
• Awareness that the use of in-house developed application can be further enhanced using a parameter option to mitigate the risk of recurrence.

What went well

• The immediate awareness of the malformed self-signed Root CA prevented any subsequent activities to generate certificates from the malformed Root CA certificate.

What didn't go well

• Despite having a second set of eyes as the witness role, the witness also did not catch that the incorrect command was used during the key ceremony.

Where we got lucky

• We caught this right away and we were able to take actions to correct it with no impact externally.

Action Items

| Action Item | Kind | Due Date |
|Removed the parameter option; future ICAs and Root CA have their own execution command without additional manual parameters| Prevent | 2024-05-09 |
|Complete evaluation of relevant external PKI tools and establish an implementation plan if our evaluation determines such tools implement stronger preventative technical controls than current tools |Prevent |2024-07-31|

Appendix

Details of affected certificates

https://crt.sh/?id=12943609985

(In reply to IdenTrust from comment #17)

Root Cause Analysis

The unintended creation of the self-signed Root CA certificate was the result of two human errors:
• A PKI operator entered the wrong command parameter using an in-house developed application for issuance of ICA and Root CA certificates.
• The ceremony witness missed noticing the invocation of the wrong command until just after the certificate was created.

Lessons Learned

• Regular reminder training for ceremony witnesses will help to catch such errors more promptly.
• Awareness that the use of in-house developed application can be further enhanced using a parameter option to mitigate the risk of recurrence.

Training and awareness don't provide assurance that similar issues will be prevented in the future. I'm sure these employees were already trained and aware.

| Action Item | Kind | Due Date |
|Removed the parameter option; future ICAs and Root CA have their own execution command without additional manual parameters| Prevent | 2024-05-09 |
|Complete evaluation of relevant external PKI tools and establish an implementation plan if our evaluation determines such tools implement stronger preventative technical controls than current tools |Prevent |2024-07-31|

Can you describe these action items in more detail? This should have been the main content of your lessons learned. What tooling, safeguards, and options are currently used during key ceremonies? What specifically will be implemented? For example, from you timeline, it looks like a correct ICA was created in preproduction. I would expect a technical measure to ensure that the the same command that were validated in preproduction are automatically ran for the production ceremony, which would eliminate human error when entering commands.

(In reply to Mathew Hodson from comment #18)

Training and awareness don't provide assurance that similar issues will be prevented in the future. I'm sure these employees were already trained and aware.

IdenTrust: We understand that the training and awareness aren’t foolproof but they are a crucial first line of defense in preventing similar issues. This incident allows us to examine the effectiveness of the current training and identify areas for improvements. Apart from the training, we are also considering other technical controls to tighten the process so we don’t run into similar issues.

| Action Item | Kind | Due Date |
|Removed the parameter option; future ICAs and Root CA have their own execution command without additional manual parameters| Prevent | 2024-05-09 |
|Complete evaluation of relevant external PKI tools and establish an implementation plan if our evaluation determines such tools implement stronger preventative technical controls than current tools |Prevent |2024-07-31|

Can you describe these action items in more detail? This should have been the main content of your lessons learned. What tooling, safeguards, and options are currently used during key ceremonies? What specifically will be implemented? For example, from you timeline, it looks like a correct ICA was created in preproduction. I would expect a technical measure to ensure that the the same command that were validated in preproduction are automatically ran for the production ceremony, which would eliminate human error when entering commands.

(In reply to Mathew Hodson from comment #18)
Can you describe these action items in more detail? This should have been the main content of your lessons learned. What tooling, safeguards, and options are currently used during key ceremonies? What specifically will be implemented? For example, from you timeline, it looks like a correct ICA was created in preproduction. I would expect a technical measure to ensure that the the same command that were validated in preproduction are automatically ran for the production ceremony, which would eliminate human error when entering commands.

IdenTrust: We have already changed the ceremony processes to use separate utilities for creating Root CAs and ICAs. The selection of the proper utility before the ceremony eliminates the need to choose Root vs. ICA creation in mid-ceremony. We believe that this will ensure that this type of error does not occur again.

(In reply to IdenTrust from comment #17)

15:05 MT: The PKI operator started a second key-ceremony, this time invoking the correct command parameter and confirmed proper creation of the expected ICA.

Did the operator decide to start this second key-ceremony on their own, or was any internal discussion held prior to this?

The reason I'm asking is, if something's gone wrong with a signing ceremony, one utilising a key that is used by a publicly trusted Root CA certificate none-the-less, my first thought wouldn't be to skip ahead, adjust and right away create a Subordinate CA with the same SubjectDN as the accidentally issued Root CA.

(In reply to Martijn Katerbarg from comment #21)
You raise an excellent point regarding the importance of meticulously following procedures when generating a root CA and ICA certificates, which must be done with utmost care and precision to ensure public trust. The question demonstrates your grasp of the critical nature of this process.

During the key ceremony, there was an unintentional deviation from the planned script. However, the PKI Administrator and witness promptly identified the mistake and paused to thoroughly discuss and understand what had occurred. Since both parties clearly recognized the error and the necessary corrective action, they made the prudent decision to proceed according to the approved script to achieve the intended outcome.

You need to log in before you can comment on or make changes to this bug.