1927384 - iTrusChina: Issuance of certificates using keys previously reported as compromised

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[iTrusChina Co.,Ltd.]

iTrusChina was notified that it may have issued certificates where the Applicant’s Private Key has been previously disclosed as having been compromised.

Comment 1

[iTrusChina Co.,Ltd.]

Preliminary Incident Report

Summary

iTrusChina was notified by the Google team that it may have issued certificates where the Applicant’s Private Key has been previously disclosed as having been compromised. One instance of the potential mis-issued certificates is
https://crt.sh/?spkisha256=bb9ae5b63e82c9c1dade301dc524cbe9d96c9ca473d46f8673db49ed9a5220e9

iTrusChina has started the investigation to confirm the number of affected certificates and the root causes. We will disclose the full incident report once the investigation is finished and in no later than two weeks.

Comment 2

[Dimitris Zacharopoulos]

Is the title of this incident a copy from another? The reported issue seems inconsistent with the title of the bug.

Comment 3

[iTrusChina Co.,Ltd.]

(In reply to Dimitris Zacharopoulos from comment #2)

Is the title of this incident a copy from another? The reported issue seems inconsistent with the title of the bug.

Hello Dimitris, thanks for your feedback.

At the time when iTrusChina first opened this incident, our preliminary investigation showed that mis-issued certificates were used in our test website for revocation, to be more specific, we used the above title.

We'd like to change the title if it doesn't fit, and please give us some instructions about how to change the title. In case the title can't be changed, we will highlight the discrepancies in our full incident reports if the title is inconsistent with the actual situation. Thanks again.

Comment 4

[Dimitris Zacharopoulos]

Thanks for the quick response. I will defer to Ben, perhaps it is a permissions issue. I think it would be more appropriate for the topic of this bug to highlight the fact that a previously compromised key was allowed to be used in a new certificate.

You will find similar incidents in Bugzilla (open or closed). It is important for CAs to review past incidents and learn from other CA's mistakes.

Comment 5

[iTrusChina Co.,Ltd.]

Incident Report

Summary

On October 26, 2024, iTrusChina was notified by the Google Team that it may have issued certificates where the applicant’s private key has been previously disclosed as having been compromised. This is considered a violation of Section 6.1.1.3 of TLS BRs, which requires that “The CA SHALL reject a certificate request if one or more of the following conditions are met: …4. notified that the Applicant’s Private Key has suffered a Key Compromise using the CA’s procedure for revocation request as described in Section 4.9.3 and Section 4.9.12;”

After investigation, we confirmed eight keys that were first marked as keyCompromise were re-used to issue test website certificates, a total of 41 certificates are involved. The root cause of this incident is a system bug, which set the system’s default revocation reason as keyCompromise, our staff’s lack of understanding of relevant requirements also contributed to this incident. The private keys of these mis-issued certificates are not compromised, so the corresponding public keys were not included in the database of compromised keys, and our staff kept using the same certificate request information to apply for test website certificates, which unfortunately caused us to reuse the “compromised” keys to issue certificates for our test websites.

Please note that the title of this incident has a few discrepancies with the actual situation, except for test websites for revoked certificates, the above-mentioned mis-issued certificates were also used on our test websites for valid certificates, and this incident mainly emphasizes that CAs should not re-use compromised keys to issue certificates.

Impact

All the mis-issued certificates are used for iTrusChina’s test websites, and the private keys of these certificates are not compromised but are first wrongly identified as compromised due to a system bug before September 2022.

Eight certificates’ private keys were first wrongly set as keyCompromise when they were revoked, additional 33 certificates were issued using the corresponding public keys of the eight private keys (please see the attachment for more details). All the mis-issued certificates have been revoked, and as guided by Mozilla Root Policy and community suggestion, these certificates’ revocation reasons have been set to keyCompromise.

Timeline (All times are UTC+8)

Time	Event
2018-12-24	iTrusChina revoked the first certificate, its CRL reason code manual entered was not keyCompromise, but due to the system bug, this certificate’s revocation reason was set as keyCompromise.
2022-08-25	iTrusChina first realized the system bug of the wrong default revocation reason (no matter what revocation reason is entered by the personnel). iTrusChina adopted a temporary revocation processing to avoid wrongly setting the non-keyCompromise keys as keyCompromise before the bug was fixed. But because the keys were not compromised, the relevant staff didn’t put these keys into the database of compromised keys, which caused the later re-usage of these keys to issue new certificates.
2022-11-23	iTrusChina updated the system and changed the system’s default revocation reason from keyCompromise to another proper one and corrected the inconsistency of the human input revocation reason and the certificates’ actual revocation reasons.
2024-10-26	The Google team sent an email to notify iTrusChina that it may issue certificates where the applicant’s private Key has been previously disclosed as having been compromised.
2024-10-28	: iTrusChina started the investigation and filed the preliminary incident report.
2024-10-29	iTrusChina confirmed the keys were not compromised, but due to the system bug, the certificates’ revocation reason was set as keyCompromise.
2024-10-31	iTrusChina designed the improvement plan.
2024-11-1	As guided by the Mozilla root policy and the suggestion from the community, iTrusChina corrected all the relevant certificates’ revocation reasons as keyCompromise and backdated their revocation time to the assumed key compromised time.

Root Cause Analysis

Two root causes incurred the re-usage of compromised keys to issue new certificates. The first one is the system bug discovered in August 2022, which set the default revocation reason as keyCompromise, no matter what revocation reason is entered by the personnel. The second one is our staff’s lack of understanding of relevant requirements, which led us not to include the corresponding public keys of these wrongly identified “compromised” private keys in our database of compromised keys.

Lessons Learned

What went well

We realized the system bug of the wrong default revocation reason and fixed the bug ASAP.

What didn't go well

Some of our staff lack the understanding of the TLS BRs‘ requirements, they didn’t realize that even though the keys were wrongly marked as compromised, they should not use these keys to issue new certificates.

Where we got lucky

All the mis-issued certificates are used for iTrusChina’s test websites.

Action Items

Action Item	Kind	Due Date
Change all the relevant certificates’ revocation reason to keyCompromise and include the corresponding public keys in the database of compromised keys.	Done	2024-11-01
Train our staff about the processes and requirements of TLS BRs, Mozilla Root Policy, and other relevant policies, enhance their understanding of how to handle compromised keys,	Process	2024-11-8
Optimize the process to better incorporate the compromised keys to the database, set up a system scanning task to scan the revocation reasons daily, and once it is found to be a key compromise, it will be automatically incorporated into the key risk database and the responsible team will be notified.	Process	2024-11-19

Appendix

Please see the attachment for more details.

Comment 6

[iTrusChina Co.,Ltd.]

Attachment: Attachment 1 - Certificates List.xlsx

Comment 7

[iTrusChina Co.,Ltd.]

UPDATED Details of Action Items

Action Item	Kind	Completed Date
Train our staff about the processes and requirements of TLS BRs, Mozilla Root Policy, and other relevant policies, enhance their understanding of how to handle compromised keys.	Done	2024-11-08

Comment 8

[iTrusChina Co.,Ltd.]

UPDATED Details of Action Items

Action Item	Kind	Completed Date
Optimize the process to better incorporate the compromised keys to the database, set up a system scanning task to scan the revocation reasons daily, and once it is found to be a key compromise, it will be automatically incorporated into the key risk database and the responsible team will be notified.	Done	2024-11-18

We have completed all the action items.

Comment 9

[Ben Wilson]

Greetings,

Going forward, CA Owners must clearly communicate that they believe that the Incident Report can be closed. This is accomplished by writing a short-summary that includes:

• a description of the incident, its root cause(s), and remediation.
• a summary of all ongoing commitments made in response to the incident.
• an attestation that all Action Items have been completed.

Here is a markdown template for this closure summary:

Incident Report Closure Summary

• Incident Description: [Two or three sentences summarizing the incident.]
• Incident Root Cause(s): [Two or three sentences summarizing the root cause(s).]
• Remediation Description: [Two or three sentences summarizing the incident's remediation.]
• Commitment Summary: [A few sentences summarizing ongoing commitments made in response to this incident.]

Please also state, "All Action Items disclosed in this Incident Report have been completed as described, and we request its closure."

Thanks,

Ben

Comment 10

[Rob Stradling]

Hi Ben.

I realise that CAs are expected to monitor all CA Compliance bugs on Bugzilla, but nonetheless ISTM that a bug comment is an unusual place for Mozilla to (unilaterally?) announce a CCADB Incident Reporting Guidelines update, especially when the CCADB Steering Committee has proposed a set of updates to those Guidelines that are currently under review and that (amongst other things) appear to mention this topic.

Do you intend to post something about this new requirement to MDSP?

Should we expect to see an updated draft of the CCADB Steering Committee's proposed updates that incorporates this new requirement?

Comment 11

[Ben Wilson]

Hi Rob,

I might be premature in treating this as a requirement, and I acknowledge that I haven't been as consistent in applying this approach as I should be, and I plan to share something on MDSP soon. Personally, I believe it is important to establish a practice where CAs provide more than just a simple statement like, "We have completed all the action items." Such statements don't clearly indicate whether they are formally requesting case closure. Moving forward, I want to encourage CAs, even before this section of the CCADB incident-reporting guidance is promulgated, to adopt the habit of submitting a comprehensive closing summary.

Yes - you should expect to see an updated draft of the CCADB Steering Committee's proposed updates that incorporates this new requirement.

Any comments on the proposed incident reporting procedures should be made here: https://github.com/mozilla/www.ccadb.org/pull/186

Thanks,

Ben

Comment 12

[Ben Wilson]

Re: Comment #9, I'll note that this currently just a request and not a requirement yet.

Comment 13

[iTrusChina Co.,Ltd.]

(In reply to Ben Wilson from comment #9)

Greetings,

Going forward, CA Owners must clearly communicate that they believe that the Incident Report can be closed. This is accomplished by writing a short-summary that includes:

• a description of the incident, its root cause(s), and remediation.
• a summary of all ongoing commitments made in response to the incident.
• an attestation that all Action Items have been completed.

Here is a markdown template for this closure summary:

Incident Report Closure Summary

• Incident Description: [Two or three sentences summarizing the incident.]
• Incident Root Cause(s): [Two or three sentences summarizing the root cause(s).]
• Remediation Description: [Two or three sentences summarizing the incident's remediation.]
• Commitment Summary: [A few sentences summarizing ongoing commitments made in response to this incident.]

Please also state, "All Action Items disclosed in this Incident Report have been completed as described, and we request its closure."

Thanks,

Ben

Thanks, Ben.

We are working on the closure summary based on the new template you provided, and we will publish it once it is completed.

Comment 14

[iTrusChina Co.,Ltd.]

Incident Report Closure Summary

• Incident Description: iTrusChina issued 41 certificates where the applicant’s private key has been previously disclosed as having been compromised, This is considered a violation of Section 6.1.1.3 of TLS BR, which requires that “The CA SHALL reject a certificate request if ...the Applicant’s Private Key has suffered a Key Compromise...”
• Incident Root Cause(s): The root causes are a system bug (first realized on August 2022) that set the default revocation reason as keyCompromise (the keys were not compromised) and our staff’s lack of understanding of relevant BR requirements.
• Remediation Description: iTrusChina has revoked all the mis-issued certificates and set their revocation reasons as keyCompromise. We also trained our staff about the requirements and handling of compromised keys and updated our systems to better and timely (once the keys were found compromised) incorporate the compromised keys into the database.
• Commitment Summary: iTrusChina will cautiously handle the compromised keys and update our systems to timely include the compromised keys in the databases to avoid their re-usage. We will also improve our processes and implementation of key management to better comply with BRs and other policies.

All Action Items disclosed in this Incident Report have been completed as described, and we request its closure.

Comment 15

[iTrusChina Co.,Ltd.]

We have completed all action items and there have been no new questions. Please check if this incident can be closed, thanks.