1798812 - CFCA: Delayed reporting of revocation of an intermediate CA certificate

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Ben Wilson]

CFCA did not report the revocation of CA certificate CFCA DV OCA within 7 days as required by section 4 of the CCADB Policy, which says, "If an intermediate certificate is revoked, the CCADB must be updated to mark it as revoked, giving the reason why, within 24 hours for a security incident, and within 7 days for any other reason." https://www.ccadb.org/policy#4-intermediate-certificates

Comment 1

[Gao Fei]

1)How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

2022-11-02, in the Bugzilla incident (https://bugzilla.mozilla.org/show_bug.cgi?id=1784820), our intermediate certificate (https://crt.sh/?id=6970868811) was reported not in compliance with CCADB policy,which is failure to mark the certificate as revoked and give the reason for the revocation within the allotted time.

2)A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

2022-06-19: Upload the intermediate certificate CFCA DV OCA (https://crt.sh/?id=6970868811) to CCADB.
2022-09-19: Revoke the intermediate certificate CFCA DV OCA (https://crt.sh/?id=6970868811).
2022-10-17: re-issued intermediate certificate CFCA DV OCA (https://crt.sh/?id=7833196483).
2022-10-20: Upload the intermediate certificate CFCA DV OCA (https://crt.sh/?id=7833196483) to CCADB. And use the "add/update PEM" function to update the old CFCA DV OCA certificate (https://crt.sh/?id=6970868811). So that there is only the newly issued CFCA DV OCA certificate (https://crt.sh/?id=7833196483) in CCADB, and the old CFCA DV OCA certificate (https://crt.sh/?id=6970868811) cannot be marked as revoked.
2022-11-04: Send an email to introduce the previous operation process of old CFCA DV OCA, and ask whether the certificate can be uploaded to CCADB again and marked as revoked.
2022-11-08: Re-upload the old CFCA DV OCA (https://crt.sh/?id=6970868811) to CCADB and mark it as revoked.

3)Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

CFCA has revoked the intermediate certificate CFCA DV OCA (https://crt.sh/?id=6970868811) on 2022-09-19, and uploaded the intermediate certificate to CCADB on 2022-11-08 and marked it as revoked. This intermediate certificate does not conduct business and does not issue any user certificates.

4)In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

Number of certificates: 1
2022-09-19: Revoke the intermediate certificate CFCA DV OCA (https://crt.sh/?id=6970868811).
2022-11-07: Upload the intermediate certificate CFCA DV OCA (https://crt.sh/?id=6970868811) to CCADB and mark it as revoked.
https://crt.sh/?id=6970868811

5)In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

Number of certificates: 1
2022-09-19: Revoke the intermediate certificate CFCA DV OCA (https://crt.sh/?id=6970868811).
2022-11-07: Upload the intermediate certificate CFCA DV OCA (https://crt.sh/?id=6970868811) to CCADB and mark it as revoked.
https://crt.sh/?id=6970868811

6)Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

According to the requirements of section 4 of the CCADB Policy:"If an intermediate certificate is revoked, the CCADB must be updated to mark it as revoked, giving the reason why, within 24 hours for a security incident, and within 7 days for any other reason".
We should follow CCADB policy to mark intermediate certificate CFCA DV OCA (https://crt.sh/?id=6970868811) as revoked in CCADB.
Before August 2022, Bi Xinlong was CFCA's main contact in CABF 。And only he was in charge of these works. Since August, Bi Xinlong's job responsibilities have changed and he was no longer responsible for SSL certificate business. During the work handover, there was a delayed revocation of the intermediate certificate CFCA DV OCA (https://crt.sh/?id=6970868811), and the work to mark the certificate as revoked in CCADB was not handed over.
2022-10-17,we re-issued the intermediate certificate CFCA DV OCA (https://crt.sh/?id=7833196483).
2022-10-20,we uploaded the intermediate certificate to CCADB.
we used the wrong function "add/update PEM" to update the old CFCA DV OCA certificate (https://crt.sh/?id=6970868811). So that there was only the newly issued CFCA DV OCA certificate (https://crt.sh/?id=7833196483) in CCADB, and the old CFCA DV OCA certificate (https://crt.sh/?id=6970868811) could not be found and marked as revoked.

7)List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

At present, we have used the double responsibility system to conduct mutual supervision and inspection.

1. Bugzilla event reception and processing
   Qiu Dawei ([e-mail]) and Li Karui ([e-mail]) pay attention to the CA compliance section. After receiving the incident, Qiu Dawei is responsible for handling it, and Li Kairui is responsible for the content review and progress review . (2022-9-20)
2. CCADB operation
   Qiu Dawei ([e-mail]) formulates the operation content, submits it to Gao Fei ([e-mail]) for review, and executes it after the review. (2022-11-08)
3. Work handover
   When an employee's work is changed, another employee shoud ensure the work running peoperly.And supervise the handover of work to ensure that nothing is missed.(2022-11-08)

Comment 2

[Chris Clements]

Hi Gao, thank you for this report. In response to the remediation plan in 7):

1. Bugzilla event reception and processing
   Qiu Dawei ([e-mail]) and Li Karui ([e-mail]) pay attention to the CA compliance section. After receiving the incident, Qiu Dawei is responsible for handling it, and Li Kairui is responsible for the content review and progress review . (2022-9-20)

• In addition to monitoring the CA Certificate Compliance component for CFCA incidents reported by third parties, how does CFCA plan to collect and apply lessons learned?
• Are these two individuals also responsible for creating self-reported incidents?

2. CCADB operation
   Qiu Dawei ([e-mail]) formulates the operation content, submits it to Gao Fei ([e-mail]) for review, and executes it after the review. (2022-11-08)

• How is this individual made aware that CCADB content needs to be formulated and sent to yourself for review?
• How are they formulating content? (e.g., will they be an active user of CCADB who regularly logs into the system?)

3. Work handover
   When an employee's work is changed, another employee shoud ensure the work running peoperly.And supervise the handover of work to ensure that nothing is missed.(2022-11-08)

• Several items were not accounted for during the most recent personnel transition, resulting in incidents. How specifically does CFCA plan to ensure this situation will not be repeated in the future?

Comment 3

[Gao Fei]

(In reply to Chris Clements from comment #2)

Hi Gao, thank you for this report. In response to the remediation plan in 7):

1. Bugzilla event reception and processing
   Qiu Dawei ([e-mail]) and Li Karui ([e-mail]) pay attention to the CA compliance section. After receiving the incident, Qiu Dawei is responsible for handling it, and Li Kairui is responsible for the content review and progress review . (2022-9-20)

• In addition to monitoring the CA Certificate Compliance component for CFCA incidents reported by third parties, how does CFCA plan to collect and apply lessons learned?

1.We analyzed all Bugzilla incident reports on CFCA since 2018.
About the error certificate issuance.We think it can be avoided by using automated lint, and have now completed the design of CFCA RA/CA Zlint on November 15th and submitted it for development. It is expected that the deployment and testing of all CFCA historical certificates will be completed in the first quarter of 2023, after which all newly issued certificates will execute Zlint. According to the detection situation, summarize the problems, and optimize the process and system
2. Subscribe to CA Certificate Compliance Components
1）Receive CFCA events in a timely manner, analyze and confirm the problem, and then complete the system repair as soon as possible according to the cause of the problem.
2) At the same time, we are also paying attention to the incidents of other CA agencies, to understand the cause and process of the incident, and to check whether the CFCA has the same problem.
3. According to the previously introduced rules,. We will report the problems found by CFCA's active detection in accordance with the requirements of Bugzilla and br. At the same time,Gao Fei, Qiu Dawei and Li Karui pay attention to RFC, BR, EVG, Mozilla/Microsoft/Google/Apple PKI policy. Keep abreast of the latest updates and apply them to the CFCA certificate processing process and system.

• Are these two individuals also responsible for creating self-reported incidents?

Qiu Dawei ([e-mail]) and Li Karui ([e-mail])are jointly responsible for incident reception, problem analysis and processing, and preparation of incident reports.Gao Fei([e-mail]) will create and submit an incident report after the problem analysis and incident report have been reviewed by him.With Gao Fei as the main contact, we think it will be easier to communicate and connect.

2. CCADB operation
   Qiu Dawei ([e-mail]) formulates the operation content, submits it to Gao Fei ([e-mail]) for review, and executes it after the review. (2022-11-08)

• How is this individual made aware that CCADB content needs to be formulated and sent to yourself for review?
• How are they formulating content? (e.g., will they be an active user of CCADB who regularly logs into the system?)

1.We will log into CCADB at least once every two weeks to check for news or status changes. At the same time, Qiu Dawei or Li Karui will log in from time to time according to work needs and my arrangement.
2.Qiu Dawei and Li Kairui are responsible for the operation of the root certificate and the intermediate certificate. If there is a business change, the operation content will be formulated under my supervision.
The operation content mainly includes:
1)The reason for the operation.
2)The steps and modifications to be performed.
3) Recovery in case of operation error.
3.After confirming that the operation is feasible, they can be authorized to use my account to operate under my supervision.

3. Work handover
   When an employee's work is changed, another employee shoud ensure the work running peoperly.And supervise the handover of work to ensure that nothing is missed.(2022-11-08)

• Several items were not accounted for during the most recent personnel transition, resulting in incidents. How specifically does CFCA plan to ensure this situation will not be repeated in the future?

1. The work of CABF is now jointly responsible by Gaol Fei, Qiu Dawei and Li Kairui.At the same time, we will ensure that no less than 3 people are jointly responsible.
2. At the end of each month, everyone feedbacks and summarizes CABF-related work to form a summary report, which can be used to prevent handover omission when a job handover occurs.
   3.When a person's job changes, that person must submit a completed handover list, which is reviewed and confirmed by two other people.

Comment 4

[Ryan Dickson]

[Lingering bug] - No further questions from the Chrome Root Program.

Comment 5

[Ben Wilson]

I'll schedule to close this on or about Wed. 19-APR-2023.