1778035 - CFCA: The wrong status of OCSP

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[bixinlong]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Firefox/102.0

Steps to reproduce:

CFCA found the OCSP has some problem while dealing with another incident (https://bugzilla.mozilla.org/show_bug.cgi?id=1771482), which also report by Michel Le Bihan on June 11, 2022.

Actual results:

When I prepared the incident report, I checked the certificate status, I found an abnormality in the OCSP status of this certificate. Over the next few days, we extract the log and analyze it, we find some cache data is not synchronized after the OCSP system update in May. We updated the test system on June 15 and update the production system in June 16, the problem has been fixed when the system was updated

Comment 1

[bixinlong]

1. Problem Report:
   CFCA found the OCSP has some problem while dealing with another incident ((https://bugzilla.mozilla.org/show_bug.cgi?id=1771482) , which also report by Michel Le Bihan on June 11, 2022.

2. Timeline:
   June 1, 2022: We received an report about a certificate was wrongly issued , we immediately communicated with our certificate issuing department and confirmed the certificate was wrongly issued. Subsequently, we planned to revoke the certificate as soon as possible and re-issue a certificate. We contacted the customer to explain the situation, the customer resubmitted a application and we issued a new certificate on the same day, they hoped to revoke the certificate after it was replaced.
   June 2, 2022: The certificate was replaced on June 2, we revoked the wrong certificate on the same day.
   June 5, 2022: When I prepared the incident report, I checked the certificate status, I found an abnormality in the OCSP status of this certificate. Over the next few days, we extract the log and analyze it, we find some cache data is not synchronized after the OCSP system update in May.
   June 15, 2022: We updated the test system on June 15 and update the production system in June 16, the problem has been fixed when the system was updated.

3. Statement
   CFCA have fixed this.

4. Summary
   The incident caused the OCSP service to be abnormal from May 5th to June 15th. According to statistics, about 14 certificates were temporarily affected during this period, but these were resolved as the system was updated.

5. Explanation:
   This problem is due some cache data is not synchronized after the OCSP system update in May.

6. Steps:
   (1). We will pay more attention to the verification after system update
   (2). We plan to add preissuance and postissuance lints on the certificate system, to ensure that errors can be discovered and resolved timely.

Comment 2

[Ben Wilson]

Please provide the actions you have taken thus far to:

• pay more attention to acceptance testing after system updates
• add pre-issuance and post-issuance lints on your certificate system

Also, please provide a schedule of your future efforts that will address the deficiencies identified by the foregoing.

Thanks,

Ben

Comment 3

[bixinlong]

(In reply to Ben Wilson from comment #2)

Please provide the actions you have taken thus far to:

• pay more attention to acceptance testing after system updates
• add pre-issuance and post-issuance lints on your certificate system

Also, please provide a schedule of your future efforts that will address the deficiencies identified by the foregoing.

Thanks,

Ben

I would like to provide some information, this might help you understand what we do.

We adjusted our testing methods. In addition to certificate management functions (such as certificate application and certificate update), we will also test non-certificate operation functions (such as the OCSP status of the current certificate, whether the CRL is normally after certificate was revoked). The scope of the test is much wider, I think it might help us identify problems in time or find some unexpected errors during the testing phase.

We've already designed pre-issuance and post-issuance lints on the certificate system, It is currently under development, according to current progress, it is expected to be ready for use in mid-October.

Thinks.

Comment 4

[Ryan Dickson]

(Following-up in advance of the Next Update date)

@bixinlong, can CFCA share more detail regarding the planned linting implementation described in Comment 3?

• Provide a more specific set of remaining actions and timebound milestones related to the mid-October deployment.
• Describe whether the lints are custom built, or whether CFCA is planning to use an open-source solution with wide community support and regular updates in response to changes to the BRs and Web PKI incident (e.g., ZLint).
• Describe, specifically, how linting will be implemented such that it prevents the pattern of mis-issuance we have observed in CFCA's bugs over the past few years.
• How do you intend on verifying the linting implementation is working as intended (i.e., do you intend on running the lints against the certificates disclosed in CFCA’s previous bugs to ensure known issues are detected?)
• Will CFCA commit to running the lints against the complete certificate corpus in effort to detect other potential issues not previously reported?
• How do you intend to maintain existing lints, or add new ones, in the future?
• Clarify “we adjusted our testing methods” to describe what specifically has been adjusted.

Thank you!

Comment 5

[Gao Fei]

(In reply to Ryan Dickson from comment #4)

(Following-up in advance of the Next Update date)

@bixinlong, can CFCA share more detail regarding the planned linting implementation described in Comment 3?

• Provide a more specific set of remaining actions and timebound milestones related to the mid-October deployment.
• Describe whether the lints are custom built, or whether CFCA is planning to use an open-source solution with wide community support and regular updates in response to changes to the BRs and Web PKI incident (e.g., ZLint).
• Describe, specifically, how linting will be implemented such that it prevents the pattern of mis-issuance we have observed in CFCA's bugs over the past few years.
• How do you intend on verifying the linting implementation is working as intended (i.e., do you intend on running the lints against the certificates disclosed in CFCA’s previous bugs to ensure known issues are detected?)
• Will CFCA commit to running the lints against the complete certificate corpus in effort to detect other potential issues not previously reported?
• How do you intend to maintain existing lints, or add new ones, in the future?
• Clarify “we adjusted our testing methods” to describe what specifically has been adjusted.

Thank you!

Hi Ryan,
In response to your question, we can provide the following information:

Provide a more specific set of remaining actions and timebound milestones related to the mid-October deployment.
Re:
(1) The testing tool is under development and is expected to be tested before October 10th and deploy on October 15th.
(2) In the future, we will upgrade the testing tool and introduce Zlint, which is scheduled to release and deploy in March 2023.

Describe whether the lints are custom built, or whether CFCA is planning to use an open-source solution with wide community support and regular updates in response to changes to the BRs and Web PKI incident (e.g., ZLint).
Re:
(1) The Lint used by our tool is custom and will introduce Zlint in the future.
(2) In addition, Lint benchmark references: 1)RFCS (eg: RFC 6818,RFC 4055,RFC 2560,RFC 8399,RFC 3467); 2) BR, EV Guidelines; 3)Mozilla PKI policy; 4)CT policy.
(3) We will arrange specially assigned employee to check the above benchmarks every month. If there is a new policy, the change will be completed before the time point required by the policy.

Describe, specifically, how linting will be implemented such that it prevents the pattern of mis-issuance we have observed in CFCA's bugs over the past few years.
Re:
(1) According to RFC, BR, EV Guidelines, Mozilla PKI Policy and CT policy, we set up detection standards and detection items
(2) Before issuing the certificate, testing tool will verify the received certificate application according to the established standards. If the verification fails, reject the application.
(3) In addition, testing tool will verify the basic certificate fields and extended fields item by item on the pre-certificate issued. If the verification fails, we will refuse to send it to CTlog.
(4) Finally we will check the basic certificate fields and extended fields item by item after the certificate is issued (this step includes the CT verification test). If the certificate fails to pass the verification, we will immediately revoke the certificate and re-check it.

How do you intend on verifying the linting implementation is working as intended (i.e., do you intend on running the lints against the certificates disclosed in CFCA’s previous bugs to ensure known issues are detected?)
Re:
Before the tool is released, we will use the tool to check the certificates that have been disclosed to have problems and other test certificates that have problems to ensure that the tool will work in the correct way to detect the corresponding problems.

Will CFCA commit to running the lints against the complete certificate corpus in effort to detect other potential issues not previously reported?
Re:
After the testing tool is released in mid-October, we will test all historical certificates issued by the CFCA EV ROOT, including the basic certificate fields and extended fields of all historical certificates.

How do you intend to maintain existing lints, or add new ones, in the future?
Re:
(1) For the existing lints, we will regularly and continuously pay attention to RFC, BR, EV Guidelines, Mozilla PKI policy and CT policy to timely obtain changes in policy requirements and update them in time.
(2) In addition, double validation will be implemented in the future by introducing Zlint.

Clarify “we adjusted our testing methods” to describe what specifically has been adjusted.
Re:
(1) We have upgraded the RA system. No matter the website or API interface, once the entered field does not meet the character type requirements or length limit requirements, an error will be prompted and the certificate application will be rejected.
(2) Add the certificate status verification steps:
a. After the certificate have been issued, we will verify the OCSP response results and status;
b. After the certificate have been revoked, we will verify OCSP and CRL response results and status

Thanks,

Gao Fei

Comment 6

[Ryan Dickson]

Hi Gao,

A few questions are inline below.

Provide a more specific set of remaining actions and timebound
milestones related to the mid-October deployment. Re: (1) The
testing tool is under development and is expected to be tested before
October 10th and deploy on October 15th. (2) In the future, we will
upgrade the testing tool and introduce Zlint, which is scheduled to
release and deploy in March 2023.

RESPONSE: This timeline is still very high-level. For the introduction of ZLint, what specific steps still need to be completed by CFCA before it's enabled by default for all pre-certificates and/or certificates issued, and when are they planned to take place? (i.e., provide more specificity as to what is happening between now and March 2023)

Describe whether the lints are custom built, or whether CFCA is
planning to use an open-source solution with wide community support
and regular updates in response to changes to the BRs and Web PKI
incident (e.g., ZLint). Re: (1) The Lint used by our tool is
custom and will introduce Zlint in the future. (2) In addition, Lint
benchmark references: 1)RFCS (eg: RFC 6818,RFC 4055,RFC 2560,RFC
8399,RFC 3467); 2) BR, EV Guidelines; 3)Mozilla PKI policy; 4)CT
policy. (3) We will arrange specially assigned employee to check the
above benchmarks every month. If there is a new policy, the change
will be completed before the time point required by the policy.

Using custom or closed-source linting tools often makes it difficult for external stakeholders to understand the types of issues that will be detected/prevented. It also makes it difficult to know how the tools will be supported long-term, for example fixing bugs or adding new lints as new requirements and/or best practices are identified.

Though referencing the basis for which lints are defined is helpful (i.e., the RFCs and policies you mentioned), it would be more helpful to the community if detail was offered to help us better understand how CFCA's past issues, and other common mis-issuance problems will be prevented through this new approach. For example, the ZLint project maintains a set of test coverage here. Can CFCA share the specific set of tests being considered through its custom tool to help us understand what is in the scope of the intended solution?

Describe, specifically, how linting will be implemented such that it
prevents the pattern of mis-issuance we have observed in CFCA's bugs
over the past few years. Re: (1) According to RFC, BR, EV
Guidelines, Mozilla PKI Policy and CT policy, we set up detection
standards and detection items (2) Before issuing the certificate,
testing tool will verify the received certificate application
according to the established standards. If the verification fails,
reject the application. (3) In addition, testing tool will verify
the basic certificate fields and extended fields item by item on the
pre-certificate issued. If the verification fails, we will refuse to
send it to CTlog. (4) Finally we will check the basic certificate
fields and extended fields item by item after the certificate is
issued (this step includes the CT verification test). If the
certificate fails to pass the verification, we will immediately revoke
the certificate and re-check it.

RESPONSE: See above response - without explicit reference to the specific tests certificates are being evaluated against, it's challenging to understand whether this will address CFCA's past mis-issuances. Summarizing your response, CFCA will perform both pre- and post-issuance linting. That's great to read.

Will CFCA commit to running the lints against the complete certificate
corpus in effort to detect other potential issues not previously
reported? Re: After the testing tool is released in mid-October,
we will test all historical certificates issued by the CFCA EV ROOT,
including the basic certificate fields and extended fields of all
historical certificates.

RESPONSE: That's great to read. Thanks for confirming. Can you confirm the same commitment once ZLint is implemented, as well?

How do you intend to maintain existing lints, or add new ones, in the
future? Re: (1) For the existing lints, we will regularly and
continuously pay attention to RFC, BR, EV Guidelines, Mozilla PKI
policy and CT policy to timely obtain changes in policy requirements
and update them in time. (2) In addition, double validation will be
implemented in the future by introducing Zlint.

RESPONSE: Can you provide more specificity to "regularly," "continuously," and "timely" to quantify the effort? For example: "Within X hours/days/weeks of an update to Y, we will accomplish Z by doing A, B, and C."

Also, given the intent to use multiple tools, how will CFCA respond if there's a discrepancy between your custom tool and ZLint?

Thanks!
-Ryan

Comment 7

[Gao Fei]

(In reply to Ryan Dickson from comment #6)

Hi Gao,

A few questions are inline below.

Provide a more specific set of remaining actions and timebound
milestones related to the mid-October deployment. Re: (1) The
testing tool is under development and is expected to be tested before
October 10th and deploy on October 15th. (2) In the future, we will
upgrade the testing tool and introduce Zlint, which is scheduled to
release and deploy in March 2023.

RESPONSE: This timeline is still very high-level. For the introduction of ZLint, what specific steps still need to be completed by CFCA before it's enabled by default for all pre-certificates and/or certificates issued, and when are they planned to take place? (i.e., provide more specificity as to what is happening between now and March 2023)
RE:A.Complete the functional design of CFCA RA/CA Zlint before 2022-11-15.
B.Complete the development of CFCA RA/CA Zlint function before 2022-12-30.
C.Complete functional verification in the test environment before 2023-1-30.
D.CFCA RA/CA Zlint will be officially launched before 2023-2-25.
E.Before March 30, 2023, complete the inspection of the CFCA historical certificate, and deal with any problems that found.

Describe whether the lints are custom built, or whether CFCA is
planning to use an open-source solution with wide community support
and regular updates in response to changes to the BRs and Web PKI
incident (e.g., ZLint). Re: (1) The Lint used by our tool is
custom and will introduce Zlint in the future. (2) In addition, Lint
benchmark references: 1)RFCS (eg: RFC 6818,RFC 4055,RFC 2560,RFC
8399,RFC 3467); 2) BR, EV Guidelines; 3)Mozilla PKI policy; 4)CT
policy. (3) We will arrange specially assigned employee to check the
above benchmarks every month. If there is a new policy, the change
will be completed before the time point required by the policy.

Using custom or closed-source linting tools often makes it difficult for external stakeholders to understand the types of issues that will be detected/prevented. It also makes it difficult to know how the tools will be supported long-term, for example fixing bugs or adding new lints as new requirements and/or best practices are identified.

Though referencing the basis for which lints are defined is helpful (i.e., the RFCs and policies you mentioned), it would be more helpful to the community if detail was offered to help us better understand how CFCA's past issues, and other common mis-issuance problems will be prevented through this new approach. For example, the ZLint project maintains a set of test coverage here. Can CFCA share the specific set of tests being considered through its custom tool to help us understand what is in the scope of the intended solution?
RE:CFCA custom lint to be used on October 10, 2022 mainly includes the following items
A.The domain is no longer than 64 characters;
B.Public_suffix_list be used for domain verification；
C.Verify whether the characters entered in the domain are within the range of letters, numbers,"-" ,"*","." and other requirements
D.Public IP cannot include reserved IP;
E.The organization name not exceed 64 characters;
F.The length of the city not exceed 128 characters;
G.The length of provinces not exceed 128 characters;
H.National/regional calibration uses ISO 3166-1;
I.The business street not exceed 128 characters;
J.Postal code not exceed 16 characters;
K.Verify that the zip code input characters are within the PrintableString range;
L.Verify OCSP address and response
M.Verify whether the user certificate enhancement key usage exists
N.Verify SubjAltName entry is not empty
O.Check CRL address
P.Verify whether the CRL and OCSP certificate status are consistent
Q.Verify whether the validity of the certificate is less than 398 days
R.Verify whether the certificate serial number is greater than 64 bits

Describe, specifically, how linting will be implemented such that it
prevents the pattern of mis-issuance we have observed in CFCA's bugs
over the past few years. Re: (1) According to RFC, BR, EV
Guidelines, Mozilla PKI Policy and CT policy, we set up detection
standards and detection items (2) Before issuing the certificate,
testing tool will verify the received certificate application
according to the established standards. If the verification fails,
reject the application. (3) In addition, testing tool will verify
the basic certificate fields and extended fields item by item on the
pre-certificate issued. If the verification fails, we will refuse to
send it to CTlog. (4) Finally we will check the basic certificate
fields and extended fields item by item after the certificate is
issued (this step includes the CT verification test). If the
certificate fails to pass the verification, we will immediately revoke
the certificate and re-check it.

RESPONSE: See above response - without explicit reference to the specific tests certificates are being evaluated against, it's challenging to understand whether this will address CFCA's past mis-issuances. Summarizing your response, CFCA will perform both pre- and post-issuance linting. That's great to read.

Will CFCA commit to running the lints against the complete certificate
corpus in effort to detect other potential issues not previously
reported? Re: After the testing tool is released in mid-October,
we will test all historical certificates issued by the CFCA EV ROOT,
including the basic certificate fields and extended fields of all
historical certificates.

RESPONSE: That's great to read. Thanks for confirming. Can you confirm the same commitment once ZLint is implemented, as well?
RE:Once ZLint is implemented, we can confirm the same commitment.And within one week,we will complete the inspection of the CFCA historical certificate, and deal with any problems that found.

How do you intend to maintain existing lints, or add new ones, in the
future? Re: (1) For the existing lints, we will regularly and
continuously pay attention to RFC, BR, EV Guidelines, Mozilla PKI
policy and CT policy to timely obtain changes in policy requirements
and update them in time. (2) In addition, double validation will be
implemented in the future by introducing Zlint.

RESPONSE: Can you provide more specificity to "regularly," "continuously," and "timely" to quantify the effort? For example: "Within X hours/days/weeks of an update to Y, we will accomplish Z by doing A, B, and C."

RE:A.Complete the functional design of CFCA RA/CA Zlint before 2022-11-15.
B.Complete the development of CFCA RA/CA Zlint function before 2022-12-30.
C.Complete functional verification in the test environment before 2023-1-30.
D.CFCA RA/CA Zlint will be officially launched before 2023-2-25.
E.Before March 30, 2023, complete the inspection of the CFCA historical certificate, and deal with any problems that found.

Also, given the intent to use multiple tools, how will CFCA respond if there's a discrepancy between your custom tool and ZLint?

RE:If there are differences between Zlint and CFCA custom lint verification results.We will be subject to the result of Zlint.
But at the same time,we will find the reason for the difference according to the warning.

Thanks!
-Ryan

Comment 8

[Gao Fei]

We have completed the application of ZLint service on February 28. Only after passing the Zlint verification, the certificates are allowed to be issued.

Comment 9

[Gao Fei]

Use Zlint to detect the previously issued certificates, which is currently being processed, and we will provide a full report before April 9.

Comment 10

[Gao Fei]

Attachment: CFCA Historically Issued SSL Certificate Zlint Detection.pdf

Comment 11

[Gao Fei]

Attachment: wrong certificate list.xlsx

Comment 12

[Gao Fei]

CFCA Work Improvement in the Past Six Months:
1.Since September 2021, the arrangement: Gao Fei, Qiu Dawei, and Li Kairui jointly completed the review and comparison of PKI policies (RFC, BR, EVG, Root policy, CT policy, etc.), Bugzilla case review and reply, CCADB maintenance, CPS revision and audit, etc.
2.From October 2022 to January 2023, we newly formulated and updated and updated the “Risk Event Handling Specification”, “Bugzilla Incident Handling Specification”, “Certificate Operation Management Specification”, “CCADB Management Measures”, “CT Management Specification”, etc.In order to more standardize the implementation of certificate operation and management.
3.In February 2023, we have applied Zlint auto-detection function. All certificates will be tested by Zlint before they are issued. Only certificates that pass the test will be allowed to be issued to avoid wrong certificates.

CFCA’s Next Work Plan:
1.Promote comprehensive automation management
(1)Based on ACME to realize automatic certificate application, issuance, installation, update and other functions.
(2)Based on Zlint detection, it realizes the function of email notification of abnormal test results and automatic revocation within 7 days.
2.Strengthen internal audit Invest more manpower and energy to increase the frequency of internal audits and improve the quality of internal audits.

Historically Issued SSL Certificate Zlint Detection:
A total of 11,724 test certificates were issued with 179 wrong certificates. The main problems include the following categories:
(1)Wrong SerialNumber encoding
Quantity: 160
This problem was caused by a bug in the CA system. These wrong certificates were issued before July 2018. The CA system function was updated in August 2018 to fix the problem. After that, this type of error did not occur. Related issues have been previously discussed in the following tickets:
https://bugzilla.mozilla.org/show_bug.cgi?id=1532559.

(2)Invalid TLD in SAN / invalid dnsNames / Internal iP Address in certificate
Quantity: 8
This problem is caused by the incomplete domain name or IP verification function of the system. These error certificates were issued before February 2019. The RA system function was updated in February 2019 to fix this problem. After that, this type of error did not occur. Related issues have been previously discussed in the following tickets:
https://bugzilla.mozilla.org/show_bug.cgi?id=1532429
https://bugzilla.mozilla.org/show_bug.cgi?id=1524733
https://bugzilla.mozilla.org/show_bug.cgi?id=1524143

3)Certificate with wrong PostalCode
Quantity: 6
This problem was caused by staff mistakes and the failure of the system to strictly verify the relevant fields. These error certificates were issued before September 2022, and the RA system function was updated in November 2022 to fix this problem. After that, this type of error did not occur. Related issues have been previously discussed in the following tickets:
https://bugzilla.mozilla.org/show_bug.cgi?id=1771482
https://bugzilla.mozilla.org/show_bug.cgi?id=1802845

(4)Certificate with wrong crlDistributionPoints
Quantity: 3
This problem was caused by the RA system configuration error during the Zlint upgrade function verification process, and Zlint was not enabled. The Zlint upgrade was completed at the end of February 2023. All certificates will be tested by Zlint, and this type of error has not occurred since then. Related issues have been previously discussed in the following tickets:
https://bugzilla.mozilla.org/show_bug.cgi?id=1809382

(5) organization more than 64 characters
Quantity: 2
This problem is caused by the incomplete organization name verification function of the system. These error certificates were issued before December 2018. The RA system function was updated in February 2019 to fix this problem. After that, this type of error did not occur. Related issues have been previously discussed in the following tickets:
https://bugzilla.mozilla.org/show_bug.cgi?id=1532113

The above-mentioned problematic certificates have all expired or been revoked. The certificates within the current validity period and in the activated state are all valid after scanning by Zlint.

Comment 13

[Ryan Dickson]

Thank you for sharing the historical lint analysis, Gao.

If you have any lessons learned that you can share related to your ZLint implementation that would benefit other CAs, we welcome you to share them.

No further questions from the Chrome Root Program.

Comment 14

[Ben Wilson]

I will schedule to close this on or about Wed. 19-Apr-2023.