1524143 - CFCA: Internal iPAddress in certificate

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Corey Bonnell]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0

Steps to reproduce:

Section 7.1.4.2.1 of the Baseline Requirements prohibits the issuance of certificates containing reserved IP addresses.

A search for reserved IP addresses in certificates has yielded the following unexpired, unrevoked certificate issued by CFCA in 2018, which contains a reserved IP address Common Name:

China Financial Certification Authority (CFCA)
crt.sh URL(s),notBefore,notAfter,subject CN,issuer CN
https://crt.sh/?id=469984340 (precert),2018-05-18,2020-05-18,10.23.1.193,CFCA OV OCA

Comment 1

[Jonathan Rudenberg]

I reported this issue to CFCA in an email to their problem reporting address ([e-mail]) at 2019-01-28 04:12 UTC.

Comment 2

[Ryan Sleevi]

I've sent an email to their Primary POC and POC listed in CCADB requesting an incident report.

Comment 3

[Wayne Thayer]

Identified a Bugzilla user and sent another email requesting a response no later than 25-February.

Comment 4

[Jonathan Sun]

Hi, this is Jonathan Sun from CFCA, the following is Bugzilla Report to 1524143:
This bug is reported from Bugzilla bug 1524143, in which wrong OV certificate issued for reserved IP.

Timeline:
In January 28th, we received the email from Jonathan Rudenberg on this bug, we immediately checked the CA records and contacted the customer (Standard Chartered Bank China) to revoke this certificate. The customer reported that this certificate is used in China NetsUnion Clearing Applications. Due to the Spring Festival holidays, the government agency- People’s Bank of China (PBC) requires that all banks in China mainland shouldn’t change anything in their production systems. We had negotiated this problem with PBC and SCB, they agreed to revoke this certificate after the holidays.
In February 11th, SCB started the certificate replacement testing and they finished the certificate replacement testing on February 18th. They arranged the certificate replacement in production system on February 20th.
In February 20th, the SCB finished replacement work in production system. CFCA revoked this wrong certificate in the same day. Now you can check from https://crt.sh/?id=469984340&opt=zlint,ocsp .
Problem analysis：
The system lacked the automatic detecting mechanism on reserved IP. The operator missed this reserved IP warning and sent to RA and CA.

   Actions:

1. The function fixing requirements had been put up to R&D department and it would be deployed by the end of February,
2. We will organize more training on operators’ skills.
3. We will add more two-step verification tools and measures to ensure the accuracy.

Comment 5

[Wayne Thayer]

Jonathan: thank you for the response. In the future, please follow the format described here: https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report

Additional questions:

• Have you scanned your database of active certificates and determined if there are any others with this problem? If so, were any found?
• The failure to revoke this certificate within 5 days is a BR violation. Have you notified your auditors of this violation?
• Mozilla's updated policy on revocation is here: https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation It requires "that you will perform an analysis to determine the factors that prevented timely revocation of the certificates, and include a set of remediation actions in the final incident report that aim to prevent future revocation delays." Please provide this information.

Also, please update this bug when action #1 has been deployed.

Comment 6

[Jonathan Sun]

1. Problem Report:
   CFCA recognized this problematic certificate via a report from Jonathan Rudenburg’s email on January 28th ,2019.
2. Timeline:
   January 28,2019: After we received his email, we immediately checked the CA database and contact the customer Standard Charter Bank. They explained that the problematic certificate is used on a server connect to clearance center (unified clearing platform) which is not used on a website, so we confirm that 0 browser user is affected by this certificate. We confirmed that this certificate is only used on a server to server SSL connection in an inner system, so there is no chance of misuse as an attack method or other misuse scenario.
   January 29,2019: Standard Charter Bank informed us there is strict regulation that forbid them to update or change the system setting during the Spring Festival period (which we call it "The stable period"). So in this case the bank can’t switch to a new certificate, if we insist to revoke the certificate immediately it may cause severe system connection error to the Bank's clearance system, since we understand that this certificate won't affect any browser user, or misuse scenario to harm other user, we decide to negotiate with Standard Charter Bank and settle the certificate switch date after the "stable period".
   February 11, 2019: Standard Charter Bank informed us the stable period is over and we could start the testing procedure for certificate switch.
   February 20, 2019: Standard Charter Bank informed us the testing is finished. CFCA revoked this problematic certificate in the same day.
3. Statement
   CFCA had stopped issuing certificates with the problem.
4. Summary
   This certificate is the only wrong inner IP address certificate which CFCA issued on May 18, 2018
5. Certificate Data:
   Please visit https://crt.sh/?id=469984340&opt=zlint to check the data.
6. Explanation:
   This problem is due to lack of the "Hard fail" detection mechanism and rely too much on the regulation and skill of employees. So this issue is not founded until we are informed.
7. Steps:
8. Update system with hard fail mechanism and this had been finished in February 27,2019.
9. Monthly training about BR requirements to employees.
10. Monthly inner audit to CFCA EV Root CA to prevent future problems.

Followings are response to Wayne :
• Have you scanned your database of active certificates and determined if there are any others with this problem? If so, were any found?
CFCA had scanned our database and there is no more certificates with this problem.

• The failure to revoke this certificate within 5 days is a BR violation. Have you notified your auditors of this violation?
We had notified our auditors of this violation.
• Mozilla's updated policy on revocation is here: https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation It requires "that you will perform an analysis to determine the factors that prevented timely revocation of the certificates, and include a set of remediation actions in the final incident report that aim to prevent future revocation delays." Please provide this information.
First, we contacted our customer Standard charter bank, it appear that they use this certificate on a server connect to the clearance center (unified clearing platform)，which is not used on a website, so we confirm that 0 browser user is affected by this certificate. We confirmed that this certificate is only used on a server to server SSL connection in an inner system, so there is no chance of misuse as an attack method or other misuse scenario.
Second, Standard charter bank informed us there is strict regulation that forbid them to update or change the system setting during the Spring festival period (which we call it "The stable period"). So in this case the bank can’t switch to a new certificate, if we insist to revoke the certificate immediately it may cause severe system connection error to the Bank's clearance system, since we understand that this certificate won't affect any browser user, or misuse scenario to harm other user, we decide to negotiate with standard charter bank and settle the certificate switch date after the "stable period".
In this case , we revoke the certificate in February 20, 2019, after the bank switch it to a new certificate.

Thanks to Wayne, Ryan, and Jonathan again for pointing out this problem.

Comment 7

[Wayne Thayer]

Jonathan: have the actions described in comment #4 been completed?

1. The function fixing requirements had been put up to R&D department and it would be deployed by the end of February,
2. We will organize more training on operators’ skills.
3. We will add more two-step verification tools and measures to ensure the accuracy.

Comment 8

[Jonathan Sun]

(In reply to Wayne Thayer [:wayne] from comment #7)

Jonathan: have the actions described in comment #4 been completed?

1. The function fixing requirements had been put up to R&D department and it would be deployed by the end of February,
2. We will organize more training on operators’ skills.
3. We will add more two-step verification tools and measures to ensure the accuracy.

Yes .the actions had been done in February 27 .

Comment 9

[Ryan Sleevi]

Jonathan: One last question:

The discussion of "holiday freezes" (or stable periods) is one that has been a frequent topic in the CA community over the past several years. Going forward, there's clear expectations that CAs will take steps to minimize the risk to missed revocation due to these - for example, by improving the tooling, training, and expectations for Subscribers, such that certificate rotation - even in such periods - can be accomplished in a timely fashion.

The expectation is that a CA MUST be able to revoke a certificate during such periods. Ideally, CAs minimize this risk through rigid adherance to the BRs, such as pre-issuance linting, careful reviews of all policy changes, a limmited number of certificate profiles (thus reducing variability and risk), not performing manual operator-approved issuance (e.g. favoring automation), etc.

In such past discussions, steps CAs can take have included highlighting the contractual expectations of the Subscriber Agreement (which is required to disclose the obligation of the CA to revoke) and working to help such customers implement automated solutions that can minimize risk.

What steps has CFCA taken, or is planning, to ensure that future misissuance can still be responded to timely, even if it needs to be replaced during a Stable period?

Comment 10

[Jonathan Sun]

(In reply to Ryan Sleevi from comment #9)

Jonathan: One last question:

The discussion of "holiday freezes" (or stable periods) is one that has been a frequent topic in the CA community over the past several years. Going forward, there's clear expectations that CAs will take steps to minimize the risk to missed revocation due to these - for example, by improving the tooling, training, and expectations for Subscribers, such that certificate rotation - even in such periods - can be accomplished in a timely fashion.

The expectation is that a CA MUST be able to revoke a certificate during such periods. Ideally, CAs minimize this risk through rigid adherance to the BRs, such as pre-issuance linting, careful reviews of all policy changes, a limmited number of certificate profiles (thus reducing variability and risk), not performing manual operator-approved issuance (e.g. favoring automation), etc.

In such past discussions, steps CAs can take have included highlighting the contractual expectations of the Subscriber Agreement (which is required to disclose the obligation of the CA to revoke) and working to help such customers implement automated solutions that can minimize risk.

What steps has CFCA taken, or is planning, to ensure that future misissuance can still be responded to timely, even if it needs to be replaced during a Stable period?

The definitions of “Holiday Freezes” and “Stable Periods” is actually not same case in China.
Generally, "Holiday Freezes" means that certificate customer want to keep their own system stable, in holiday they may not have enough engineer or resources to handle the system change or deal with the following work load or mistake that may happen. In this case, If CAs like us can improve service for example like "Operation Free - exchange certificate with one click", or even send an engineer (or online service) to customer to ensure 100% problem free, then it's possible to break "Holiday Freezes". What CFCA can provide now is holiday on-site or on-line service, we do have many maintenance engineers at work on holidays to handle related situations. Still it depends on customers’ arrangement.
On the other side “Stable Periods” means the national administration order directly to our customer which is mandatory to all important departments and enterprises(including banks, government departments, electricity, energy source and so on), normally, even we can provide service like "Operation Free - exchange certificate with one click" they still will refuse the suggestion because of the “Stable Periods”.
However, if the certificate is so “wrong” to "damage" real browser user, for example potential risk of fraud or hacking, we will revoke the certificate regardless of "stable period".
(As stated above, only some kind of company like banks or government departments etc. have this “Stable Periods”, normal company only have “Holiday Freezes”, which is possible to negotiate).
As we stated in the case with the Standard Charter Bank, the certificate is used on an internal server to server connection that not even relate to browser nor the public, we understand that if that certificate is placed on a website for public, or even used on internal employee proxy or cases like that, we will revoke the certificate.
In the end, CFCA will keep trying to negotiate with the government agency about the risk of “replacing a certificate” is acceptable during the “Stable Periods” but it's not likely to success in the near future.

Comment 11

[Ryan Sleevi]

Thanks for the update and clarification, Jonathan.

I think it's important to highlight that "Stable Periods" may be problematic if they prevent revocation. The Baseline Requirements require all CA Subscribers, via the Terms of Use / Subscriber Agreement, agree to recognize that certificate revocation may occur for any of the reasons specified in the Baseline Requirements or the CA's CP/CPS. To that end, understanding what steps CFCA will be taking to ensure all of their Subscribers are capable of adhering to the BRs is important, especially if such a delay were to happen in the future.

The goal of the system is to ensure all CAs - and their customers - are capable of timely replacement of certificates, independent of / regardless of the CA's assessment, if it's for the reasons mentioned in the BRs.

Comment 12

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.