Closed Bug 1532113 Opened 6 years ago Closed 5 years ago

CFCA: O > 64 characters

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: michel, Assigned: sunny_bxl)

References

Details

(Whiteboard: [ca-compliance] [leaf-revocation-delay])

User Agent: Mozilla/5.0 (Android 9; Mobile; rv:65.0) Gecko/65.0 Firefox/65.0

Steps to reproduce:

Hello,

I found certificates issued by CFCA that have the organizationName longer than 64 characters. They also have keyUsage not marked as critical.

https://crt.sh/?id=1057895165&opt=cablint,x509lint,zlint
https://crt.sh/?id=1196150093&opt=cablint,x509lint,zlint
https://crt.sh/?id=673767650&opt=cablint,x509lint,zlint
https://crt.sh/?id=726116242&opt=cablint,x509lint,zlint

Jonathan: Please provide an incident report, as per https://wiki.mozilla.org/CA/Responding_To_An_Incident

Assignee: wthayer → jonathansshn
Whiteboard: [ca-compliance]
QA Contact: kwilson → wthayer
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Flags: needinfo?(jonathansshn)

Emailed POCs on 2019-07-04 regarding this issue, highlighting https://wiki.mozilla.org/CA/Responding_To_An_Incident#Keeping_Us_Informed

(In reply to Wayne Thayer [:wayne] from comment #1)

Jonathan: Please provide an incident report, as per https://wiki.mozilla.org/CA/Responding_To_An_Incident

  1. Problem Report:
    CFCA recognized this problematic certificate via a report from Michel Le Bihan’s email on March 2th ,2019.
  2. Timeline:
    March 4 , 2019:After we received his email, we immediately checked the CA database and contact the customer “Department of Human Resources and Social Security of Jiangxi Province” and “Ganzi Government Affairs Service and Public Resources Exchange Service Center”. They explained that the certificates was planned to be used on a website for some new servers.
    March 18 ,2019: Department of Human Resources and Social Security of Jiangxi Province informed us there is strict regulation that forbid them to update or change the system setting, because of they are used a wildcard certificate, they need time to sort out the systems. So in this case the Department can’t switch to a new certificate, if we insist to revoke the certificate immediately it may cause severe system connection error, we decide to negotiate with Department of Human Resources and Social Security of Jiangxi Province and settle the certificate switch date after the "stable period".
    March 18, 2019: Ganzi Government Affairs Service and Public Resources Exchange Service Center informed us that they didn’t started using the certificate yet, we can revoked the certificate. So CFCA revoked this certificate in the same day.
  3. Statement
    CFCA had stopped issuing certificates with the problem.
  4. Summary
    CFCA only issued two certificates of the organizationName longer than 64 characters certificate
  5. Certificate Data:
    Please visit https://crt.sh/?id=1057895165&opt=cablint,x509lint,zlint
    https://crt.sh/?id=1196150093&opt=cablint,x509lint,zlint
    https://crt.sh/?id=673767650&opt=cablint,x509lint,zlint
    https://crt.sh/?id=726116242&opt=cablint,x509lint,zlint to check the data.
  6. Explanation:
    This problem is due to the bug of RA, the RA didn't checking the length of organizationName. So this issue is not founded until we are informed.
  7. Steps:
  8. We update RA system to fixing the function and this had been finished in February 27, 2019.
  9. Monthly training about BR requirements to employees, especially the R&D people.
  10. Monthly inner audit to CA database to prevent future problems.

Oliver,

Thank you for providing your details. However, this problem report is quite inadequate.

  • It does not address why the problem happened.
  • It is not clear why the RA system is relevant, since the CA is responsible for signing.
  • It does not suggest that you have examined your systems for other longstanding (20+ year) requirements. These are expressed within the ASN.1 schema itself, and so it's unclear how and why they were violated.
  • it does not provide assurance that, going forward, CFCA will perform timely and BR-required revocation. For example, the notion of "stable periods" is not an acceptable reason to violate the BRs, and the CAs are responsible for both ensuring customers are aware of this and making sure that its own systems support these customers. For example, your Subscriber Agreement is required to contractually give the CA the ability to revoke, and that's something CAs can and should regularly remind their customers of, so that their customers can plan accordingly.
Flags: needinfo?(jonathansshn) → needinfo?(sunny_bxl)

Also:

  • when will *.jxhrss.gov.cn be revoked?
  • please explain the "monthly inner audit to CA database". What percentage of certificates will be audited? Will linters be used to detect problems?
  • is CFCA planning to implement pre-issuance linting as a way to detect and prevent these kinds of system "bugs"?

(In reply to Ryan Sleevi from comment #4)

Oliver,

Thank you for providing your details. However, this problem report is quite inadequate.

  • It does not address why the problem happened.
  • It is not clear why the RA system is relevant, since the CA is responsible for signing.
  • It does not suggest that you have examined your systems for other longstanding (20+ year) requirements. These are expressed within the ASN.1 schema itself, and so it's unclear how and why they were violated.
  • it does not provide assurance that, going forward, CFCA will perform timely and BR-required revocation. For example, the notion of "stable periods" is not an acceptable reason to violate the BRs, and the CAs are responsible for both ensuring customers are aware of this and making sure that its own systems support these customers. For example, your Subscriber Agreement is required to contractually give the CA the ability to revoke, and that's something CAs can and should regularly remind their customers of, so that their customers can plan accordingly.

Hi, Ryan:

1、The reason for the problem is the previous versions of RA only limited the total length of DN characters, but didn’t limit the length of organizationName, so the organizationName is longer than 64 characters
2、The pattern we designed before was that the rule validation was carried out by RA, and the CA was only responsible for issuing certificates. In light of this incident, We will re-evaluate the reliability of this validation mode, in order to find a better solution and avoid similar problems in the future.
3、The recent feedback problems has attracted the attention of our product team and senior leaders. We will conduct an in-depth inspection of RA and CA systems, existing data and other related businesses to ensure that longstanding requirements are met. Meanwhile, in the following business, we will strengthen remind the customers of the CA revocation ability in the agreement. When the similar events happens, CFCA can timely implement the revocation required.

Flags: needinfo?(sunny_bxl)

(In reply to Wayne Thayer [:wayne] from comment #5)

Also:

  • when will *.jxhrss.gov.cn be revoked?
  • please explain the "monthly inner audit to CA database". What percentage of certificates will be audited? Will linters be used to detect problems?
  • is CFCA planning to implement pre-issuance linting as a way to detect and prevent these kinds of system "bugs"?

Hi, Wayne:
1、*.jxhrss.gov.cn was revoked on July 9, 2019.
2、The previous monthly inner audit focused more on the newly added data of the month, and we will carry out a comprehensive and in-depth inspection to cover all the data and this work is now underway.
3、Thanks for Wayne, your suggestion sounds very good. We'll do some research before we decide.

(In reply to Oliver Bi from comment #7)
Hi Oliver,

1、*.jxhrss.gov.cn was revoked on July 9, 2019.

Why was this certificate not revoked until 4 months after the incident was reported, in violation of the BRs and Mozilla policy?

2、The previous monthly inner audit focused more on the newly added data of the month, and we will carry out a comprehensive and in-depth inspection to cover all the data and this work is now underway.

Please explain what this audit will examine and when it will be completed. Will this comprehensive and in-depth inspection be performed only once,? Also, please respond in this bug with a summary of the results, including any issues identified by the audit.

3、Thanks for Wayne, your suggestion sounds very good. We'll do some research before we decide.

When will CFCA be prepared to announce the decision?

Flags: needinfo?(sunny_bxl)

(In reply to Wayne Thayer [:wayne] from comment #8)

(In reply to Oliver Bi from comment #7)
Hi Oliver,

1、*.jxhrss.gov.cn was revoked on July 9, 2019.

Why was this certificate not revoked until 4 months after the incident was reported, in violation of the BRs and Mozilla policy?

2、The previous monthly inner audit focused more on the newly added data of the month, and we will carry out a comprehensive and in-depth inspection to cover all the data and this work is now underway.

Please explain what this audit will examine and when it will be completed. Will this comprehensive and in-depth inspection be performed only once,? Also, please respond in this bug with a summary of the results, including any issues identified by the audit.

3、Thanks for Wayne, your suggestion sounds very good. We'll do some research before we decide.

When will CFCA be prepared to announce the decision?

  1. As the report said, we communicate with our customer, told the customer about this problem and hope to revoked timely, but they feedback that the certificate has been deployed on a system, any changes need to be approved. We issued new certificate for them on April 4, 2019, so that the replacement can be completed as soon as possible. But the work wasn't finished until July 9, then we revoked the certificate at the same day.
  2. The monthly inner audit examine whether the content and the length of the certificate exceed the standard, this will be done by the end of this month. The audit cycle will be changed to a quarterly basis after July, including data on certificates issued, application materials, CSR compliance and so on.
  3. This date are not sure now. We are researching some technical documents, and our R&D department will evaluate the technical feasibility. Before that, we will add a secondary inspection mechanism by human.
Flags: needinfo?(sunny_bxl)

Oliver: thank you for the update. Please add the results of this month's internal audit when you have them, and also provide an update when CFCA has decided if and when to implement pre-issuance linting.

Assignee: jonathansshn → sunny_bxl
Whiteboard: [ca-compliance] → [ca-compliance] - Next Update - 01-August 2019

Oliver: It has now been 2 months since I asked the questions in comment #10. Why has CFCA not responded?

Flags: needinfo?(sunny_bxl)

(In reply to Wayne Thayer [:wayne] from comment #11)

Oliver: It has now been 2 months since I asked the questions in comment #10. Why has CFCA not responded?

Sorry for the late reply. We finished the internal audit in early August, and then we started the Webtrust external audit. I planned to submit the results after the external audit, but the external audit is still in progress, this is my personal fault.

Now, I would like to explain our internal audit from May 1, 2019 to July 31, 2019:

  1. During this period, CFCA has issued 33 EV certificates, 176 OV certificates and 26 document signature certificates. We have audited these 235 records, and the audit results show that the certificate acceptance, audit process, etc are in line with the requirements.
  2. However, it is pointed out in the report that at present, certificate processing is over-dependent on labor, and there is a risk of not rigorous audit. Hope improve it timely.

As I said before, we have realized the transition rely on artificial risks, so we have already start the automatic ordering system development work, the system is suitable for final subscriber and the internal audit personnel, through the big data technology to complete the authentication of the enterprise, realize automatic domain authentication, thus reducing the risk of manual audit. Considering other factors, we will retain the manual secondary audit process in the early stage.

Due to the complex design, we plan to complete it in stages. Currently, the first version of the system has been developed. The system passed the review on September 10, 2019, the test system is currently being deployed, if there is no problem with the test system, we will start phase ii construction and join the automatic audit. We hope to finish it by the end of this year, there maybe some uncertainty, but we will try to solve them.

Flags: needinfo?(sunny_bxl)

Oliver: What is the status of this issue?

Also, please explain why CFCA has not provided any updates on this issue for 4 months. As already pointed out in bug #1524733, Mozilla expects weekly updates (https://wiki.mozilla.org/CA/Responding_To_An_Incident#Keeping_Us_Informed).

Flags: needinfo?(sunny_bxl)

(In reply to Wayne Thayer [:wayne] from comment #13)

Oliver: What is the status of this issue?

Also, please explain why CFCA has not provided any updates on this issue for 4 months. As already pointed out in bug #1524733, Mozilla expects weekly updates (https://wiki.mozilla.org/CA/Responding_To_An_Incident#Keeping_Us_Informed).

Wayne:

I think this issue has been solved now, according to the plan, the automated ordering system has been developed and completed (https://cloudpki.cfca.com.cn), although a little trouble in the process.

The internal training has be done on January 15th. Since January 25 is the Chinese Spring Festival, there will be a holiday between January 23 and 30, so we plan to officially launch the system after the Spring Festival and then update the report.

Sorry for the late update, because it is the Spring Festival in China these days, I might respond slowly, I will submit the final report before February 5th, is that all right?

Flags: needinfo?(sunny_bxl)

(In reply to Oliver Bi from comment #14)

Sorry for the late update, because it is the Spring Festival in China these days, I might respond slowly, I will submit the final report before February 5th, is that all right?

Yes - I have set the "Next Update" to February 5th.

Whiteboard: [ca-compliance] - Next Update - 01-August 2019 → [ca-compliance] - Next Update - 05-Febrruary 2020

(In reply to Wayne Thayer from comment #15)

(In reply to Oliver Bi from comment #14)

Sorry for the late update, because it is the Spring Festival in China these days, I might respond slowly, I will submit the final report before February 5th, is that all right?

Yes - I have set the "Next Update" to February 5th.

Final Report

1. Problem Report:

CFCA recognized this problematic certificate via a report from Michel Le Bihan’s email on March 2th ,2019.

2. Timeline:

March 4 , 2019:After we received his email, we immediately checked the CA database and contact the customer “Department of Human Resources and Social Security of Jiangxi Province” and “Ganzi Government Affairs Service and Public Resources Exchange Service Center”. They explained that the certificates was planned to be used on a website for some new servers.
March 18 ,2019: Department of Human Resources and Social Security of Jiangxi Province informed us there is strict regulation that forbid them to update or change the system setting, because of they are used a wildcard certificate, they need time to sort out the systems. So in this case the Department can’t switch to a new certificate, if we insist to revoke the certificate immediately it may cause severe system connection error, we decide to negotiate with Department of Human Resources and Social Security of Jiangxi Province and settle the certificate switch date after the "stable period".
March 18, 2019: Ganzi Government Affairs Service and Public Resources Exchange Service Center informed us that they didn’t started using the certificate yet, we can revoked the certificate. So CFCA revoked this certificate in the same day.

3. Statement

CFCA had stopped issuing certificates with the problem and fixed the system bugs.

4. Summary

CFCA only issued two certificates of the organizationName longer than 64 characters certificate, no impact on other customers.

5. Certificate Data:

Please visit https://crt.sh/?id=1057895165&opt=cablint,x509lint,zlint
https://crt.sh/?id=1196150093&opt=cablint,x509lint,zlint
https://crt.sh/?id=673767650&opt=cablint,x509lint,zlint
https://crt.sh/?id=726116242&opt=cablint,x509lint,zlint to check the data.

6. Explanation:

This problem is due to the bug of RA, the previous versions of RA only limited the total length of DN characters, didn’t limit the length of organizationName.

7. Steps:

  1. The RA bugs had been fixed on February 27, 2019.
  2. Monthly training about BR requirements to employees, especially the R&D people.
  3. More stricter monthly inner audit to CA database to prevent future problems.
  4. Besides, we add multiple validations, include an automated order system(https://cloudpki.cfca.com.cn), the system is audited prior to RA, check more strictly. This will reduce the probability of accident, be able to detect and deal with problems in time.

Oliver: Thank you for providing an update.

However, there are still a number of unresolved items in this issue that remain concerning.

Comment #10 requested:

an update on when CFCA has decided if and when to implement pre-issuance linting.

It is unclear if the "automated order system" is pre-issuance linting, and if all tbsCertificates are tested before being signed with the CA's key material.

Comment #13 requested:

Also, please explain why CFCA has not provided any updates on this issue for 4 months.

Flags: needinfo?(sunny_bxl)
Whiteboard: [ca-compliance] - Next Update - 05-Febrruary 2020 → [ca-compliance] [delayed-revocation-leaf]
Regressions: 1613409

I filed Bug 1613409 to track the request in Comment #13

(In reply to Ryan Sleevi from comment #17)

It is unclear if the "automated order system" is pre-issuance linting, and if all tbsCertificates are tested before being signed with the CA's key material.

Ryan: I think it is necessary to explain the workflow of the system, that might explain some of your doubt.

Users can submit information in the system, the system will verify the information first, include CSR, domain name, company information and so on, Then feedback the preliminary verify results (success or fail). If it fails, the system will display the cause of the error. If successful, it will review by human auditors again. After human verify, it will be sent to RA (RA will also check the content again) and then issue certificate. The process has been tested.

Thanks

Comment #13 requested:

Also, please explain why CFCA has not provided any updates on this issue for 4 months.

I will explain this in Bug 1613409 later

Flags: needinfo?(sunny_bxl)

Wayne: Comment #16 captures where things stand. Pre-issuance linting was not implemented (per Comment #19), but I don't have follow-up questions.

Flags: needinfo?(wthayer)

It appears that all questions have been answered and remediation is complete.

Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Flags: needinfo?(wthayer)
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] [delayed-revocation-leaf] → [ca-compliance] [leaf-revocation-delay]
You need to log in before you can comment on or make changes to this bug.