Open Bug 1532436 Opened 7 months ago Updated 2 months ago

Chunghwa Telecom: Test certificate with unregistered domain name

Categories

(NSS :: CA Certificate Compliance, task)

task
Not set

Tracking

(Not tracked)

ASSIGNED

People

(Reporter: wayne, Assigned: realsky)

Details

(Whiteboard: [ca-compliance] - 14-October 2019)

Li-Chun Chen submitted the following incident report to the mozilla.dev.security.policy mailing list:

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Ans:
One of our staffs in PKI group was taking samples of the issued certificates from crt.sh database for some reasons and found a mis-issued certificate which has a test (unregistered) FQDN on February 15, 2019 at 1:55 (UTC). He then notified us (Public CA) of the incident. We decide to make an initial investigation and found another mis-issued certificate which also has a test FQDN on February 15, 2019 at 2:30 (UTC).

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

Ans:
Timeline (all times are UTC):
15 February 2019 1:55: Find a mis-issued certificate with a FQDN www.raotest.com.tw
15 February 2019 1:59: Revoke the first mis-issued certificate
15 February 2019 2:07: Public CA starts an action plan and initial investigation
15 February 2019 2:17: Further certificate issuing suspended
15 February 2019 2:30: Find another mis-issued certificate with a FQDN publicca.rao.com.tw
15 February 2019 4:10: Initial investigation completed
15 February 2019 4:25: Certificate issuing restarted
15 February 2019 4:40: Hold an investigation meeting

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

Ans:
Yes, we had stopped issuing certificates before we investigated and revoked any certificate with an unregistered FQDN. We have asked our CA system vendor to include an automatic FQDN-checking function in the hotfix which will be released not after 30 March 2019 to avoid making the same mistakes.

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

Ans:
Number of certs: 2
First cert: issued on Nov 12, 2018 at 11:53:02 (UTC)
Last cert: issued on Jan 29, 2019 at 06:43:59 (UTC)

  1. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Ans:
https://crt.sh/?id=1153958924 with FQDN www.raotest.com.tw
https://crt.sh/?id=940459864 with FQDN publicca.rao.com.tw

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Ans:
For the certificate with FQDN www.raotest.com.tw:
One of our RAOs intended to take a screenshot of certificate application process for training material in test environment, but she was not aware that she is in the formal environment. After the certificate was issued, the RAO forgot to revoke it.

For the certificate with FQDN publicca.rao.com.tw:
The same as the former cause, but the mis-issued certificate had been revoked immediately without notifying Public CA.

The mistakes have not been detected (even by our internal self-audit) because the amount of mis-issued certificates is minor. The RAO involved in this incident has been verbally warned and needs to undergo a retraining process in accordance with our employment contract.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Ans:
To avoid making the same mistakes, the following steps will newly be introduced:

Step 1. Implementation of a two-stage manual verification by different RAOs. Effective 26/02/2019.

Step 2. Implementation of an automatic FQDN-checking function prior to issuing certificates. Effective 30/03/2019.

Step 3. Implementation of a scheduling program to periodically and automatically check the newly-issued certificates from our repository. Effective 30/05/2019.

Assignee: wthayer → realsky
Status: NEW → ASSIGNED
Summary: Chunghwa Telecom: → Chunghwa Telecom: Test certificate with unregistered domain name

The automatic FQDN-checking function has been implemented and enforced in Public CA on March 15.
The issuance of any cert with unregistered FQDN can no longer be carried out now.

Whiteboard: [ca-compliance] - Next Update - 01-April 2019 → [ca-compliance] - Next Update - 01-June 2019
  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Ans:
To avoid making the same mistakes, the following steps will newly be introduced:

Step 1. Implementation of a two-stage manual verification by different RAOs. Effective 26/02/2019.

Step 2. Implementation of an automatic FQDN-checking function prior to issuing certificates. Effective 30/03/2019.

Step 3. Implementation of a scheduling program to periodically and automatically check the newly-issued certificates from our repository. Effective 30/05/2019.

The scheduling program that can periodically and automatically check the newly-issued certificates from our repository has been implemented on May 6th.

To avoid the same problem happened in those previously-issued certificates, we scanned the whole repository and found that some FQDNs in the issued certificates are inaccurate due to a change to domain’s ownership, or a relevant licensing or services agreement between the Domain Name Registrar and our customers has terminated. For the former case, we have revoked the certs immediately according to the provisions in Section 4.9.1 of our CP/CPS. For the latter case, we have asked our customers who still want to have the certificate to renew their certificates; otherwise, we will revoke these certs within 5 days according to the provisions in Section 4.9.1 of our CP/CPS.

Thanks for the update, and for taking the appropriate steps to adhere to your CP/CPS with respect to domain ownership changes. Given that the Subscriber Agreement requires Subscribers to notify the CA of such changes, have any steps been taken to examine other certificates from those Subscribers? Have any steps been taken to notify those Subscribers - or all Subscribers - of their obligations under their existing Subscriber Agreements of Terms of Use?

Can you provide further technical details about what the automatic FQDN-checking function does, and what the scheduling program does? Please provide a meaningful overview of the logic and tasks that these systems do.

Flags: needinfo?(realsky)

(In reply to Ryan Sleevi from comment #3)

Thanks for the update, and for taking the appropriate steps to adhere to your CP/CPS with respect to domain ownership changes. Given that the Subscriber Agreement requires Subscribers to notify the CA of such changes, have any steps been taken to examine other certificates from those Subscribers? Have any steps been taken to notify those Subscribers - or all Subscribers - of their obligations under their existing Subscriber Agreements of Terms of Use?

The scheduling program can help us to find the issued TLS/SSL certs with unreliable FQDNs caused by domain ownership changes or a relevant licensing or services agreement between the Domain Name Registrar and the subscribers has terminated, not only focus on a specific subscriber.

As to the obligations of the subscribers which we have claimed in the Subscriber Agreements or
Terms of Use, they must adhere to our CP/CPS and shall own or be authorized to use the domain names listed in the certs during use. If there is a violation, the issuing CA may revoke the certs without any notification; or the issuing CA may give the subscribers a grace period and ask them who still want to have the certs to renew their certs. In this case (please refer to the recent report), we had informed our customers about this matter and the effect of revoking their certs upon we scanned the whole repository.

Can you provide further technical details about what the automatic FQDN-checking function does, and what the scheduling program does?
Please provide a meaningful overview of the logic and tasks that these systems do.

With regard to the automatic FQDN-checking function, it can help us to check whether a FQDN listed in a cert to be issued is registered prior to issuing that cert.

The logic and tasks of this program do are as follows:

Part I. Checking the Name Server or IP Address of the FQDN using Dig instruction
(1) Ask the NS record from a DNS to check is there a record of the (parent) domain? If the answer is no, go to (2); if the answer is yes, then the program continues to Part II.
(2) Ask the A record from a DNS to check is there a record of its IP? If the answer is yes, the program continues to Part II; otherwise, the program stops the issuance of the cert.

Part II. RAOs manually check the FQDN, if everything is ok, then the cert is issued.

With regard to the scheduling program which we have stated above, this program is an extension of the FQDN-function but aiming at all issued and valid (not expired and revoked) TLS/SSL certs, which can be enforced periodically.

Flags: needinfo?(realsky)

Wayne: Is there anything further that you'd like with this bug?

The described algorithm seems that it will require more work of the CA - legitimate changes of IP addresses may require additional review, while changes to WHOIS information may go undetected - but on the balance, over-triggering instead of under-triggering seems preferable?

Flags: needinfo?(wthayer)

Li-Chun: How was an RAO able to issue two certificates without performing domain control validation (which is impossible if the name isn't registered)? Based on your description of the automatic FQDN checking system, I am assuming that the RAO must still manually verify that the Subscriber controls the domain name? If so, I think there is still a very significant opportunity for human error that is only partially addressed by the two-stage manual process. Is there an opportunity to automatically check both the existence of the domain and that domain control has been properly verified?

Flags: needinfo?(wthayer) → needinfo?(realsky)
Whiteboard: [ca-compliance] - Next Update - 01-June 2019 → [ca-compliance]

(In reply to Wayne Thayer [:wayne] from comment #6)

How was an RAO able to issue two certificates without performing domain control validation?

Ans: That’s why we implement the automatic FQDN-checking function. Actually, the program executed once during Part I (checking the Name Server or IP Address of the FQDN using Dig instruction) and it would execute once again after Part II at the time that a RAO performs the action of approving certificate request (prior to issuing certs of the CA ). As we said before, the issuance of any cert with unregistered FQDN can no longer be carried out after the automatic FQDN-checking function is implemented and enforced.

Is there an opportunity to automatically check both the existence of the domain and that domain control has been properly verified?

Ans: Yes, of course we can ask our CA system vendor to include this function that can automatically check the existence of the domain name and validate the domain control rights. However, our company’s security policy does not allow us to do the part regarding the automatically validation of the domain control rights. Because the CA system has to send a request to the domain to be verified and the firewall’s policy must set to any policy accordingly which will increase the chances of our company’s network being attacked. About this part, we are still seeking a better solution and may implement this function within three months.

 Thanks.
Flags: needinfo?(realsky)

Thank you for the reply Li-Chun,

(In reply to Li-Chun CHEN from comment #7)

(In reply to Wayne Thayer [:wayne] from comment #6)

How was an RAO able to issue two certificates without performing domain control validation?

Ans: That’s why we implement the automatic FQDN-checking function.

From your description, I do not think that the automatic FQDN-checking function solves the problem of an RAO being able to approve a registered domain that is not controlled by the requestor.

Actually, the program executed once during Part I (checking the Name Server or IP Address of the FQDN using Dig instruction) and it would execute once again after Part II at the time that a RAO performs the action of approving certificate request (prior to issuing certs of the CA ). As we said before, the issuance of any cert with unregistered FQDN can no longer be carried out after the automatic FQDN-checking function is implemented and enforced.

Is there an opportunity to automatically check both the existence of the domain and that domain control has been properly verified?

Ans: Yes, of course we can ask our CA system vendor to include this function that can automatically check the existence of the domain name and validate the domain control rights. However, our company’s security policy does not allow us to do the part regarding the automatically validation of the domain control rights. Because the CA system has to send a request to the domain to be verified and the firewall’s policy must set to any policy accordingly which will increase the chances of our company’s network being attacked. About this part, we are still seeking a better solution and may implement this function within three months.

Please update this bug when a solution has been identified.

 Thanks.

Setting a Needs-Info. Li-Chun, it would be good to provide weekly updates, per https://wiki.mozilla.org/CA/Responding_To_An_Incident#Keeping_Us_Informed , until concrete timelines are identified and discussed. We want to ensure timely progress on issues and the proposed solutions.

Flags: needinfo?(realsky)

After discussed with our colleagues and CA system vendor, we made the following conclusions:

(1) As specified in our CPS, Public CA allow subscribers to submit their OV applications in writing. Note that the previous two mis-issued certs are OV certs not DV certs. In comparison to DV certs application which only needs to verify the applicant’s domain control, OV certs application sometimes needs additional review work like the information listed on the written application (where non-automated processes are still needed for organization identity authentication) that cannot be automatically performed by a program but can only be done by our RAOs. Therefore, as discussed in the following IETF link, we think that a program to automatically check both the existence of the domain and that domain control has been properly verified is only applicable to DV certs. (https://ietf-wg-acme.github.io/acme/draft-ietf-acme-acme.html)

(2) To solve the problem of an RAO may be able to approve a registered domain that is not controlled by the requestor, we have asked our CA system vendor to include the two validation methods specified in BR 3.2.2.4.6 (Agreed-Upon Change to Website) and 3.2.2.4.7 (DNS Change) for validation of authority in the hotfix within three months. After implemented the new checking function, our CA system will automatically check both the existence of the domain and that domain control prior to issuing OV certs.

Flags: needinfo?(realsky)

(In reply to Li-Chun CHEN from comment #10)

After discussed with our colleagues and CA system vendor, we made the following conclusions:

(1) As specified in our CPS, Public CA allow subscribers to submit their OV applications in writing. Note that the previous two mis-issued certs are OV certs not DV certs. In comparison to DV certs application which only needs to verify the applicant’s domain control, OV certs application sometimes needs additional review work like the information listed on the written application (where non-automated processes are still needed for organization identity authentication) that cannot be automatically performed by a program but can only be done by our RAOs. Therefore, as discussed in the following IETF link, we think that a program to automatically check both the existence of the domain and that domain control has been properly verified is only applicable to DV certs. (https://ietf-wg-acme.github.io/acme/draft-ietf-acme-acme.html)

I do not understand this response. While OV certificates require additional checks that may be difficult to fully automate, they also require the same domain control checks performed for DV certificates.

(2) To solve the problem of an RAO may be able to approve a registered domain that is not controlled by the requestor, we have asked our CA system vendor to include the two validation methods specified in BR 3.2.2.4.6 (Agreed-Upon Change to Website) and 3.2.2.4.7 (DNS Change) for validation of authority in the hotfix within three months. After implemented the new checking function, our CA system will automatically check both the existence of the domain and that domain control prior to issuing OV certs.

This is good to hear. Please update us when this functionality is in place, and if anything happens to delay it. I have set the 'next update' for this bug to 3 months from now.

Whiteboard: [ca-compliance] → [ca-compliance] - 14-October 2019

(In reply to Wayne Thayer [:wayne] from comment #11)

(1) As specified in our CPS, Public CA allow subscribers to submit their OV applications in writing. Note that the previous two mis-issued certs are OV certs not DV certs. In comparison to DV certs application which only needs to verify the applicant’s domain control, OV certs application sometimes needs additional review work like the information listed on the written application (where non-automated processes are still needed for organization identity authentication) that cannot be automatically performed by a program but can only be done by our RAOs. Therefore, as discussed in the following IETF link, we think that a program to automatically check both the existence of the domain and that domain control has been properly verified is only applicable to DV certs. (https://ietf-wg-acme.github.io/acme/draft-ietf-acme-acme.html)

I do not understand this response. While OV certificates require additional checks that may be difficult to fully automate, they also require the same domain control checks performed for DV certificates.

I also don't quite understand this. Note that ACME is specified as RFC 8555. ACME's design permits CAs to add additional steps during the verification process, whether they be requiring explicit approval by the account holder or additional validation of other certificate fields (consistent with the RA/CA processes). This is why handling such as "External Account Binding" or "Responding to Challenges" exist, allowing clients to associate with their OV account and to allow requests to be in "pending" states while the CA performs additional validations.

To be clear: ACME can be used with DV, OV, EV, QWAC, or virtually any certificate.

You need to log in before you can comment on or make changes to this bug.