Closed Bug 1866448 Opened 1 years ago Closed 1 year ago

NAVER Cloud Trust Services: DV Certificate issued with improperly validated

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: hanyong.park, Assigned: hanyong.park)

Details

(Whiteboard: [ca-compliance] [dv-misissuance])

NAVER Cloud Trust Services inform that a DV certificate was improperly validated and issued on November 24 during test for a newly released certificate issuance service. We revoked it immediately when became aware of the improper. The certificate is as follows:
https://crt.sh/?id=11202882602

We have started investigations and a full report will be posted in the coming days.

Incident Report

Summary

This is an update to the preliminary report(#c0).

NAVER Cloud Trust Services inform that a DV certificate was improperly validated and issued on November 24 during test for a newly released certificate issuance service, NCP Certificate Manager(hereinafter NCP CM). We revoked it immediately when became aware of the improper. The certificate is as follows:
https://crt.sh/?sha256=79C7A9B2D8F2A9F43E1A0083C229B34C4FCB70EB7244C91EDECD992DC2A0DC19

The team performing QA testing had access to the outbound mailer(Internal Email Sending System), and during the functional test, one of the team members accessed the approval link in the body of the email sent through a sending history inquiry and clicked the approval button. As a result, the NCP CM deemed that domain validation was successful and the certificate was issued.

Impact

This incident can be classified as a human error that internally bypassed the normal process in domain validation using email, and there was a lack of technical/logical prevention. We identified the root cause, removed unnecessary access from the outbound mailer and the same/similar events would no longer occur as the test ended, so did not cease issuance.

The type of affected certificate was Domain Validation Certificate, and the number of it is one(1). It was immediately revoked upon recognition after issuance, and by the internal investigation, there are no other cases of identical mis-issuance or any event of misuse.

Timeline

All times are UTC.

2023-11-23:

  • 13:00 NCP CM newly released and start to issue DV certificate
  • 17:10 The initial QA testing finished

2024-11-24:

  • 05:00 Detailed functional validation testing for NCP CM was start
  • 05:26 One of the testing member requested a DV certificate with Email validation method, and Click on the approval page link of the email sent in the outbound mailer sending history, access the page, and proceeded the approval.
  • 05:31 The affected certificate was issued
  • 06:00 The detailed functional validation testing was finished
  • 06:30 The CA management team initially suspected that the affected certificate had been improperly validated
  • 06:35 Recognized that there was Email validation using sending history of outbound mailer during QA testing
  • 06:40 Affected certificate was identified and immediately revoked
  • 07:30 Investigation started
  • 08:00 Manual access to the contents of sent emails through the outbound mailer was identified as the root cause, and decided to investigate by November 27th whether any other bypass point were exist except for this finding
  • 08:30 Removed access rights for personnel except validation specialists and system administrators from the outbound mailer
  • 10:14 Posted a preliminary report on Bugzilla

2024-11-27:

  • 01:00 No other causes had been found except for the previously specified root cause
  • 02:20 Establish measures to prevent recurrence (Refer to Action Items)

Root Cause Analysis

  • Having unnecessary or inappropriate access rights - The members of QA testing team were able to see the sent email through Outbound Mailer System, and the domain validation approval page that linked on the email body content also accessible
  • The domain validation approval page accessible through Outbound Mailer - The domain validation approval page, linked in the email body, must be accessible from transmitted mail, but it also can be accessible from the Outbound Mailer internally

Lessons Learned

What went well

  • When we recognized the mis-issuance, the affected certificate was revoked without delay.
  • After revoking the affected certificate, we immediately started investigating the root cause, the same/similarly affected certificates.

What didn't go well

  • There was a lack of sharing of related departments regarding restrictions and precautions when testing functional validation.
  • Email validation uses the internal outbound mailer system, but access rights and technical prevent actions have not been thoroughly reviewed in advance before released to the production environment.

Where we got lucky

  • One day after the launch of the NCP CM, the defect that could bypass the normal process were found, and root causes could be eliminated and recurrence prevention measures could be prepared without damaging Internet user and customer. To avoid rely on luck again such as early detection, we will taking technical and logical action to block access from internal operation tools, including outbound mailers, to the the domain validation approval page sent as a link in the email body.

Action Items

Action Item Kind Due Date
(Completed) Removed unnecessary or inappropriate access rights from the outbound mailer Mitigate and Prevent 2023-11-24
(In-progress) The domain validation approval page sent as a link in the email body is blocked from being accessed from internal operating tools, including the outbound mailer Prevent 2023-12-01

Appendix

Details of affected certificates

The affected certificate:
https://crt.sh/?sha256=79C7A9B2D8F2A9F43E1A0083C229B34C4FCB70EB7244C91EDECD992DC2A0DC19

Assignee: nobody → hanyong.park
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] [dv-misissuance]

Can you please explain and lay out the exact issuance flow? Can you also please match each step of the issuance flow to what parts of the BRs support it?

(In reply to amir from comment #2)

Can you please explain and lay out the exact issuance flow? Can you also please match each step of the issuance flow to what parts of the BRs support it?

The flow and each procedure from application to certificate issuance are as follows. If a procedure matches the BRs, we were marked.

Applicants apply for a certificate to the NCP CM.

Step Procedure Matched BRs
Certificate Application Input domains
Certificate Application Select Donamin Validation Method (Email or DNS)
Certificate Application Reqeust complete and submit to the RA for NCP CM

NCP CM submits the application information to the internal RA designated for NCP CM.

Step Procedure Matched BRs
Validation of Application Information and Domain CSR Verification 4.2.2 Approval or rejection of certificate applications, 6.1.5 Key sizes
Validation of Application Information and Domain Check Public Suffix 3.2.2.6 Wildcard Domain Validation
Validation of Application Information and Domain Check restricted domain (eg. "onion") 3.2.2.4 Validation of Domain Authorization or Control
Validation of Application Information and Domain Check Google Safe Browsing domain 4.2.1 Performing identification and authentication functions
Validation of Application Information and Domain Check CAA Record 3.2.2.8 CAA Records
Validation of Application Information and Domain Branch to the domain validation method selected by the applicant
Validation of Domain - Case1. Email to Domain Contact WHOIS record lookup
Validation of Domain - Case1. Email to Domain Contact Email sending to 5 reserved and 3 WHOIS lookup addresses 3.2.2.4.2 Email, Fax, SMS, or Postal Mail to Domain Contact, 3.2.2.4.4 Constructed Email to Domain Contact
Validation of Domain - Case1. Email to Domain Contact Email validation success or timeout
Validation of Domain - Case2. DNS Change Generate DNS random values and request input to CNAME record 3.2.2.4.7 DNS Change
Validation of Domain - Case2. DNS Change DNS validation success or timeout

When the validation of application information and domain is completed, RA requests certificate issuance to CA.

Step Procedure Matched BRs
Certificate Issuance Precertificate issue and pre-linting 7.1.2.9 Precertificate Profile
Certificate Issuance SCT 7.1.2.11.3 Signed Certificate Timestamp List
Certificate Issuance Certificate issue 7.1.2.7 Subscriber (Server) Certificate Profile, 7.1.2.7.2 Domain Validated, 7.1.2.7.6 Subscriber Certificate Extensions

Thank you for the answer.

In the initial incident reported you stated:

"The team performing QA testing had access to the outbound mailer(Internal Email Sending System), and during the functional test, one of the team members accessed the approval link in the body of the email sent through a sending history inquiry and clicked the approval button. As a result, the NCP CM deemed that domain validation was successful and the certificate was issued."

Am I wrong in reading that, this QA member was able to click a button and prove domain validation? Does that imply that the approval button requires no authentication/authorization once you click on it?

If so, I do not think the action items are adequate. This seems like a pretty big gap in security, as it places an unreasonable amount of trust in mail providers.

Considering many enterprise mail providers do deep scanning of links in emails, it doesn't seem far fetched that one of these systems may accidentally authorize certificate issuance?

Outbound mailer is a system that requires approval from administrators to access. It is accessible only within the internal network and requires authentication per user to access.

In principle, QA team is not allowed to have access to production environment. However, with the launch of NCP CM service, QA team needed access to the production environment for monitoring purposes and they obtained some access through appropriate approval procedures. Unintentionally, excessive rights were given to QA team, which caused this issue. As soon as we were aware of this issue, we informed the team and retrieved QA team’s access rights to the outbound mailer system.

Currently, only the outbound mailer administrators can access the system, and we regularly review their access rights. In addition, to fundamentally prevent such human error, we are in the process of a patch that restricts access to the approval link of the sent email records from our internal network.

I am not talking about the mailing system at all here. For example, if someone forwarded me the email with the link/button to press to issue a certificate for example.com, and I pressed that button, would that authorize issuance?

If so, the real problem here is that the button to acknowledge certificate issuance through email is unprotected.

I think restricting access to the outbound emailer is necessary. However, in my opinion, that is not enough to mitigate this issue.

I would like to clarify that the "approval" I am referring to is the applicant's or domain owner's approval of ownership/control of the domain. The steps by which the CA issues a certificate follows comment #3, and email approval corresponds to "Validation of Domain - Case1. Email to Domain Contact".

The approval method via email contact consists of two specific steps:

  1. An email body containing certificate application information and a link for domain ownership/control approval
  2. Click the link in the email body to access the approval page, click the approve button with enter a pre-generated random value

We believe that this complies with BR and is a procedure commonly used in the industry. However, to prevent recurrence, we will consider improvement such as requiring additional authentication value known only to NCP CM applicants.

Click the link in the email body to access the approval page, click the approve button with enter a pre-generated random value.

Does this mean that just clicking the link isn't sufficient for authorization, and that there is some random value that has been shared with the applicant/subscriber out of band?

Application information and a random value are sent to 5 reserved addresses and 3 WHOIS lookup addresses. Just clicking the link is not sufficient for authorization; approval is completed by clicking the approve button with the random value entered.

It’s concerning that a staff member went through the entire process of both clicking the link and then actively entering the code that was in that email. It’s also concerning that there was no technical limitation to block this. I would also like to remind NAVER that mis-issuance of test certificates have been discussed here in the past with one of the most famous instances of it being: https://bugzilla.mozilla.org/show_bug.cgi?id=1214321

I do think that given this information, the action items provided are not sufficient here. This random code should be sent out of band of the approval link, preferably through a medium that’s not email and has effective access control. This is similar to the suggestion you mentioned on https://bugzilla.mozilla.org/show_bug.cgi?id=1866448#c7.

This also brings up another concern: How can you assure us that any certificate issued so far with this method has actually been authorized by the subscriber? Do you have audit logs on both the authorization link and the outbound emails?

Before the launch of NCP CM, did you issue any certificates? Have you ensured that those certificates were appropriately validated?

@bwilson, given the information here do you also believe that:

  1. The action item in https://bugzilla.mozilla.org/show_bug.cgi?id=1866448#c1 does not go far enough to prevent mis-issuances in the future?
  2. There is not enough information in the incident report on how the analysis on the rest of the certificates happened.

I do think that given this information, the action items provided are not sufficient here. This random code should be sent out of band of the approval link, preferably through a medium that’s not email and has effective access control. This is similar to the suggestion you mentioned on https://bugzilla.mozilla.org/show_bug.cgi?id=1866448#c7.

In order to ensure that it is authenticated only with a random value sent out of band by email, we will modify the email approval procedure so that a random value sent out of band by email can only be approved by accessing the applicant's account in the NCP CM environment and entering it. This will be added to action item 3.
NCP CM's email validation method has been suspended from 2023-12-14 10:00 UTC until action item 3 is completed.

This also brings up another concern: How can you assure us that any certificate issued so far with this method has actually been authorized by the subscriber? Do you have audit logs on both the authorization link and the outbound emails?

We record audit logs for authentication links and outbound mail (success or failure of email sending, and whether emails were received). It was confirmed that the affected certificate was recorded in the audit log as [Email not received] and [Authentication link approved]. Through the investigation, we confirmed that a total of 20 certificates were issued using email validation after the launch of NCP CM before the issuance of the affected certificate, and through the audit log, we confirmed that no certificate was issued in the same case as the affected certificate.

Before the launch of NCP CM, did you issue any certificates? Have you ensured that those certificates were appropriately validated?

NAVER Domain Certificate Manager (NDCM), a system for issuing certificate of domains owned by NAVER, was operated the only issuing system and was used until the launch of NCP CM. NDCM uses an internal outbound mail system that is different from NCP CM, and we can only view mail sending results, statistics, and monitoring for the mail system and do not have permission to view the sent mail body, so similar to this issue could not and did not happen. NDCM also recorded and stored audit logs for authentication links and outbound emails (success or failure of email sending and whether emails were received), and reviewed the audit logs periodically.

We will wait and see if any additional opinions on this bug, and I will post an updated report soon. the action item 1 has been completed, and action item 2 has been taken today(2023-12-14 9:47 UTC). The action item 3 is added as mentioned above.

Action Item Kind Due Date
(Completed) Removed unnecessary or inappropriate access rights from the outbound mailer Mitigate and Prevent 2023-11-24
(Completed) The domain validation approval page sent as a link in the email body is blocked from being accessed from internal operating tools, including the outbound mailer Prevent 2023-12-01 -> 2023-12-14
Modify the email approval procedure so that a random value sent out of band by email can only be approved by accessing the applicant's account in the NCP CM environment and entering it Prevent 2023-12-28

Modify the email approval procedure so that a random value sent out of band by email can only be approved by accessing the applicant's account in the NCP CM environment and entering it

Thank you for adding this action item!

NDCM also recorded and stored audit logs for authentication links and outbound emails (success or failure of email sending and whether emails were received), and reviewed the audit logs periodically.

Were these logs reviewed as part of responding to this incident? If not, when were the last time they were investigated?

Thank you!

Were these logs reviewed as part of responding to this incident? If not, when were the last time they were investigated?

Yes. As a part of responding to this incident, to 2024-11-24 approximately 09:32, the full inspection of all certificates issued via email validation (total of 21 including the affected 1 certificate) since the launch of NCP CM has been completed. I will include this in the timeline in the to be updated report.

This is updated report from #c1

Incident Report

Summary

NAVER Cloud Trust Services inform that a DV certificate was improperly validated and issued on November 24 during test for a newly released certificate issuance service, NCP Certificate Manager(hereinafter NCP CM). We revoked it immediately when became aware of the improper. The certificate is as follows:
https://crt.sh/?sha256=79C7A9B2D8F2A9F43E1A0083C229B34C4FCB70EB7244C91EDECD992DC2A0DC19

The team performing QA testing had access to the outbound mailer(Internal Email Sending System), and during the functional test, one of the team members accessed the approval link in the body of the email sent through a sending history inquiry and clicked the approval button. As a result, the NCP CM deemed that domain validation was successful and the certificate was issued.

Impact

This incident can be classified as a human error that internally bypassed the normal process in domain validation using email, and there was a lack of technical/logical prevention. We identified the root cause, removed unnecessary access from the outbound mailer and the same/similar events would no longer occur as the test ended, so did not cease issuance.

The type of affected certificate was Domain Validation Certificate, and the number of it is one(1). It was immediately revoked upon recognition after issuance, and by the internal investigation, there are no other cases of identical mis-issuance or any event of misuse.

Timeline

All times are UTC.

2023-11-23:

  • 13:00 NCP CM newly released and start to issue DV certificate
  • 17:10 The initial QA testing finished

2024-11-24:

  • 05:00 Detailed functional validation testing for NCP CM was start
  • 05:26 One of the testing member requested a DV certificate with Email validation method, and Click on the approval page link of the email sent in the outbound mailer sending history, access the page, and proceeded the approval.
  • 05:31 The affected certificate was issued
  • 06:00 The detailed functional validation testing was finished
  • 06:30 The CA management team initially suspected that the affected certificate had been improperly validated
  • 06:35 Recognized that there was Email validation using sending history of outbound mailer during QA testing
  • 06:40 Affected certificate was identified and immediately revoked
  • 07:30 Investigation started
  • 08:00 Manual access to the contents of sent emails through the outbound mailer was identified as the root cause, and decided to investigate by November 27th whether any other bypass point were exist except for this finding
  • 08:30 Removed access rights for personnel except validation specialists and system administrators from the outbound mailer
  • 09:32 The full inspection of all certificates issued via email validation (total of 21 including the affected 1 certificate) since the launch of NCP CM has been completed, and have not been affected except for 1 certificate that has already been revoked
  • 10:14 Posted a preliminary report on Bugzilla

2024-11-27:

  • 01:00 No other causes had been found except for the previously specified root cause
  • 02:20 Establish measures to prevent recurrence (Refer to Action Items)

Root Cause Analysis

  • Having unnecessary or inappropriate access rights - The members of QA testing team were able to see the sent email through Outbound Mailer System, and the domain validation approval page that linked on the email body content also accessible
  • The domain validation approval page accessible through Outbound Mailer - The domain validation approval page, linked in the email body, must be accessible from transmitted mail, but it also can be accessible from the Outbound Mailer internally

Lessons Learned

What went well

  • When we recognized the mis-issuance, the affected certificate was revoked without delay.
  • After revoking the affected certificate, we immediately started investigating the root cause, the same/similarly affected certificates.

What didn't go well

  • There was a lack of sharing of related departments regarding restrictions and precautions when testing functional validation.
  • Email validation uses the internal outbound mailer system, but access rights and technical prevent actions have not been thoroughly reviewed in advance before released to the production environment.

Where we got lucky

  • One day after the launch of the NCP CM, the defect that could bypass the normal process were found, and root causes could be eliminated and recurrence prevention measures could be prepared without damaging Internet user and customer. To avoid rely on luck again such as early detection, we will taking technical and logical action to block access from internal operation tools, including outbound mailers, to the the domain validation approval page sent as a link in the email body.

Action Items

Action Item Kind Due Date
(Completed) Removed unnecessary or inappropriate access rights from the outbound mailer Mitigate and Prevent 2023-11-24
(Completed) The domain validation approval page sent as a link in the email body is blocked from being accessed from internal operating tools, including the outbound mailer Prevent 2023-12-01 -> 2023-12-14
(In-progress) Modify the email approval procedure so that a random value sent out of band by email can only be approved by accessing the applicant's account in the NCP CM environment and entering it Prevent 2023-12-28

Appendix

Details of affected certificates

The affected certificate:
https://crt.sh/?sha256=79C7A9B2D8F2A9F43E1A0083C229B34C4FCB70EB7244C91EDECD992DC2A0DC19

Thank you! I do not have any further questions with the additional action item.

This is updated from #c14. Action Item 3 was completed on December 21, 2023 at 05:22 UTC.

Action Items

Action Item Kind Due Date
(Completed) Removed unnecessary or inappropriate access rights from the outbound mailer Mitigate and Prevent 2023-11-24
(Completed) The domain validation approval page sent as a link in the email body is blocked from being accessed from internal operating tools, including the outbound mailer Prevent 2023-12-01 -> 2023-12-14
(Completed) Modify the email approval procedure so that a random value sent out-of-band by email can only be approved by accessing the applicant's account in the NCP CM environment and entering it Prevent 2023-12-28 -> 2023-12-21

On behalf of Naver Cloud Trust Services, we believe that all analysis and actions have been completed. There is no new information in this bug since #c16.

I'll close this on Wednesday, 14-Feb-2024, unless there are any questions.

Flags: needinfo?(bwilson)

To what extent should this be categorized as "testing in a production environment"? The certificate was issued to test.com.

I don't believetest.com is a valid test domain anyway. If this was for example.com then sure, this might've been less of an issue.

We tested for QA purposes after the production environment was launched. If it was as expected, the domain could not have been validated, but an unintended bypass validation occurred, resulting in the mis-issuance. Action items have been taken to prevent such issue from occurring fundamentally, and if QA testing is needed, we take the same procedure as the subscriber only on domains owned by NAVER.

I appreciate the clarification.

(In reply to amir from comment #20)

I don't believetest.com is a valid test domain anyway. If this was for example.com then sure, this might've been less of an issue.

The BRs don't make any distinction for "test domains", and example.com isn't a test domain in any case (it's meant for use in documentation, but nevertheless is a functional domain name with a website operated by ICANN and a valid SSL certificate). Misissuing for example.com is just as serious as misissuing for other domains: https://groups.google.com/g/mozilla.dev.security.policy/c/fyJ3EK2YOP8/m/yvjS5leYCAAJ

(In reply to Andrew Ayer from comment #23)

(In reply to amir from comment #20)

I don't believetest.com is a valid test domain anyway. If this was for example.com then sure, this might've been less of an issue.

The BRs don't make any distinction for "test domains", and example.com isn't a test domain in any case (it's meant for use in documentation, but nevertheless is a functional domain name with a website operated by ICANN and a valid SSL certificate). Misissuing for example.com is just as serious as misissuing for other domains: https://groups.google.com/g/mozilla.dev.security.policy/c/fyJ3EK2YOP8/m/yvjS5leYCAAJ

Agreed.

Agreed, too. I'm going to close this, but for clarification, CAs should either maintain a completely separate testing environment (not signing certificates from publicly trusted CAs), or if QA is done in a production environment, which is discouraged, domain-vetting procedures of BR section 3.2.2.4 should be followed completely.

Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.