1908128 - NAVER Cloud Trust Services: Certificate issued with incorrect OCSP URI in AIA

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Hogeun Yoo]

Preliminary Incident Report

On July 16, 2024, NAVER Cloud Trust Services G1 ECC CA1, an intermediate CA of NAVER Cloud Trust Services ECC Root G1 added to the Microsoft root program on May 28, issued certificates for the test websites.

One of these certificates is being monitored by sslmate OCSP WATCH due to the incorrect OCSP URI in AIA, with the issue “error parsing OCSP response: bad OCSP signature: x509: ECDSA verification failure”, and as soon as we became aware of it, the affected certificate was revoked within 30 minutes.

In the Initial investigation, we found that the certificate profile was created with the OCSP URI for the CA Certs rather than the OCSP URI for the End-Entity Certs.

We are currently suspending certificate issuance pending further investigation and will provide a full incident report no later than July 23, 2024.

Comment 1

[Hogeun Yoo]

Incident Report

Summary

2024-07-16 UTC, NAVER Cloud Trust Services issued certificates with incorrect OCSP URI in Authority Information Access. Within 30 minutes of discovering this issue, the certificates were promptly revoked.
The root cause of this mis-issuance was identified as a human error during the creation of the certificate profile, where the URI value in the AIA extension field was incorrectly entered.

Impact

A certificate was incorrectly issued by the NCTS CA team for a test website and was discovered before being used. NAVER Cloud Trust Services identified the mis-issuance and immediately suspended certificate issuance to prevent any further issues.

Time

All times are UTC.
2024-07-16:

• 07:00 NCTS CA System created a certificate profile for DV certificates using the ECDSA algorithm in the production environment. The OCSP URI in the AIA field was manually entered by a person.
• 07:25 NCTS CA Team issued an ECDSA DV certificate with this profile for the CA's test websites.
• 07:41 Through OCSP WATCH, it was confirmed that one of the issued certificates includes an incorrect OCSP URI in the AIA field.
• 07:43 NCTS CA has revoked the certificate and suspended the issuing system.
• 07:45 Investigation started
• 09:00 In the initial investigation, we found that the certificate profile was created with the OCSP URI for the CA Certificates rather than the OCSP URI for the End-Entity Certificates.
• 13:34 Posted a preliminary report on Bugzilla.

Root Cause Analysis

• When creating or modifying a certificate profile, the AIA extension field is set to be entered manually. In this process, the OCSP URI for the End-Entity Certificates was incorrectly input as the OCSP URI for the CA Certificates.

Lessons Learned

What went well

N/A

What didn't go well

• A dictionary of allowable AIA extension field values should have been implemented within the certificate issuance system for creating and modifying certificate profiles. There is a risk of human error when entering values manually.
• If there had been a verification logic in the certificate issuance process to check whether the OCSP URI was designated for a specific CA purpose, the issue could have been prevented, but such logic was not in place.

Where we got lucky

• The issue occurred before the certificate was supplied to external customers and was identified before being used in the actual web service.

Action Items

Action Item	Kind	Due Date
(Design In-progress) The certificate issuance system is being improved by changing the AIA extension field's caIssuer and OCSP URI values to constant selections, so that they are not entered manually during certificate profile creation and modification.	Prevent	TBD
(Design In-progress) Adding a verification logic to the certificate issuance process to ensure that the OCSP URI and caIssuer URI designated for specific CA purposes are correct. In other words, implementing a cross-verification mechanism to account for cases where the constants set in the first action item might be incorrectly configured.	Prevent	TBD

Appendix

Details of affected certificates

No.	Precertificate	Certificate
1	https://crt.sh/?id=13759476583	https://crt.sh/?id=13759477023

Comment 2

[Clint Wilson]

Can NAVER Cloud Services please provide some additional detail regarding the necessity and appropriateness of leveraging a manual certificate issuance process which allows for non-compliant input?

It's unclear to me if the issues encountered here and in Incident 1908130 are solely the result of misconfigurations in profiles used by an otherwise static certificate issuing system.

Can you describe the process for creating profiles in more detail? It appears that only 25 minutes passed between the profile being created and it being used to issue an end-entity certificate from a production system. What steps were in place at the time of this incident to define, test, review, stage, verify, and/or release such configuration changes? What steps took place prior to the 07:00 entry in your timeline and which ones took place between the 07:00 and 07:25 entries?

Prior to the changes described as Action Items in Incident 1908130, what technical limitations existed between certificate issuance and profile configuration? What I mean by this, is what technical controls were in place such that the certificate issuance system would refuse to issue a certificate matching a given (misconfigured) certificate profile?
Can you provide a more detailed breakdown of these controls 1) prior to and 2) as planned for after these incidents?

Finally, to what extent do the systems relevant to these 2 incidents overlap with the systems used in the operation of the NAVER Global Root Certification Authority (88F438DCF8FFD1FA8F429115FFE5F82AE1E06E0C70C375FAAD717B34A49E7265)? i.e. are these the same systems and processes for managing certificate profile configuration and certificate issuance?

Comment 3

[Hogeun Yoo]

Hello Clint,
Thank you for your question, and here is our response.

Can NAVER Cloud Services please provide some additional detail regarding the necessity and appropriateness of leveraging a manual certificate issuance process which allows for non-compliant input?

These two incidents occurred in the process of generating the certificate profile. Certificate profiles can be created or modified as needed, so they could be done manually, but we would like to effectively remove and control the elements that could have resulted in non-compliant inputs. The details of these technical measures are detailed in the answers below.

It's unclear to me if the issues encountered here and in Incident 1908130 are solely the result of misconfigurations in profiles used by an otherwise static certificate issuing system.

Can you describe the process for creating profiles in more detail? It appears that only 25 minutes passed between the profile being created and it being used to issue an end-entity certificate from a production system. What steps were in place at the time of this incident to define, test, review, stage, verify, and/or release such configuration changes? What steps took place prior to the 07:00 entry in your timeline and which ones took place between the 07:00 and 07:25 entries?

For the definition of the certificate profile, we start with the creation of the 'Certificate Profile Definition Document'. When we defined the ECC certificate, we used and modified the existing RSA certificate profile that had already been defined.
Next, the certificate profile for the new ECC certificate was created in the CA management application. We can register or modify certificate profiles. Each profile has a unique name and code number.
Please refer to the following for the steps that took place prior to the 07:00 entry and those that occurred between 07:00 and 07:25.

All times are UTC.

2024-07-15:

• 04:00 - Update the certificate profile definition document. This document includes the certificate profile for the ECDSA algorithm.

2024-07-16:

• 06:10 - Create a new certificate profile based on the updated certificate profile definition document in a development environment configured similarly to the production environment. This certificate profile is for issuing ECDSA certificates.

• 06:30 - Issue certificates in the development environment using the newly created profile from the certificate profile definition document. The certificate issuance logs were monitored and reviewed, confirming that the issuance process successfully passed the internal validation procedure, including pre-lint.

• 06:40 - Report that the certificates were issued correctly in the development environment and conduct approval reviews for profile creation and certificate issuance in the production environment.

• 07:00 - 07:25 - Create a new profile in the production environment based on the certificate profile definition document. This process involves either entering values manually for each field or selecting predefined values from the system.

Prior to the changes described as Action Items in Incident 1908130, what technical limitations existed between certificate issuance and profile configuration? What I mean by this, is what technical controls were in place such that the certificate issuance system would refuse to issue a certificate matching a given (misconfigured) certificate profile?
Can you provide a more detailed breakdown of these controls 1) prior to and 2) as planned for after these incidents?

Previously, when issuing certificates from certain certificate profiles, technical control was made to reject certificates that were incorrectly configured through pre-lint. Thus, the action items were set to improve the elements that can be input when creating and modifying certificate profiles to align with the publicly trusted TLS rather than within common X.509.

First, these are fields (or extensions) that already controlled to have the compliant single selection or to be created automatically without manual intervention.

Fields	Input values ​​(form)	Note
Authority Key Identifier	AuthorityKeyIdentifier: keyIdentifier	(Required) keyIdentifier is automatically generated from issuer information
Subject Key Identifier	SubjectKeyIdentifier: keyIdentifier	(Required) keyIdentifier is automatically generated from subject information
Basic Constraints	Basic Constraints: Subject Type=End Entity, Path Length Constraint=none	(Required) SSL/TLS certificate is fixed to the same value as the Input values (form)
Subject Alternative Names	Subject Alternative Names: dNSName = DOMAIN	(Required) The 'DOMAIN' value is automatically filled in during the issuance process based on the certificate application information
CRL Distribution Points	Automatically applied with preset End-entity CRL Distribution Points	(Required)

Next are fields (or extensions) that will be improved as action items, either because they have information that is able to manually entered or because non-compliant values can be set.

Fields	As-is	To-be	Note
Key Usage	(Required) Choose from digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment, keyAgreement, encipherOnly, decipherOnly	(RSA public key) digitalSignature is required, keyEncipherment and dataEncipherment are optional, (ECC public key) digitalSignature is required, keyAgreement is optional	Before the change, any of the options could be selected and there was no defined acceptable values ​​for each public key algorithm
Certificate Policies	(Optional) Manually enter the PolicyIdentifier value	(Required) PolicyIdentifier value must be selected from 2.23.140.1.2.1 and 2.23.140.1.2.2	No missing or incorrect entry when improvement is completed
Extended Key Usage	(Optional) Choose from id-kp-serverAuth, id-kp-clientAuth, id-kp-codeSigning, id-kp-emailProtection, id-kp-timeStamping, id-kp-timeStamping	(Required) id-kp-serverAuth is required, id-kp-clientAuth is optional	EKUs that violate BR are not selected when improvements are completed
Authority Information Access	(Optional) Enter the uniformResourceIdentifier of caIssuers directly, enter the uniformResourceIdentifier of ocsp directly	(Required) The uniformResourceIdentifier input value of caIssuers and ocsp are fixed as constants and automatically entered	No missing or incorrect input when improvement is completed
Signature Algorithm, Key Size	(Required) Only ECDSA can be selected, and signature algorithms and key sizes other than BR are allowed	Only signature algorithms and key sizes allowed by BR can be selected	Currently, only RSA can be selected for RSA CA

The above technical control will be applied and the normally issued certificate will be cross-verified by performing external lint once again (also included in the action items). The overall process will be improved to be applied to the production environment after undergoing profile creation or modification, issuance of certificates in test environment, OCSP, CRL test, cross-verification with external lint and pre-review by at least two individuals.

Finally, to what extent do the systems relevant to these 2 incidents overlap with the systems used in the operation of the NAVER Global Root Certification Authority (88F438DCF8FFD1FA8F429115FFE5F82AE1E06E0C70C375FAAD717B34A49E7265)? i.e. are these the same systems and processes for managing certificate profile configuration and certificate issuance?

The system relevant to these 2 incidents are built separately based on the same application as NAVER Global Root Certification Authority. The action items we have derived will also be taken action because these are issues that potentially possible occur in the NAVER Global Root Certification Authority system. However, we would like to make it clear that the certificates issued through the certificate profile used in the NAVER Global Root Certification Authority system are normal, and there are no plans to change or create new certificate profiles during the action period.

Comment 4

[Hogeun Yoo]

This is an update from Comment #1, refreshing the status of action items.

Action Items

Action Item	Kind	Due Date
(Implementation In-progress) The certificate issuance system is being improved by changing the AIA extension field's caIssuer and OCSP URI values to constant selections, so that they are not entered manually during certificate profile creation and modification.	Prevent	2024-08-16
(Implementation In-progress) Adding a verification logic to the certificate issuance process to ensure that the OCSP URI and caIssuer URI designated for specific CA purposes are correct. In other words, implementing a cross-verification mechanism to account for cases where the constants set in the first action item might be incorrectly configured.	Prevent	2024-08-16

Comment 5

[sooyoung.eo]

This is an update from Comment #4, refreshing the status of action items.

Action Items

Action Item	Kind	Due Date
(Completed) The certificate issuance system is being improved by changing the AIA extension field's caIssuer and OCSP URI values to constant selections, so that they are not entered manually during certificate profile creation and modification.	Prevent	2024-08-16
(Completed) Adding a verification logic to the certificate issuance process to ensure that the OCSP URI and caIssuer URI designated for specific CA purposes are correct. In other words, implementing a cross-verification mechanism to account for cases where the constants set in the first action item might be incorrectly configured.	Prevent	2024-08-16

Comment 6

[Hogeun Yoo]

On behalf of Naver Cloud Trust Services, we believe that all analysis and actions have been completed. There is no new information in this bug since Comment 5.

Comment 7

[Ben Wilson]

I will look at closing this next Wednesday, 28-Aug-2024.