1785865 - NAVER Cloud: DV certificate issued with no subject alternative name extension

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Han Yong, Park]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.87 Whale/3.16.138.22 Safari/537.36

Steps to reproduce:

On August 18, NAVER Cloud informs that NAVER Secure Certification Authority 1 issued 21 DV certificates without subject alternative name extension for our test web site (Valid, Revoked, Expired) domains and revoked them within about 3 hours:

https://search.censys.io/certificates?q=parent_spki_subject_fingerprint%3A+4ef411fcfda41412941c629b42ab0951d9329b88baaf3088287f93aecf1a5689+and+parsed.validity.start%3A+%5B2022-08-18+TO+2022-08-18%5D+

A full report will be posted in the coming days.

Comment 1

[Matthias]

I'm interested in why you issued 21 certificates for what looks like 3 test domains - 2 for test1- and test2-domain.naver.com each, and the remaining 17 to test-domain.naver.com.

Comment 2

[Han Yong, Park]

Incident Report

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP mailing list, a Bugzilla bug, or internal self-audit), and the time and date.

On August 18th, NAVER Cloud deployed a new internal certificate issuance system (internally, it is called "NAVER Domain Certificate Manager", hereinafter referred to as NDCM), and issued certificates for three test domains from UTC 2022-8-18 11:07 to UTC 2022-8-18 13:50.
At UTC 2022-8-18 14:26, It was became aware of the Subject Alternative Name (hereinafter referred to as SAN) extension field was missing. All 21 certificates were revoked until UTC 2022-8-18 14:29, and we have stopped issuing certificates through NDCM system.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

YYYY/MM/DD - Time in UTC	Description of actions
2022-08-18 11:07	Initial issuance of certificates without SAN extension fields
2022-08-18 13:50	Last issuance of certificates without SAN extension fields
2022-08-18 14:26	Detection of the fact that 21 certificates without SAN extension field were issued
2022-08-18 14:29	Revocation of all affected certificates
2022-08-18 14:35	Stop the certificate issuance process through NDCM
2022-08-18 14:51	(Root cause found) A bug was found that when requesting a certificate from NDCM to the CA engine, the value to be written in the SAN extension field DNS was sent as NULL value, and issued without an error. The bug caused the issuing CA to issue certificates with missing SAN extension field.
2022-08-18 17:25	Issuance of a preliminary incident report comment#0
2022-08-21 07:03	(CA engine hotfix release) Deploy error handling process if SAN extension field value is missing when issuing a subscriber certificate
2022-08-22 10:18	(NDCM hotfix release) Modify DNS values to be written normally without missing them in the SAN extension field

3.Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

Upon become aware of the misissuance, NAVER Cloud revoked the affected certificates and stopped using NDCM to issue a subscriber certificate.
The CA system is in operation and subscriber certificates can be issued normally through the existing issuance system where this bug has not been occurred.

4 A summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued.

Total number of affected certificates: 21

• The initial certificate was issued on 2022-08-18 11:07
• The last certificate was issued on 2022-08-18 13:50

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Please see:
https://crt.sh/?id=7360678082
https://crt.sh/?id=7360714536
https://crt.sh/?id=7360714664
https://crt.sh/?id=7360772128
https://crt.sh/?id=7360797785
https://crt.sh/?id=7360797800
https://crt.sh/?id=7360797930
https://crt.sh/?id=7360874498
https://crt.sh/?id=7360878767
https://crt.sh/?id=7360905499
https://crt.sh/?id=7360905500
https://crt.sh/?id=7360980842
https://crt.sh/?id=7361008532
https://crt.sh/?id=7361034441
https://crt.sh/?id=7361040059
https://crt.sh/?id=7361040088
https://crt.sh/?id=7361250402
https://crt.sh/?id=7361250431
https://crt.sh/?id=7361474219
https://crt.sh/?id=7361497180
https://crt.sh/?id=7361497266

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Certificate issuance without SAN extension fields first occurred in NDCM. The existing certificate issuance system had its own validation procedures, so the same bugs did not occur. In the NDCM operational environment, the SAN extension field value was sent as a NULL value to the CA system, and the certificates were revoked immediately when we recognized the SAN extension field was missing.
The bug was fixed through the August 21st, August 22nd hotfix release.

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

The error handling process when missing SAN extension field values was addressed by release hotfixes on August 21st and 22nd.
To prevent recurrence, we plan to add the Precertificate Linting procedure to NDCM and will resume opreation after testing whether it is operating properly.

Comment 3

[Ben Wilson]

How is your progress on adding a Precertificate Linting procedure to NDCM and testing?

Comment 4

[Han Yong, Park]

We have planned to add the Precertificate Linting procedure to NDCM, and It has been completed in the test environment on September 22. It will be released to the real environment NDCM in next week. After this integration, automated linting procedure will prevent a certificate mis-issuance. I will share the result when the release in next week is done.

Comment 5

[Han Yong, Park]

The release and testing of automated Precertificate Linting procedure to NDCM has been finished on October 4. This is based on X.509 lint, and it will be prevent to issue a subscriber certificate with warnings or errors, including this bug case. NDCM will issue DV certificates and OV certificates, and the testing of Linting Procedure was finished successfully for both type of certificates.

Comment 6

[Han Yong, Park]

Recently in the internal audit conducted after comment #5, 1 additional same case was found regarding the issuance of certificates for the past 1 year.
Please see: https://crt.sh/?id=5797717831
It is identified a same bug occured from the same CA system used by NDCM. The certificate was issued to one of our test web sites domain and it was revoked immediately after issuance. This bug was caused by the same root cause as previously reported, measures had been completed to prevent a reoccurrence from CA system patch on October 4.
Please let us know if you need to an additional information or any other things regarding to this bug report.

Comment 7

[Ben Wilson]

There haven't been any questions or comments for this incident, and I am inclined to close it on or about Wed. 12-Apr-2023, unless there are reasons why it should remain open.