1847193 - KAMU SM: commonName not in SAN

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Melis Şimşek]

Steps to reproduce:

Kamu SM discovered that a certificate was issued without a CN field in the SAN extension. The affected certificate was revoked immediately.
https://crt.sh/?id=10058290547

We continue to investigate and a full incident report will be posted in the coming days.

Comment 1

[Melis Şimşek]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

On 2023-08-04, KAMU SM became aware of an issue where a certificate was issued for a domain name owned by the company itself. After the issuance, the routine post-linting process was conducted and the in-house lint application called ARMA reported an error that the value in the Subject CN field of the certificate was missing in the SAN extension. Subsequently, the certificate was promptly revoked at 06:07:39 UTC.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

YYYY/MM/DD - Time in UTC	Description of actions
2023-08-04 06:00	Certificate with commonName not in SAN issued.
2023-08-04 06:03	Before sending the certificate to the customer, the certificate undergoes a linting process. ARMA reported misissuance.
2023-08-04 06:07	Certificate revoked.
2023-08-04 06:15	Notification of the compliance team by the certificate issuance personnel and investigation started by compliance team for finding root cause.
2023-08-04 06:30	Misissuance confirmed. It has been decided to stop further certificate issuance.
2023-08-04 07:45	Root cause found. For details, see Q6.
2023-08-04 09:30	Compliance team requested to DBA investigate SSL certificates which might be misissued with a same error. Subsequent investigation confirmed that there were no additional certificates affected by the issue.
2023-08-04 10:00	Upon discovering that the problem resulted from human error, the certificate issuance was restarted.
2023-08-04 11:02	Preliminary incident report issued.

3.Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

As soon as we became aware of this misissuance, necessary steps have been taken for certificate revocation and stopped to issue subscriber certificates throughout the investigation process.

4. A summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued.

Total number of affected certificates: 1
The only one (1) affected certificate was issued on 2023-08-04 06:00:24 UTC

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Please see: https://crt.sh/?id=10058290547

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The certificate issuance process at KAMU SM can be summarized as follows:

• Subscribers fill out the "SSL Application Form and Subscriber Agreement" and send it to KAMU SM.
• KAMU SM performs organization validation and checks the CSR file sent by the customer using the ARMA program.
• If there are additional SANs to be included in the certificate, they are specified in the application form. SANs in the CSR, if any, are not considered.
• After domain ownership validation, the CSR file to be certified is added to ESYA, our internal certificate management infrastructure. Then, the Subject-CN field specified in the CSR and additional domain names, if any, specified in the application form are manually added to the SAN extension by certificate issuance personnel.
• The generated pre-certificate is sent to log servers, SCT values are obtained and SSL certificate issuance is completed.
• Before sending the SSL certificate to the customer, it undergoes a final check in the ARMA program. If any issues are found, the certificate is immediately revoked.
• The entire process, from receiving the application form to issuing and delivering the certificate to the customer, is monitored through a checklist prepared by the compliance team, which is regularly updated to reflect changes in the certificate management process.

Upon investigating the process of the incorrectly issued certificate, it was discovered that multiple wildcard certificate requests were present in the application form. While handling these simultaneous wildcard requests, the certificate issuance personnel mistakenly entered the SAN field incorrectly, resulting in the misissuance. However, the ARMA application quickly detected the problem after the issuance and the certificate is revoked immediately.

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

KAMU SM plans to update the ESYA application by adding a check to ensure that the CN field in the CSR is included in the SAN extension. Furthermore, a warning will be included in the application interface if the SAN extension is empty. The software development team confirmed that these updates can be incorporated into the sprint for the upcoming week, with the software update expected to be completed in the middle of the September.

Meanwhile, to proactively prevent similar incidents, KAMU SM will apply additional controls in the checklist used for certificate issuance. These measures will help bolster the overall security and reliability of the certificate issuance process until the software update is fully deployed.

Schedule	Description of actions
2023-08-08	New revision of the checklist including the related controls will be published.
From the time the checklist is published until the software update	The certificate issuance process will be completed through the updated checklist.
In the middle of the September	A software update will be deployed in production.

Comment 2

[Daniel McCarney]

Meanwhile, to proactively prevent similar incidents, KAMU SM will apply additional controls in the checklist used for certificate issuance. These measures will help bolster the overall security and reliability of the certificate issuance process until the software update is fully deployed.

Why have you not considered implementing linting pre-issuance as a remediation step?

This is a practice other CAs have adopted and is called out explicitly as a recommended practice by Mozilla: https://wiki.mozilla.org/CA/Required_or_Recommended_Practices#Pre-Issuance_Linting

Comment 3

[Melis Şimşek]

(In reply to Daniel McCarney from comment #2)

Meanwhile, to proactively prevent similar incidents, KAMU SM will apply additional controls in the checklist used for certificate issuance. These measures will help bolster the overall security and reliability of the certificate issuance process until the software update is fully deployed.

Why have you not considered implementing linting pre-issuance as a remediation step?

This is a practice other CAs have adopted and is called out explicitly as a recommended practice by Mozilla: https://wiki.mozilla.org/CA/Required_or_Recommended_Practices#Pre-Issuance_Linting

Thanks for your comment, Daniel.

Our certificate management infrastructure serves as our operations, enabling the issuance of secure and trusted certificates to our clients for all of our products, not limited to SSL certificates. As a result, upgrading our PKI software to enhance pre-issuance linting could potentially impact our other business activities.

It is worth noting that we apply various validations within the scope of pre-issuance checks for SSL certificates. Let us briefly describe the controls implemented in our certificate production process.

As indicated in the incident report, the CSR submitted by the customer undergoes various compliance checks within the ARMA application by RA personnel. ARMA processes the CSR to verify the compliance of Subject fields with the Baseline Requirements, and the parsed CSR is presented to RA personnel. The RA personnel review the results of the ARMA controls and, if any discrepancies are found, inform the customer. The ARMA control mechanism encompasses, among others:

• Proper combinations of OV Subject DN fields
• Inclusion of "TR" in the Country field
• Inclusion of a Turkish Province in the "ST (State or Province)" field
• DNS name encoding rules for CN and, if any, additional SAN fields
• Allowed public key size/algorithm (e.g., weak keys)

Other cases related to pre-issuance linting of the SSL certificate are actually checked through the certificate profiles within our certificate management infrastructure. These profiles are templates created based on the requirements outlined in CA/B Forum BR, RFC 5280 standard, and other industry benchmarks. As KAMU SM, we issue all SSL certificates with this template. The creation or modification of these templates is done exclusively by authorized personnel under the supervision of the compliance team. At this stage, for the SAN extension, which is at the discretion of production personnel, the planned to be added control for matching SAN and CN significantly minimizes the occurrence of problematic scenarios, thus enhancing the reliability and security of the certificates we issue.

Comment 4

[Melis Şimşek]

Update Q7 in Comment#1: We have published the new version of the checklist including the related controls. From now on, we will proceed with our process using the updated checklist.

Comment 5

[Daniel McCarney]

Hi Melis, thanks for your response.

As a result, upgrading our PKI software to enhance pre-issuance linting could potentially impact our other business activities.

It's not very reassuring to hear that you don't have confidence making changes to your CA software that would help improve your ability to consistently meet root program requirements. Are there actions you could take that would increase your confidence doing so? Is the portion of your business that is subject to the Mozilla root program requirements not isolated from other activities?

It is worth noting that we apply various validations within the scope of pre-issuance checks for SSL certificates. Let us briefly describe the controls implemented in our certificate production process.

These validations are helpful, but they are not pre-issuance linting. Applying one of the open source linters at pre-issuance time will offer many additional controls above and beyond those you list here and these controls will not be subject to human error by RA personnel.

Other cases related to pre-issuance linting of the SSL certificate are actually checked through the certificate profiles within our certificate management infrastructure.

Many CAs configure certificate profiles within their management infrastructure that aim to meet root program requirements and additionally rely on linting pre/post issuance. The Mozilla required or recommended practices wiki page says:

Because BR or RFC violations are generally considered by Mozilla to be misissuance, such integration will reduce the number of misissuance events a CA experiences, if earlier parts of their pipeline fail in their job of keeping certificates compliant.

Pre-issuance linting acts as a check against failure in earlier parts of the pipeline, such as a mistake in certificate profile configuration.

Please re-consider adopting an open source linter and applying it pre-issuance as a remediation item. It is a valuable tool and I don't believe you've offered sufficient justification for avoiding it.

Comment 6

[Melis Şimşek]

It's not very reassuring to hear that you don't have confidence making changes to your CA software that would help improve your ability to consistently meet root program requirements. Are there actions you could take that would increase your confidence doing so? Is the portion of your business that is subject to the Mozilla root program requirements not isolated from other activities?

In Kamu SM, all products share a common underlying infrastructure, yet they each possess their own distinct modules that are exclusive to them. Each product includes specialized modules like CT for SSL certificates, aligned with requirements from root programs, CA/B BR, and other industry standards.

These validations are helpful, but they are not pre-issuance linting. Applying one of the open source linters at pre-issuance time will offer many additional controls above and beyond those you list here and these controls will not be subject to human error by RA personnel.

While these controls have a significant impact on preventing misissuance, we are aware that they do not provide the equivalent effectiveness of pre-issuance linting. However, due to the fact that making developments in the certificate management infrastructure within the scope of pre-linting would take more time than working on SAN-CN control improvements, we prioritized SAN-CN development. As a temporary measure, we are enhancing our checklist and refining the issuance process by incorporating CN-SAN control.

Pre-issuance linting acts as a check against failure in earlier parts of the pipeline, such as a mistake in certificate profile configuration.

Please re-consider adopting an open source linter and applying it pre-issuance as a remediation item. It is a valuable tool and I don't believe you've offered sufficient justification for avoiding it.

We are in discussions with our CA software development department regarding ways to accelerate our adoption of pre-linting. We will provide further updates regarding the result of the discussion.

Comment 7

[Daniel McCarney]

We are in discussions with our CA software development department regarding ways to accelerate our adoption of pre-linting. We will provide further updates regarding the result of the discussion.

Thank you

Comment 8

[Melis Şimşek]

(In reply to Melis Şimşek from comment #6)

We are in discussions with our CA software development department regarding ways to accelerate our adoption of pre-linting. We will provide further updates regarding the result of the discussion.

Our CA software development department has begun analysing on adding the pre-issuance linting mechanism to our certificate management infrastructure. As previously mentioned, the ARMA application performs checks such as the Subject field and key/algorithm compatibility in the CSR file and it verifies whether an SSL certificate aligns with both CA/B Forum and other industry standards, as well as Kamu SM's certificate profile, by examining extensions and other certificate fields.

The post-issuance linting checks conducted in ARMA will also be integrated into our certificate management infrastructure as a pre-issuance linting mechanism. This will enable the detection of any issues before generating a pre-certificate, ensuring that errors are identified prior to the issuance of the certificate.

We will provide information on the schedule for this development next week.

Comment 9

[Melis Şimşek]

The development mentioned in Comment #1 has reached completion and is currently undergoing testing. Simultaneously, progress is being made in the development of integrating the pre-issuance linting mechanism into our certificate management infrastructure. Our intention is to implement this development in the software update scheduled for the end of September.

Comment 10

[Ben Wilson]

Dear Melis,
Please provide an update by the end of the week.
Thanks.
Ben

Comment 11

[Melis Şimşek]

Dear Ben,

Pre-issuance linting and the development mentioned in Comment #1 are operational in the production environment as of 27-09-2023. Remediation is completed. We kindly request that this incident be closed, as all planned activities have been completed.

Thanks.

Comment 12

[Ben Wilson]

I will close this tomorrow, 29-Sept-2023, unless there are any objections.
Thanks,
Ben