1938167 - NETLOCK: CRL not published in DER Encoded Format

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Nikolett]

Preliminary Incident Report

Summary

2024-12-17 20:12 UTC - A notification was received by email reporting that the the following CRLs are improperly encoded and in violation of RFC 5280 Section 4.2.1.13 which states: “When the HTTP or FTP URI scheme is used, the URI MUST point to a single DER encoded CRL as specified in [RFC2585].”

To fix the bug, the default value of the current 'crlformat' has been changed from 'PEM' to 'DER'. By default, this will return the URL as a binary (DER) format revocation list. The Base64 format is still available if the crlformat value is set to 'PEM'.
e.g. http://crl.ecc.netlock.hu/index.cgi?crl=tlseveccca&crlformat=pem
If no 'crlformat' variable is specified, the default format is binary.

Impact

We have not yet identified any associated impact.
Our CAs in the EUTL list were registered in PEM format, but our Subscribers were not impacted.
NETLOCK has not stopped the issuance of any certificate because these certificates are not malformed.
We began and finished the updating process of the 4 certificates referenced in the Appendix and are currently running further checks.

Timeline

All times are UTC.
2024-12-17 20:12: A notification was received by email reporting the then potential issue.
2024-12-18 08:03: Our Team confirms that the 4 certificates in question were PEM encoded and not DER encoded.
2024-12-18 09:12: NETLOCK continues to investigate for other wrongly encoded certificates. The drafting of the parameter modification plan began.
2024-12-18 13:50: NETLOCK changed the format of the current CA certificates and CRLs to "DER" referencing the finished modification plan
2024-12-18 17:04: We have answered the e-mail notification and we began to raise this ticket. We are concluding further investigation.

Root Cause Analysis

NETLOCK is going to update this report by, 2025-01-01 with a completed Root Cause Analysis.

Lessons Learned

NETLOCK is going to update this report by, 2025-01-01 with a completed Root Cause Analysis.

What went well

• We were able to react quickly, assessing the reported issue and raising this ticket.

What didn't go well

• Sadly, NETLOCK did not have a process to verify that the files are DER encoded we had a process for PEM validation.

Where we got lucky

• While RFC 5280 requires the files to be DER encoded, there was no impact for our Subscribers.

Action Items

Action Item	Kind	Due Date
Replace the 4 wrongly encoded PEM files with the new DER encoded files at same AIA location	Mitigate	2024-12-18
Planing to update the AIA publishing process to check for DER encoding and finishing the integrating process for pkimetal linter	Mitigate	To be determined

Appendix

Details of affected certificates

The 4 CA Certificates that were posted wrongly with a PEM encoded Certificate:
https://crt.sh/?id=14599347345
https://crt.sh/?id=13708984315
https://crt.sh/?id=13618597454
https://crt.sh/?id=13572525934

Impacted Requirements
• CA/B Forum Baseline Requirements:Section 7.1.2 Certificate Content and Extensions
“all certificates that it issues MUST comply with one of the following certificate profiles, which incorporate, and are derived from RFC 5280. Except as explicitly noted, all normative requirements imposed by RFC 5280 shall apply, in addition to the normative requirements imposed by this document.”
• RFC 5280: Section 4.2.2.1. Authority Information Access
“Where the information is available via HTTP or FTP, accessLocation MUST be a uniformResourceIdentifier and the URI MUST point to either a single DER encoded certificate as specified in [RFC2585] or a collection of certificates in a BER or DER encoded "certs-only" CMS message as specified in [RFC2797].”

---

Based on Incident Reporting Template v. 2.0 effective from October 17, 2023

Comment 1

[Nikolett]

Incident Report

Summary

2024-12-17 20:12 UTC - A notification was received by email reporting that the the following CRLs are improperly encoded and in violation of RFC 5280 Section 4.2.1.13 which states: “When the HTTP or FTP URI scheme is used, the URI MUST point to a single DER encoded CRL as specified in [RFC2585].”

To fix the bug, the default value of the current 'crlformat' has been changed from 'PEM' to 'DER'. By default, this will return the URL as a binary (DER) format revocation list. The Base64 format is still available if the crlformat value is set to 'PEM'.
e.g. http://crl.ecc.netlock.hu/index.cgi?crl=tlseveccca&crlformat=pem
If no 'crlformat' variable is specified, the default format is binary.

Impact

We have not identified any associated impact.
Our CAs in the EUTL list were registered in PEM format, but our Subscribers were not impacted.
NETLOCK has not stopped the issuance of any certificate because these certificates are not malformed.
We began and finished the updating process of the 4 certificates referenced in the Appendix and are currently running further checks.

Timeline

All times are UTC.
2024-12-17 20:12: A notification was received by email reporting the then potential issue.
2024-12-18 08:03: Our Team confirms that the 4 certificates in question were PEM encoded and not DER encoded.
2024-12-18 09:12: NETLOCK continues to investigate for other wrongly encoded certificates. The drafting of the parameter modification plan began.
2024-12-18 13:50: NETLOCK changed the format of the current CA certificates and CRLs to "DER" referencing the finished modification plan
2024-12-18 17:04: We have answered the e-mail notification and we began to raise this ticket. We are concluding further investigation. (Completed)

Root Cause Analysis

Recently, we have introduced several innovations in development and processes. As for the issue, the root cause lies in NETLOCK overlooking the specific file encoding requirement specified in the RFC and lacking a validation step to confirm that the files are DER encoded. This was a minor lapse in our supervison, which we will address by updating our publishing process (We used to monitor the process in an other way but now we want to implement an automatic process with the pkimetal linter) to ensure the compliance of this process in the future.

Lessons Learned

The takeaway for us from this issue is the importance of automating our tasks, regardless of how often it is performed in any way. Alongside the audits and self-assessments required of any publicly trusted CA, we are truly committed to improve all aspects of our service. This involves performing more detailed evaluations of our processes to reduce the risk of similar oversights occurring in the future.

What went well

• We were able to react quickly, assessing the reported issue and raising this ticket. We were also able to resolve the problem quickly.

What didn't go well

• Sadly, NETLOCK did not have a process to verify that the files are DER encoded we had a process for PEM validation.

Where we got lucky

• While RFC 5280 requires the files to be DER encoded, there was no impact for our Subscribers.

Action Items

Action Item	Kind	Due Date
Replace the 4 wrongly encoded PEM files with the new DER encoded files at same AIA location	Mitigate	2024-12-18 - Completed
Planing to update the AIA publishing process to check for DER encoding and finishing the integrating process for pkimetal linter	Mitigate	2025-01-31

Appendix

Details of affected certificates

The 4 CA Certificates that were posted wrongly with a PEM encoded Certificate:
https://crt.sh/?id=14599347345
https://crt.sh/?id=13708984315
https://crt.sh/?id=13618597454
https://crt.sh/?id=13572525934

Impacted Requirements
• CA/B Forum Baseline Requirements:Section 7.1.2 Certificate Content and Extensions
“all certificates that it issues MUST comply with one of the following certificate profiles, which incorporate, and are derived from RFC 5280. Except as explicitly noted, all normative requirements imposed by RFC 5280 shall apply, in addition to the normative requirements imposed by this document.”
• RFC 5280: Section 4.2.2.1. Authority Information Access
“Where the information is available via HTTP or FTP, accessLocation MUST be a uniformResourceIdentifier and the URI MUST point to either a single DER encoded certificate as specified in [RFC2585] or a collection of certificates in a BER or DER encoded "certs-only" CMS message as specified in [RFC2797].”

Based on Incident Reporting Template v. 2.0 effective from October 17, 2023

Comment 2

[Jesper Kristensen]

I am not sure I understand how you determined that your Subscribers were not impacted. Can you explain how you were able to determine this? Have you verified that no relying party has failed to trust a certificate because the relying party failed to parse the CRL? Have you verified that no client could possibly have trusted a revoked certificate because they failed to parse the CRL?

Comment 3

[Martijn Katerbarg]

Could you please clarify: The title and initial sections of this report talk about the CRL having been incorrectly encoded. However, later on in the report, and in your action items you mention having supplied incorrectly encoded certificates and are referencing the AIA extension, which does not contain CRLs.

Is this issue related to incorrectly encoded CRLs, incorrectly encoded AIA -> CA Issuers certificates, or both?

Comment 4

[Ben Wilson]

Please respond to Comment #3 as soon as possible. Also, weekly updates are expected for all CA incident bugs unless a "Next update" has been set in the Whiteboard.

Comment 5

[Nikolett]

Hello everyone!

Answer for comment3
Action items were defined to correct the AIAs because CRLs with incorrect coding were corrected immediately.
After the correction, we discovered that these AIAs also needed correction.
We apologize, this was really not clear in the ticket. This issues is related to both.

The issuer certificate was also available in the same Base64 (PEM) format by default on the "Authority Information Access" path of the CA certificate. We fixed this immediately as well, so that by default the issuer certificate is also available in binary format.
Since we discovered this at the same time as the CRL repair, we treated them as one issue and they were resolved together.

Action Item	Kind	Due Date
The wrongly encoded CRLs have been corrected	Mitigate	2024-12-18 - Completed
Replace the 4 wrongly encoded PEM files with the new DER encoded files at same AIA location	Mitigate	2024-12-18 - Completed
Planing to update the AIA publishing process to check for DER encoding and finishing the integrating process for pkimetal linter	Mitigate	2025-01-31 - Ongoing

We will update the last action item on 2025.03.15.

Comment 6

[Dorottya]

In our previous comment, we indicated that we would update the description of the actions taken. In consultation with our expert colleagues, we have split the implementation of the pkimetal linter into two parts. In the first phase, our developers have prepared a solution to be adapted to our systems, which allows the linter to indicate any existing error when issuing a certificate and CRL and not to allow the certificate to be issued in case of an error. In the second phase we adopt the linter in the system of OCSP response.

The development took longer this way we implemented the first phase until 15 March furthermore we will be implement the second phase in fully production environment on 17 April 2025 instead of the planned date of 15 March 2025. Our experts will perform and document the test procedures required before implementing and using the changes to the system.
After this, the error of encoding CRL in a non-DER format should not occur in the future.

Thank you

Comment 7

[Nikolett]

Dear Community Members,
please be informed that we are proceeding with the development as planned.

Comment 8

[Nikolett]

Dear Community Members,
please be informed, that the new implementation date for the linter: Thursday 1 May 2025, due to needing more time to run the post-tests, the second phase completion expected by the end of the month.

Comment 9

[Nikolett]

Dear Community Members,
We inform you that the complete adaptation of the linter has been done, for CRL and OCSP alike. We will submit a closure summary soon.

Comment 10

[Nikolett]

Dear Community members,

We are working on the closure summary and it will be submitted until this friday (05-09-25).

Comment 11

[Nikolett]

Action Item	Kind	Due Date
The wrongly encoded CRLs have been corrected	Mitigate	2024-12-18 - Completed
Replace the 4 wrongly encoded PEM files with the new DER encoded files	Mitigate	2024-12-18 - Completed
Planning to update the publishing process to check for DER encoding	Mitigate	2025-01-31 - Completed on 2024-12-18
Finishing the integrating process for pkimetal linter	Mitigate	2025-01-31 - Completed on 2025-05-01

Incident Description:
NETLOCK published CRLs in PEM format, which violates RFC 5280 section 4.2.1.13 requiring DER encoding for HTTP/FTP URIs.
The issue was reported on 2024-12-17 and confirmed internally on 2024-12-18. The wrongly encoded CRLs had been corrected the same.

Root Cause(s):
CRL and AIA publication processes did not enforce DER encoding. The default format of our system was PEM, and no validation was in place to detect misconfigured encoding.

Remediation Description:

• Replaced all affected CRL with correctly encoded DER format.
• Implemented pkimetal linter-based validation in two phases:
  Issuance-time validation for certificates and CRLs (March 15, 2025).
  OCSP response integration (April 17, 2025).
  Final testing and production rollout completed on May 1, 2025.

Commitment Summary:
NETLOCK confirms that all action items listed in this report have been completed. The implemented measures—including automation and encoding validation—ensure similar misconfigurations will not occur in the future. We commit to ongoing evaluation and periodic testing of these controls.

We respectfully request that this incident be considered closed.

Comment 12

[Nikolett]

Dear Community Members,
do you have any further questions about the ticket?
All Action Items disclosed in this report have been completed as described, and we request its closure.

Comment 13

[Nikolett]

Dear Community Members,
do you have any further questions about the ticket?

Comment 14

[Nikolett]

Dear Community Members,
All Action Items disclosed in this report have been completed as described, and we request its closure.

Comment 15

[incident-reporting]

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2025-06-10.