1965807 - certSIGN: Findings in 2025 ETSI Audit - Audit Incident Report #4 – Expired cert with bad order of attributes

Status: RESOLVED DUPLICATE of bug 1886624

(CA Program :: CA Certificate Compliance, task)

Description

[Gabriel PETCU]

Full Incident Report

Finding #4 – Expired cert with bad order of attributes

Summary

The test certificate from 2023 : "testssl-expired-evcp.certsign.ro.crt" did not respected the relative order of the Subject attributes as required in BR #7.1.4.2 (CA/Browser Forum v2.1.2 #7.1.4.2).

• CA Owner CCADB unique ID: A000013

• Incident description: The test certificate issued in 2023 : "testssl-expired-evcp.certsign.ro.crt" did not respected the relative order of the Subject attributes.

• Timeline summary:

  • Non-compliance start date: 3-Nov-2023
  • Non-compliance identified date: 29-Apr-2025
  • Non-compliance end date: 4-Nov-2023

• Relevant policies: CA/Browser Forum v2.1.2 #7.1.4.2.

• Source of incident disclosure:
  CAB-Forum_AAL_Standard_Audit_1612-377-v2.pdf
  CAB-Forum_AAL_TLS-BR_Audit_1612-377-v2.pdf
  CAB-Forum_AAL_TLS-EV_Audit_1612-377-v2.pdf

Impact

There is no impact.

• Total number of certificates: 1
• Total number of "remaining valid" certificates: 0
• Affected certificate types: This incident was related to EV certificates
• Incident heuristic: No other certificates had been affected by this incident
• Was issuance stopped in response to this incident, and why or why not?: No, as no other certificates had been affected.
• Analysis: The certificate was a test certificate that expired in a day after issuance, so it was not “valid” to be replaced.
• Additional considerations:

Timeline

All the times are UTC time.

29-Apr-2025 13:06 – Awareness of the incident - email receiving the Audit Attestation Letters from the auditors.

29-Apr-2025 15:00 – Internal certSIGN meeting discussing the finding received and checking that the correction was already done.

30-Apr-2025 09:53 – Open preliminary report - Bugzilla ticket Bug 1963546

Related Incidents

N/A

Bug	Date	Description
#1886624 https://bugzilla.mozilla.org/show_bug.cgi?id=1886624	2024-03-20	“certSIGN: Certificates with incorrect Subject attribute order” The certificate was included in the list of bugs, attached to the ticket, that had been issued with an incorrect order of attributes

Root Cause Analysis

Root Cause #1

Contributing Factor #: title Not revoking and replacing expired certificates.

• Description: When created the list for the bug #1886624 “certSIGN: Certificates with incorrect Subject attribute order”, we included the certificate above: crt.sh/?serial=234F86B9A9DC729F0531. The fix was to revoke and replace all valid certificates, and as this certificate was already expired, it remained as it was.

• Timeline:
  On 3-Nov-2023 a test certificate was issued for the EV type – to test the expiration – and the order of two attributes had been incorrect.
  On 4-Nov-2023 the test certificate expired and remained on the test site.

• Detection: The ETSI auditors detected the error during the 2025 audit and marked it as a finding in the AAL.

• Interaction with other factors: N/A

• Root Cause Analysis methodology used: N/A

Lessons Learned

• What went well: The expiration testing of SSL certificates was not impacted.
• What didn’t go well: On issuance, two attributes had been with incorrect order.
• Where we got lucky: There was no impact on the certificates nor on the users.
• Additional:

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Replace the certificate	Correct	Root Cause # 1	Check content	2025-02-24	Complete
Quarterly review	Prevent	Root Cause # 1	Internal audit Report	quarterly	Ongoing

Appendix

BCDF32DFE3E5331C3B62D59FE5F1B2FF7E6D8A8A4658BD4C9420D5B1FE0F8004

3A2F4F779076D23D6864BC478F666951B1F92FD8CF603AD2F7B413FEC80BE2BD

CN=testssl-expired-evcp.certsign.ro, O=certSIGN SA, L=Bucharest, C=RO, jurisdictionC=RO, serialNumber=J40/484/2006, businessCategory=Private Organization, organizationIdentifier=VATRO-18288250

CN=certSIGN Web CA, O=CERTSIGN SA, C=RO, organizationIdentifier=VATRO-18288250

Fri, 03 Nov 2023 103501 UTC

Sat, 04 Nov 2023 103501 UTC

234F86B9A9DC729F0531

DNS:testssl-expired-evcp.certsign.ro

N/A

N/A

Comment 1

[HARICA]

Thank you for the report. A few questions that will -hopefully- help you to improve the quality of incident reports going forward.

One of the lessons learned in bugzilla and public incident handling is that the actual timeline should include events that occurred even before the non-compliance started, in order to help you and the community understand the factors and conditions that lead to the non-compliance. Can you please update the timeline to include more events that lead to the mis-issuance?

You indicate that the Non-compliance end date was 4-Nov-2023, yet Non-compliance identified date was 29-Apr-2025. Didn't you identify the non-compliance yourselves in November 2023? If you did, why wasn't an incident report filed according to MRSP?

Also, do you support pre-sign linting and if so, which linters do you use?

Comment 2

[Dimitris Zacharopoulos]

Apologies, this should have been submitted from my personal account (Ben, not sure if there is a way to delete comment 1 as redundant). Re-submitting here:

Thank you for the report. A few questions that will -hopefully- help you to improve the quality of incident reports going forward.

One of the lessons learned in bugzilla and public incident handling is that the actual timeline should include events that occurred even before the non-compliance started, in order to help you and the community understand the factors and conditions that lead to the non-compliance. Can you please update the timeline to include more events that lead to the mis-issuance?

You indicate that the Non-compliance end date was 4-Nov-2023, yet Non-compliance identified date was 29-Apr-2025. Didn't you identify the non-compliance yourselves in November 2023? If you did, why wasn't an incident report filed according to MRSP?

Also, do you support pre-sign linting and if so, which linters do you use?

Comment 3

[chrome-root-program]

Thank you for creating separate incident reports for the non-conformities identified in your most recent audit. This aligns with the proposed update to the CCADB Policy.

We have listed several updates below, which are directed to this incident report, but should also be considered for updates to the other incident reports certSIGN has posted.

Requested Updates

(1) The “Summary” field in Bugzilla (i.e., “Subject line”) should be updated with a title that highlights the type of incident being reported (i.e., something more descriptive of the specific issue being disclosed). The CCADB Incident Reporting Guidelines (IRGs) describe this here with an example.

(2) The “Source of incident disclosure” in the report should be a choice of “Self Reported”, “Third Party Reported”, or “Audit”, as described in the CCADB IRGs.

(3) Comment 1 accurately identifies the issue with the “Timeline” section. This should be corrected.

(4) The “Related Incidents” Section is supposed to include a list of all incidents disclosed to the ‘CA Certificate Compliance’ Bugzilla Component related to this incident that have taken place minimally in the last two years and must consider incidents beyond those corresponding to the CA Owner subject of this report. A quick search highlights several incidents that we’d mostly consider “related”.

(5) We would not consider the “Root Cause Analysis” Section to contain a detailed analysis of the conditions which combined to give rise to the issue. Rather, it just points to 1886624, which is/was also lacking in RCA. In 1886624 the root cause is a single sentence that states “Our linter software was not catching the incorrect order due to a misconfiguration.” Further questioning by a Root Store Operator was able to extract additional action items not included in the original report. In general, we would encourage certSIGN to document a much more robust RCA.

(6) The “Lessons Learned” Section shares very little that could be helpful to all CA Owners in building better systems, policies and/or processes. This is another area where we would encourage certSIGN to offer more detail.

(7) The “Action Items” Section incorrectly lists “Correct” as a kind of Action Item. The kind of Action Items listed on CCADB.org are Prevent, Mitigate, and Detect. Further, the evaluation criteria does not include how the public can measure the impact. In general, the Action Items described would benefit from a more robust RCA and Lessons Learned, if not in this report, then in other and future reports by certSIGN.

Comment 4

[Gabriel PETCU]

(In reply to Dimitris Zacharopoulos from comment #2)

Apologies, this should have been submitted from my personal account (Ben, not sure if there is a way to delete comment 1 as redundant). Re-submitting here:

Thank you for the report. A few questions that will -hopefully- help you to improve the quality of incident reports going forward.

One of the lessons learned in bugzilla and public incident handling is that the actual timeline should include events that occurred even before the non-compliance started, in order to help you and the community understand the factors and conditions that lead to the non-compliance. Can you please update the timeline to include more events that lead to the mis-issuance?

You indicate that the Non-compliance end date was 4-Nov-2023, yet Non-compliance identified date was 29-Apr-2025. Didn't you identify the non-compliance yourselves in November 2023? If you did, why wasn't an incident report filed according to MRSP?

Also, do you support pre-sign linting and if so, which linters do you use?

I will complete the full timeline to this ticket also.
We considered that the non-compliance begin when the certificate was issued, and ended when the certificate expired. The non-compliance was identified by the auditors after the 2025 audit, and communicated on 29-Apr-2025.
Yes, we do support pre-sign linting and we are using an internally developed linter.

Comment 5

[Gabriel PETCU]

(In reply to chrome-root-program from comment #3)

Thank you for creating separate incident reports for the non-conformities identified in your most recent audit. This aligns with the proposed update to the CCADB Policy.

We have listed several updates below, which are directed to this incident report, but should also be considered for updates to the other incident reports certSIGN has posted.

Requested Updates

(1) The “Summary” field in Bugzilla (i.e., “Subject line”) should be updated with a title that highlights the type of incident being reported (i.e., something more descriptive of the specific issue being disclosed). The CCADB Incident Reporting Guidelines (IRGs) describe this here with an example.

(2) The “Source of incident disclosure” in the report should be a choice of “Self Reported”, “Third Party Reported”, or “Audit”, as described in the CCADB IRGs.

(3) Comment 1 accurately identifies the issue with the “Timeline” section. This should be corrected.

(4) The “Related Incidents” Section is supposed to include a list of all incidents disclosed to the ‘CA Certificate Compliance’ Bugzilla Component related to this incident that have taken place minimally in the last two years and must consider incidents beyond those corresponding to the CA Owner subject of this report. A quick search highlights several incidents that we’d mostly consider “related”.

(5) We would not consider the “Root Cause Analysis” Section to contain a detailed analysis of the conditions which combined to give rise to the issue. Rather, it just points to 1886624, which is/was also lacking in RCA. In 1886624 the root cause is a single sentence that states “Our linter software was not catching the incorrect order due to a misconfiguration.” Further questioning by a Root Store Operator was able to extract additional action items not included in the original report. In general, we would encourage certSIGN to document a much more robust RCA.

(6) The “Lessons Learned” Section shares very little that could be helpful to all CA Owners in building better systems, policies and/or processes. This is another area where we would encourage certSIGN to offer more detail.

(7) The “Action Items” Section incorrectly lists “Correct” as a kind of Action Item. The kind of Action Items listed on CCADB.org are Prevent, Mitigate, and Detect. Further, the evaluation criteria does not include how the public can measure the impact. In general, the Action Items described would benefit from a more robust RCA and Lessons Learned, if not in this report, then in other and future reports by certSIGN.

We will update all incidents reported according to the observations.
The updates for this issue are:
(1) Subject line” - the Title was appended with: " – Expired cert with bad order of attributes"
(2) “Source of incident disclosure” is "Audit"
(3) "Timeline" - add :
2024-03-18:

• 13:07 an email was received with a Certificate Problem Report about a non-conformity on the Subject Attribute Encoding order
• 16:00 an analysis of the cause of the non-conformity started
• 16:10 the third party informing certSIGN on the Certificate Problem Report was acknowledged
• 16:30 certSIGN stopped the issuance of TLS certificates
• 16:30 a search started on all certificates issued since the effective date of the CA/Browser Forum version 2.0.0, that is September 15, 2023.
• 17:00 the affected main Subscribers were identified, and certSIGN started to notify them

2024-03-19:

• 17:30 the search of the certificates issued was finalized

2024-03-20:

• 08:00 certSIGN informed the WebTrust auditors about the incident
• 14:00 the new update was deployed in Production
• 15:30 certSIGN restarted the issuance of TLS certificates
• 22:30 the incident report was registered in Bugzilla

2024-03-26 14:00 The software update testing process had been reviewed.

(4) “Related Incidents” - add the preliminary report - Bugzilla ticket Bug 1963546
(5) “Root Cause Analysis” - add the initial root cause: Our linter software was not catching the incorrect order due to a misconfiguration. We revoked and reissued all affected valid certificates. The process is correct but the steps had not all been followed. An extra check was added in the internal procedure for the validator to explicitly verify the configuration files.
(6) “Lessons Learned” - no change
(7) “Action Items” - replace "Correct" with "Mitigate"

Comment 6

[Dimitris Zacharopoulos]

(In reply to Gabriel PETCU from comment #4)

(In reply to Dimitris Zacharopoulos from comment #2)

Apologies, this should have been submitted from my personal account (Ben, not sure if there is a way to delete comment 1 as redundant). Re-submitting here:

Thank you for the report. A few questions that will -hopefully- help you to improve the quality of incident reports going forward.

One of the lessons learned in bugzilla and public incident handling is that the actual timeline should include events that occurred even before the non-compliance started, in order to help you and the community understand the factors and conditions that lead to the non-compliance. Can you please update the timeline to include more events that lead to the mis-issuance?

You indicate that the Non-compliance end date was 4-Nov-2023, yet Non-compliance identified date was 29-Apr-2025. Didn't you identify the non-compliance yourselves in November 2023? If you did, why wasn't an incident report filed according to MRSP?

Also, do you support pre-sign linting and if so, which linters do you use?

I will complete the full timeline to this ticket also.
We considered that the non-compliance begin when the certificate was issued, and ended when the certificate expired. The non-compliance was identified by the auditors after the 2025 audit, and communicated on 29-Apr-2025.

If the non-compliance was when the certificate was issued, in November 2023, your incident report should address this issue, perhaps separately from the current incident.

Yes, we do support pre-sign linting and we are using an internally developed linter.

Section 4.3.1.2 of the TLS BRs has a link pointing to recommended Linting tools that have been widely adopted by the industry. Assuming this incident could have been avoided if CertSign had decided to use those tools, perhaps in addition to your internally-developed linter, you can address this issue by adding an action to implement some of those tools? By reviewing similar incidents I'm sure you will find several CAs that decided to implement pkimetal which covers a large number of cases.

Comment 7

[Gabriel PETCU]

(In reply to Dimitris Zacharopoulos from comment #6)

(In reply to Gabriel PETCU from comment #4)

(In reply to Dimitris Zacharopoulos from comment #2)

Apologies, this should have been submitted from my personal account (Ben, not sure if there is a way to delete comment 1 as redundant). Re-submitting here:

Thank you for the report. A few questions that will -hopefully- help you to improve the quality of incident reports going forward.

One of the lessons learned in bugzilla and public incident handling is that the actual timeline should include events that occurred even before the non-compliance started, in order to help you and the community understand the factors and conditions that lead to the non-compliance. Can you please update the timeline to include more events that lead to the mis-issuance?

You indicate that the Non-compliance end date was 4-Nov-2023, yet Non-compliance identified date was 29-Apr-2025. Didn't you identify the non-compliance yourselves in November 2023? If you did, why wasn't an incident report filed according to MRSP?

Also, do you support pre-sign linting and if so, which linters do you use?

I will complete the full timeline to this ticket also.
We considered that the non-compliance begin when the certificate was issued, and ended when the certificate expired. The non-compliance was identified by the auditors after the 2025 audit, and communicated on 29-Apr-2025.

If the non-compliance was when the certificate was issued, in November 2023, your incident report should address this issue, perhaps separately from the current incident.

But we have opened previously the incident report in ticket #1886624 , that included this certificate in the list of bad attribute order certificates. At that moment the fix revoked and reissued the certificates in the list.

Yes, we do support pre-sign linting and we are using an internally developed linter.

Section 4.3.1.2 of the TLS BRs has a link pointing to recommended Linting tools that have been widely adopted by the industry. Assuming this incident could have been avoided if CertSign had decided to use those tools, perhaps in addition to your internally-developed linter, you can address this issue by adding an action to implement some of those tools? By reviewing similar incidents I'm sure you will find several CAs that decided to implement pkimetal which covers a large number of cases.

certSIGN is using both its own linter and pkimetal, since last year. But at the moment of the issuance of the certificate and of the correction of the initial incident report ticket, in March 2024, the pkimetal was not yet implemented in certSIGN.

Comment 8

[incident-reporting]

This report has gone stale. certSIGN, please provide (a) an update or (b) a Closure Report if you feel this is ready for closure.

Comment 9

[Gabriel PETCU]

Closure Report

Report Closure Summary

• Incident description: The test certificate issued in 2023 : "testssl-expired-evcp.certsign.ro.crt" did not respected the relative order of the Subject attributes.

• Incident Root Cause(s): When created the list for the bug #1886624 “certSIGN: Certificates with incorrect Subject attribute order”, we included the certificate above: crt.sh/?serial=234F86B9A9DC729F0531. The fix was to revoke and replace all valid certificates, and as this certificate was already expired, it remained as it was.

• Remediation description: Replace the certificate, and add an extra step of verification through internal audit.

• Commitment summary: Permanently add, to the yearly Internal Audit Plan, an overall verification of a set of certificates, done by a third party, like the internal auditor, checking all the test SSL certificates with certSIGN Linter.

All Action Items disclosed in this report have been completed as described, and we request its closure.

Comment 10

[Martijn Katerbarg]

Based on the fact that this certificate itself was already part of bug 1886624, and correctly listed in that bug's affected certificates list, I am wondering if the current bug is valid at all.

My understanding of the root store and CCADB policies is that any finding within an audit report needs to be covered by an incident report. Since this finding was already covered by bug 1886624, it seems to me that this bug 1965807, is not required.

@incident-reporting, based on the above, would it be reasonable to mark this bug Closed as a Duplicate of 1886624?