Open Bug 1889672 Opened 26 days ago Updated 5 hours ago

Disig: Certificates with incorrect Subject attribute order

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
ASSIGNED

People

(Reporter: jozef.nigut, Assigned: jozef.nigut)

Details

(Whiteboard: [ca-compliance] [ov-misissuance])

Attachments

(1 file, 1 obsolete file)

updated_certificates_1889672.csv
711 bytes, application/vnd.ms-excel
Details
Attached file certificates___.csv (obsolete) —

This is a preliminary report.

Summary

During the investigation of a previous bug (https://bugzilla.mozilla.org/show_bug.cgi?id=1888104), we used pkilint on certificates and found certificates with subject attribute order nonconforming to TLS BR Section 7.1.2.4.

Appendix

Details of affected certificates attached.

Assignee: nobody → jozef.nigut
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] [ov-misissuance]

Incident Report

Summary

This is the final incident report.

After September 15th, 2023 Disig issued TLS certificates with incorrect subject attribute order.
During the investigation of a previous bug (https://bugzilla.mozilla.org/show_bug.cgi?id=1888104), we used pkilint on certificates and found certificates with subject attribute order nonconforming to TLS BR Section 7.1.4.2.
We notified customers about the issue.
On April 9th, 2024 last affected certificate has been revoked.

Impact

The incident impacted 7 TLS OV certificates. Disig has decided to stop the issuance of TLS certificates until a root cause has been found.

Timeline

All times are UTC.

2024-04-04:

  • 11:14 During the investigation of a previous bug, we checked certificates with pkilint and found certificates with incorrect subject attribute order.
  • 11:15 The issuance of the TLS certificates was stopped.
  • 11:26 An incident was created internally and internal investigation has started.
  • 12:22 We have confirmed that the recently issued TLS certificates are not affected by this issue.
  • 15:16 The preliminary bug was filed in bugzilla.
  • 15:49 - 17:14 - We notified customers about the issue, and that affected certificates must be revoked within 5 days.

2024-04-05:

  • 07:04 An analysis of the reasons for issuing a TLS certificates in violation of the requirements of TLS BR 7.1.4.2 was performed.
    The investigation showed that TLS certificates in question were issued from multiple profiles. The issue was not connected with
    profiles configuration. The analysis revealed that subject attribute order of received CSRs was identical to subject attribute order of
    issued certificates.

  • 07:32 CA manager prepared notification that CSRs received from customers must be check in ASN.1 editor for correct subject attribute order.
    Correct subject attribute order must be as follows:
    Attribute OID
    countryName 2.5.4.6
    localityName 2.5.4.7
    organizationName 2.5.4.10
    commonName 2.5.4.3

  • 07:33 The issuance of the TLS certificates was allowed.

  • 07:45 Disig issued new certificate for the FQDN 9999.EESSI-ACC.udzs-sk.sk [https://crt.sh/?id=10568345043].
    Certificate was verified with pkilint and zlint. No issues have been found.

  • 10:28 Disig issued new certificate for the FQDN jira.disig.sk [https://crt.sh/?id=12609286902].
    Certificate was verified with pkilint and zlint. No issues have been found.

  • 11:37 jira.disig.sk has been revoked.

  • 12:50 We issued certificates from different profiles. CA manager tested all with pkilint (version 0.9.10 ) and confirmed that no issues have been found.
    CA manager assigned the task for DevOps team to implement a subject attribute order control within CA system.

2024-04-08:
Disig received another CSRs from customers and issued new certificates.
Certificates were verified with pkilint and zlint. No issues have been found.

2024-04-09:

  • 06:17 Last certificate has been revoked. All affected certificates are revoked.

Root Cause Analysis

The investigation showed that TLS certificates in question were issued from multiple profiles. The issue was not connected with one particular
profile configuration. The analysis revealed that subject attribute order of CSRs sent by customers was identical to subject attribute order of
issued certificates. We found with DevOps and DB teams that CA system used same order (from the CSR) to create new certificate.
We did not realize that subject attribute order in the profile configuration itself (which was correct) has not been used to build the certificate.

Lessons Learned

We learned from this case that also members of DevOps and DB teams, except CA team, must be informed when new requirements need to be implemented.

What went well

We were able quickly respond to incident and coordinate with customer to change the certificate.

What didn't go well

Linters (in crt.sh) we used didn’t show any warning or error. Currently only pkilint detects the error.

Where we got lucky

There have been no disruptions to our customers' websites or online services due to the incorrect subject attribute order.

Action Items

| Action Item | Kind | Due Date |

| Revoke and reissue all affected TLS certificates | Mitigate | 2024-04-09 |
| DevOps and DB teams must be informed when new TLS BRs arise and need to be implemented. | Prevent | 2024-04-15 |
| We will implement a subject attribute order control into the CA system. | Prevent | 2024-06-01 |

Appendix

Details of affected certificates

All affected certificates are listed in the attachment.
https://bugzilla.mozilla.org/attachment.cgi?id=9395046

Jozef,

I noticed at least one more certificate that was affected by this issue, but that has not been added to your list of affected certificates:
https://crt.sh/?sha256=64DC77ADFBC84FB7D8FCA3B88E9E57DB44DAD9EFE256B8F21EE1C607FDE382E4

Seeing as this one has also not been revoked, was this certificate not discovered during your investigation?
If so, can you elaborate on how you've done your internal investigation in order to identify affected certificates?

(In reply to Jozef Nigut from comment #1)

A manager prepared notification that CSRs received from customers must be check in ASN.1 editor for correct subject attribute order.

Also I have a question related to this. From the above explanation, I take it that Disig uses the Subject RDN order as provided in the CSR, for inclusion in the certificate. I wonder what has caused you to take that approach, rather than just using the CSR as a method of Public Key transportation?

This is updated list of affected certificates.

Attachment #9395046 - Attachment is obsolete: true

(In reply to Martijn Katerbarg from comment #2)
Thank you Martijn

We double checked our list , and yes, the certificate was also there, but we did not spotted it.
Our list was exported from DB to excel file. Since we don't have yet automatic linting , we have checked the list manually, and
overlooked that record. After further four eyes investigation we did not find any other affected certificate.

Yes, in our solution, CSR transports not only public key but also the subject.
This gave us more flexibility and allowed us to support any order of subject items required by the client.
This decision was made considering that at the time, existing standards did not require the exact order of items in the subject of the certificate.

Jozef

Update to the Final Incident Report

Incident Report

Summary

This is an update of the final report.

After September 15th, 2023 Disig issued TLS certificates with incorrect subject attribute order.
During the investigation of a previous bug (https://bugzilla.mozilla.org/show_bug.cgi?id=1888104), we used pkilint on certificates and found certificates
with subject attribute order nonconforming to TLS BR Section 7.1.4.2.
We notified customers about the issue.
One affected certificate was reported by Martijn Katerbarg on April 10th, 2024 and we added it to the list.
On April 12th, 2024 last affected certificate has been revoked.

Impact

The incident impacted 8 TLS OV certificates. Disig has decided to stop the issuance of TLS certificates until a root cause has been found.

Timeline

All times are UTC.

2024-04-04:

  • 11:14 During the investigation of a previous bug, we checked certificates with pkilint and found certificates with incorrect subject attribute order.
  • 11:15 The issuance of the TLS certificates was stopped.
  • 11:26 An incident was created internally and internal investigation has started.
  • 12:22 We have confirmed that the recently issued TLS certificates are not affected by this issue.
  • 15:16 The preliminary bug was filed in bugzilla.
  • 15:49 - 17:14 - We notified customers about the issue, and that affected certificates must be revoked within 5 days.

2024-04-05:

  • 07:04 An analysis of the reasons for issuing a TLS certificates in violation of the requirements of TLS BR 7.1.4.2 was performed.
    The investigation showed that TLS certificates in question were issued from multiple profiles. The issue was not connected with
    profiles configuration. The analysis revealed that subject attribute order of received CSRs was identical to subject attribute order of
    issued certificates.

  • 07:32 CA manager prepared notification that CSRs received from customers must be check in ASN.1 editor for correct subject attribute order.
    Correct subject attribute order must be as follows:
    Attribute OID
    countryName 2.5.4.6
    localityName 2.5.4.7
    organizationName 2.5.4.10
    commonName 2.5.4.3

  • 07:33 The issuance of the TLS certificates was allowed.

  • 07:45 Disig issued new certificate for the FQDN 9999.EESSI-ACC.udzs-sk.sk https://crt.sh/?id=10568345043 .
    Certificate was verified with pkilint and zlint. No issues have been found.

  • 10:28 Disig issued new certificate for the FQDN jira.disig.sk https://crt.sh/?id=12609286902
    Certificate was verified with pkilint and zlint. No issues have been found.

  • 11:37 jira.disig.sk has been revoked.

  • 12:50 We issued certificates from diffrent profiles. CA manager tested all with pkilint (version 0.9.10 ) and confirmed that no issues have been found.
    CA manager assigned the task for DevOps team to implement a subject attribute order control within CA system.

2024-04-08:
Disig received another CSRs from customers and issued new certificates.
Certificates were verified with pkilint and zlint. No issues have been found.

2024-04-09:

  • 06:17 Last certificate from the previous list has been revoked.

2024-04-10:

  • 09:49 Disig was notified that at least one more certificate was affected by this issue, but that has not been added to our list of affected certificates.
  • 10:08 Customer was informed about the issue.
  • 11:12 We have double checked our list. We did not find any other affected certificate.
  • 13:03 New certificate has been issued.

2024-04-12:

Root Cause Analysis

The investigation showed that TLS certificates in question were issued from multiple profiles. The issue was not connected with one particuliar
profile. Configuration The analysis revealed that subject attribute order of CSRs sent by customers was identical to subject attribute order of
issued certificates. We found with DevOps and DB teams that CA system used same order (from the CSR) to create new certificate.
We did not realize that subject attribute order in the profile configuration itself (which was correct) has not been used to build the certificate.

Lessons Learned

We learned from this case that members of DevOps and DB teams, except CA team, must be informed when new requirements need to be implemented.

What went well

We were able quickly respond to incident and coordinate with customer to change the certificate.

What didn't go well

Linters (in crt.sh) we used didn’t show any warning or error. Currently only pkilint detects the error.

Where we got lucky

There have been no disruptions to our customers' websites or online services due to the incorrect subject attribute order.

Action Items

Action Item Kind Due Date
Revoke and reissue all affected TLS certificates Mitigate 2024-04-12
DevOps and DB teams must be informed when new TLS BRs arise and need to be implemented. Prevent 2024-04-15
We will implement Four Eyes Principle during the investigation of incidents (e.g. review of list of certificates). Detect 2024-04-15
We will implement a subject attribute order control into the CA system. Prevent 2024-06-01

Appendix

Details of affected certificates

All affected certificates are listed in the attachment.
https://bugzilla.mozilla.org/attachment.cgi?id=9396379

Action Item Update

Status : Done
Today we have successfully implemented control of a Subject RDN order. It is performed before a precertificate and leaf certificate are issued. It will prevent the issuance of certificates with incorrect subject attribute order.

Action Item Update

Status : Done
Both action items (due date: 2024-04-15) were approved today.

(In reply to Jozef Nigut from comment #4)

(In reply to Martijn Katerbarg from comment #2)
Thank you Martijn

We double checked our list , and yes, the certificate was also there, but we did not spotted it.
Our list was exported from DB to excel file. Since we don't have yet automatic linting , we have checked the list manually, and
overlooked that record. After further four eyes investigation we did not find any other affected certificate.

Yes, in our solution, CSR transports not only public key but also the subject.
This gave us more flexibility and allowed us to support any order of subject items required by the client.
This decision was made considering that at the time, existing standards did not require the exact order of items in the subject of the certificate.

Jozef

Hi Josef,

In light of many incidents associated with encoding or other possible errors coming from subject attributes included in CSRs, have you considered changing your past decision to rely on CSR for transporting more than just the public key?

Hi Dimitris,
Yes we did and we decided there’s no need to change the current implementation as our systems perform complex validation of each individual entry in the subject. We don’t just use it „as it is“ without inspection and re-encoding.

We have no new information in this bug.

We have no new information in this bug.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: