Closed Bug 1889699 Opened 8 months ago Closed 3 months ago

Microsec: Disallowed subject attribute field in DV certificate

Categories

(CA Program :: CA Certificate Compliance, task)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: szoke.sandor, Assigned: szoke.sandor)

Details

(Whiteboard: [ca-compliance] [dv-misissuance])

Attachments

(1 file)

24.33 KB, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Details

Incident Report

Summary

It was reported on an MELASZ's internal mailing list that the Digicert linter (https://github.com/digicert/pkilint) already supports the current CABF BR requirements.
Three misissued DV certificates was reported to Microsec.
The problem is that the reported DV certificates contain the SerialNumber extension (2.5.4.5), which is not allowed in DV certificates.
The next day, Microsec received another email from Sectigo reporting a misissued DV certificate with the same problem.

Impact

The misissued DV certificates contain the SerialNumber (2.5.4.5) extension, which is not allowed in DV certificates.
The presence of this extra field has no impact on the usability or security of the certificate, but it unnecessarily increases the size of the certificate.

Timeline

1. Processing MELASZ email

2024-04-03

  • 15:31 UTC (07:31 CETS)
  • some of our colleagues received an email on the MELASZ's internal mailing list about a potential problem with our DV certificates
  • 08:30 UTC (10:30 CETS)
    • Microsec began the investigation of the issue.
    • Microsec studied the current CABF BR requirements and verified that the presence of SerialNumber (2.5.4.5) extension is not allowed in DV certificates, and NOT RECOMMENDED in IV and OV certificates.
    • Microsec recognized that the problem exists in all issued DV certificates, so all valid DV certificates must be replaced
  • 10:27 UTC (12:27 CETS)
    • the director of Microsec was informed of the result of the investigation with the preliminary findings and proposals for solving the problem
    • Microsec decided to remove this extension from DV, IV, and OV certificate profiles, but leave it in the EV profiles
    • Microsec decided to modify its public documents with urgency
  • 14:00 UTC (16:00 CETS)
    • development and testing of modified profiles is complete, new profiles are installed on the live system
    • 1 user domain and 3 test domains are affected as follows:
      • www.taxipay.hu
      • edvtlsca2023-valid.e-szigno.hu
      • edvtlsca2023-expired.e-szigno.hu
      • edvtlsca2023-revoked.e-szigno.hu
  • 14:25 UTC (16:25 CETS)
    • version 3.12 CP and CPS were issued
    • 4 new DV certificates were issued for the 4 domains

2. Processing Sectigo email

2024-04-03

  • 10:54 UTC (12:54 CETS)
    • Microsec received an email from Sectigo to the general purpose email address info@... provided in CCADB reporting a misissued certificate.
    • Microsec's OTRS system received the email messages and sent an automatic notification email with a registration number: Ticket#2024040384005781
  • 12:54 UTC (14:59 CETS)
    • OTRS ticket was moved from "standard INBOX" to "High Priority" waiting list

2024-04-04

  • 07:22 UTC (09:22 CETS)
    • ticket was assigned to the head of Customer Service
  • 07:48 UTC (09:48 CETS)
    • JIRA ticket was opened to manage the issue
    • MSTeams notification was sent to Sándor Szőke, dep. director
  • 09:05 UTC (11:05 CETS)
    • OTRS ticket was assigned to Sándor Szőke
  • 09:14 UTC (11:14 CETS)
    • Sándor Szőke sent a manual notification email to the sender.
    • Microsec began investigating the issue and discovered that it was the same issue as managed the previous day

3. Further processing the issue

2024-04-04

  • 09:20 UTC (11:20 CETS)

    • Microsec contacted the owner of www.taxipay.hu, who asked one day to change the certificate on his site
  • 13:30 UTC (15:30 CETS)

    • Microsec begun revoking the misissued DV certificates
  • 14:10 UTC (16:10 CETS)

    • Microsec finished revoking the misissued DV certificates, except the one which will be revoked next day
  • 17:00 UTC (19:00 CETS)

    • This incident report was opened in Bugzilla.

Root Cause Analysis

2023-08-29

  • Microsec assigns a unique OID to each Client and places this OID into this field for each certificate type to easily identify the Client. The only exception was EV certificates, where this field is used to store other information.
  • Microsec made a mistake when it failed to recognize that this field is not allowed in DV certificates
  • Due to the small number of DV certificates issued, this problem has remained unexplored until now
  • Microsec uses two linters prior the issuance (certlint and zlint), but none of them could indicate this problem.

Lessons Learned

  • We need to study more carefully the requirements for any type of certificates, especially if it is not issued in quantity

What went well

  • we responded to the notification in a timely manner
  • we found the root of the problem
  • We have changed our policies and certificate profiles
  • we have collected the affected certificates
  • we issued the new certificates
  • most of the misissued certificates have been revoked

What didn't go well

  • we should speed up our process of managing High Priority messages

Where we got lucky

  • We were lucky because this problem does not affect the intended use and security of the certificates.

Action Items

Action Item Kind Due Date Status
Issuing new certificates using the corrected certificate profiles Repair 2024-04-04 Done
Contacting all involved Subscribers and informing them of the necessary measures Repair 2024-04-04 Done
Revoking the misissued certificates Repair 2024-04-05 In Progress

Appendix

Details of affected certificates

There are 1 user domain and 3 Microsec domains used for test web sites, the below list contains all the available precertificates:

crt.sh ID Not Before Subject Name serial Revocation time
https://crt.sh/?id=12588022553 2024.04.03 14:54:52 UTC C=HU, CN=www.taxipay.hu 04:a8:e4:83:cb:81:f5:cc:c4:c7:c6:17:ab:0a valid
https://crt.sh/?id=12588014992 2024.04.03 14:53:39 UTC C=HU, CN=www.taxipay.hu 04:a8:e3:76:81:1c:18:91:9a:f5:3f:f8:cd:0a 2024-04-03 14:54:20 UTC
https://crt.sh/?id=11182230554 2023.11.22 C=HU, CN=www.taxipay.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.2550 04:82:90:1d:63:b2:5e:b7:89:bd:99:39:6d:0a to be revoked
https://crt.sh/?id=12587853169 2024.04.03 14:35:00 UTC C=HU, CN=edvtlsca2023-expired.e-szigno.hu 04:a8:d7:25:66:2e:cf:5e:e8:5c:fd:81:4f:0a expired
https://crt.sh/?id=12587909922 2024.04.03 14:35:00 UTC C=HU, CN=edvtlsca2023-expired.e-szigno.hu 04:a8:d7:25:66:2e:cf:5e:e8:5c:fd:81:4f:0a expired
https://crt.sh/?id=12587878558 2024.04.03 14:34:34 UTC C=HU, CN=edvtlsca2023-revoked.e-szigno.hu 04:a8:da:71:1f:d7:27:d3:d0:ba:f1:7e:c6:0a 2024-04-03 14:35:09 UTC
https://crt.sh/?id=12587909087 2024.04.03 14:25:52 UTC C=HU, CN=edvtlsca2023-valid.e-szigno.hu 04:a8:d2:05:a5:2f:d7:07:02:48:3e:01:91:0a valid
https://crt.sh/?id=12587810999 2024.04.03 14:25:52 UTC C=HU, CN=edvtlsca2023-valid.e-szigno.hu 04:a8:d2:05:a5:2f:d7:07:02:48:3e:01:91:0a valid
https://crt.sh/?id=12577744401 2024.04.02 C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560 04:a8:34:87:6c:29:20:c7:e7:65:5a:20:c4:0a 2024-04-04 13:33:03 UTC
https://crt.sh/?id=12577607644 2024.04.02 C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560 04:a8:34:87:6c:29:20:c7:e7:65:5a:20:c4:0a 2024-04-04 13:33:03 UTC
https://crt.sh/?id=12515508437 2024.03.27 C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560 04:a7:02:a2:cf:5d:ed:8f:cb:5a:37:41:2c:0a 2024-04-04 14:08:02 UTC
https://crt.sh/?id=12498961907 2024.03.27 C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560 04:a7:02:a2:cf:5d:ed:8f:cb:5a:37:41:2c:0a 2024-04-04 14:08:02 UTC
https://crt.sh/?id=12514262556 2024.03.27 C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560 04:a6:ac:8f:1b:62:af:08:26:22:e1:dd:c6:0a 2024-04-04 14:08:05 UTC
https://crt.sh/?id=12496767101 2024.03.27 C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560 04:a6:ac:8f:1b:62:af:08:26:22:e1:dd:c6:0a 2024-04-04 14:08:05 UTC
https://crt.sh/?id=12520787537 2024.03.27 C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560 04:a6:a9:58:80:ce:e9:19:88:11:4a:21:58:0a 2024-04-04 14:03:02 UTC
https://crt.sh/?id=12513919538 2024.03.27 C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560 04:a6:a3:62:5b:99:dc:e4:1b:4c:68:a7:0b:0a 2024-04-04 14:03:06 UTC
https://crt.sh/?id=12513834167 2024.03.27 C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560 04:a6:9a:6b:13:22:a9:eb:63:fc:e6:79:fb:0a 2024-04-04 14:03:09 UTC
https://crt.sh/?id=12513827024 2024.03.27 C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560 04:a6:97:da:92:97:9e:5c:19:5b:87:b0:c2:0a 2024-04-04 13:53:03 UTC
https://crt.sh/?id=12508856594 2024.03.26 C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560 04:a6:81:0f:1c:0f:c6:df:d3:65:89:a6:d3:0a 2024-04-04 13:53:06 UTC
https://crt.sh/?id=12508815531 2024.03.26 C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560 04:a6:7c:d8:94:70:d5:2d:7e:3e:90:64:a2:0a 2024-04-04 13:53:09 UTC
https://crt.sh/?id=12495568331 2024.03.26 C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560 04:a6:7c:d8:94:70:d5:2d:7e:3e:90:64:a2:0a 2024-04-04 13:53:09 UTC
https://crt.sh/?id=12291399508 2024.03.06 C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560 04:9e:e2:ed:bc:a6:92:e0:0a:5e:30:72:eb:0a 2024-04-04 13:38:03 UTC
https://crt.sh/?id=12291337592 2024.03.06 C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560 04:9e:e2:ed:bc:a6:92:e0:0a:5e:30:72:eb:0a 2024-04-04 13:38:03 UTC
https://crt.sh/?id=12291093513 2024.03.06 C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560 04:9e:db:5c:c1:98:1c:cf:48:04:a6:fc:4d:0a 2024-03-06 09:34:22 UTC
https://crt.sh/?id=12290628172 2024.03.06 C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560 04:9e:cb:c8:91:8f:0a:a7:bc:9b:e1:d0:67:0a 2024-03-06 09:33:51 UTC
https://crt.sh/?id=11693263276 2024.01.08 C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560 04:8d:48:f5:da:33:8c:18:f9:7f:47:ab:35:0a 2024-04-03 14:46:59 UTC
https://crt.sh/?id=11680562040 2024.01.08 C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560 04:8d:48:f5:da:33:8c:18:f9:7f:47:ab:35:0a 2024-04-03 14:46:59 UTC

https://crt.sh/?sha256=[sha256 fingerprint of the certificate]

Based on Incident Reporting Template v. 2.0

(In reply to dr. Sándor SZŐKE from comment #0)

Impact

The misissued DV certificates contain the SerialNumber (2.5.4.5) extension, which is not allowed in DV certificates.
The presence of this extra field has no impact on the usability or security of the certificate, but it unnecessarily increases the size of the certificate.

CCADB is clear what information belongs in the Impact section. It's not an opportunity to downplay the importance of an incident. It should contain the important information upfront for a reader to asses the size and scope of misissuance. From CCADB:

The Impact section should contain a short description of the size and nature of the incident. For example: how many certificates, OCSP responses, or CRLs were affected; whether the affected objects share features (such as issuance time, signature algorithm, or validation type); and whether the CA Owner had to cease issuance during the incident.

These questions were not answered in the report:

  • How many certificates were misissued?
  • When was the first misissuance and the start of the incident?
  • Did Microsec cease issuance during the incident?

Additionally the timeline included doesn't meet expectations. It should include events from before Microsec became aware of the issue (e.g. certificate issuance, policy reviews, software changes, policies coming into effect). From CCADB:

The Timeline section must include a detailed timeline of all events and actions leading up to and taken during and after the incident. The timeline must include not just the actual discovery of the incident and subsequent events, but also relevant events occuring beforehand (e.g. something changed or was introduced).

Thank you for your comments, we collect the requested information and respond soon.

Hi Sándor,

Section 4.9.1.1 of the TLS BRs states the following:

With the exception of Short-lived Subscriber Certificates, the CA SHOULD revoke a certificate within 24 hours and MUST revoke a Certificate within 5 days and use the corresponding CRLReason (see Section 7.2.2) if one or more of the following occurs:
...
...
12. The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement (CRLReason #4, superseded);
...
...

I noticed a few of these certificates were revoked with a reason of cessationOfOperation (e.g., https://crt.sh/?id=12577744401, https://crt.sh/?id=12577607644, and https://crt.sh/?id=12515508437). Can you help us understand this discrepancy and how it should be considered against the language found in BRs (above) and the corresponding CPS?

Thanks,
Ryan

Flags: needinfo?(szoke.sandor)

Hi Ryan,

I checked the revoked certificates and unfortunately you are rigth, 10 misissued certificates were revoked with a reason code of cessationOfOperation yesterday afternoon.

We started the investigation for this problem.

Flags: needinfo?(szoke.sandor)

Incident Status Report 2024-04-06

Impact - further information

  • In the Appendix of the incident report we provided a list of misissued certificates. We added more information to that list and loaded a detailed excel table with the relevant information to this ticket
  • Microsec issued faulyt certificates for 1 user domain and 3 Microsec domains used for test websites
  • the total number of misissued certificates is 23 and there were 4 precertificates whithout leaf certificates issued
  • the incident did not affect the issuance of the OCSP responses and CRLs
  • after receiving the notification about the misissuance, the problem was resolved within half a day and during that time there was no further DV certificate issuance

Timeline - further information

2023-04-11

  • Baseline Requirements for the Issuance and Management of Publicly Trusted Certificates Version 2.0.0 was published
    • section 7. CERTIFICATE, CRL, AND OCSP PROFILES has been totally rewritten with several new requirements
    • the use of the SerialNumber extension (2.5.4.5) in DV certificates was not previously prohibited, but according to the new requirements, the use of this extension is
      • NOT RECOMMENDED in case of IV and OV certificates
      • not allowed in case of DV certificates
      • still mandatory in case of EV certificates according to EV Guidelines
    • the deadline to update the certificate profiles was 2023-09-15

2023-08-30

  • version 3.7 of our CP and CPS were published with the following main changes
    • Global revision
    • Revocation reasons
    • Certificate validity times
    • Key Usage
    • Identity validation by a third party
    • Managing security incidents
    • Supported cryptographic algorithms
    • New, dedicated TLS hierachy
  • the use of SerialNumber extension is still mandatory for DV, IV and OV certificates in our CP and CPS
  • it was our fault not to recognize the change in the CABF BR requirement for DV certificates

2023-10-30

  • version 3.10 of our CP and CPS were published with the following main changes
    • Identity validation of natural persons
    • CA hierarchies
  • the use of SerialNumber extension is still mandatory for DV, IV and OV certificates

2023-12-22

2023-12-19

  • version 3.11 of our CP and CPS were published with the following main changes
    • Revision, some small changes
  • the use of SerialNumber extension is still mandatory for DV, IV and OV certificates

2024-04-03

  • first notification email received about the issue
  • version 3.12 of our CP and CPS were published with the following main changes
    • Removing SerialNumber extension
  • certificate policies upgraded according to the CABF BR requirement

2024-04-04

  • This incident report was opened in Bugzilla
  • Comment in this Bugzilla ticket from Mathew Hodson requesting more information

2024-04-05

  • Comment in this Bugzilla ticket from Ryan Dickson indicating the use of an incorrect revocation reason code in some cases
    • Microsec realized that during the revocation of the misissued certificates on 2024-04-04 afternoon, not the correct revocation reason code was set
    • the fault affected only the domain edvtlsca2023-valid.e-szigno.hu, which is used for our test site in accordance with the Root Program Requirements
    • the revocation for this test domain was made manually and the operator set an incorrect reason code value
    • to solve the issue, Microsec decided to retire the affected key
      • the valid certificate was revoked with reason code keyCompromise
      • the affected revocation reason codes were upgraded from cessationOfOperation to keyCompromise
      • new key was generated for that test site
      • new certificate was issued for that test site
    • currently we have only one revoked DV certificate with reason code cessationOfOperation, it belongs to another test site from edvtlsca2023-revoked.e-szigno.hu
      • this revocation happend only for test purposes, so we left this case as it is

2024-04-06

  • we collected all our DV certificates issued after 2023-09-15
    • from crt.sh
    • from our internal certificate database
  • we merged the 2 lists and uploaded the result to this Bugzilla ticket
  • we also collected the requested additional information and upload it to Bugzilla as part of this status report

Action Items

Action Item Kind Due Date Status
Issuing new certificates using the corrected certificate profiles Repair 2024-04-04 Done
Contacting all involved Subscribers and informing them of the necessary measures Repair 2024-04-04 Done
Revoking the misissued certificates Repair 2024-04-05 Done

Appendix

Details of affected certificates uploaded in an excel file

Incident Status Report - 2024-04-09

Timeline

2024-04-09

Action Items

Action Item Kind Due Date Status
Issuing new certificates using the corrected certificate profiles Repair 2024-03-20 Done
Contacting all involved Subscribers and informing them of the necessary measures Repair 2024-03-20 Done
Revoking 44 of 46 misissued certificates Repair 2024-03-24 Done
Revoking 2 misissued PSD2 certificates Repair 2024-04-09 Done
Supervision of our practice in case of changes in requirements Prevent 2024-04-30 Started

Appendix

Details of affected certificates

Misissued certificate (or precertificate) Revocation time / OCSP answer
https://crt.sh/?id=10727629590 2024-04-09 12:08:03 UTC
https://crt.sh/?id=12240328933 2024-04-09 12:58:03 UTC

Please ignore my previous comment, it belongs to another bug.

Assignee: nobody → szoke.sandor
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] [dv-misissuance]

Incident Status Report 2024-04-30

Action Item: Supervision of our practice in case of changes in requirements

CP/CPS change management

In order to better understand what happened, we briefly explain how Microsec handles the changes in its public documents (CP and CPS).

  • Public documents are generated using LATEX from a common source.
    We have several trust services and currently maintain 32 public documents in two languages from this common source.
    To make this manageable, we use common versioning for all documents.
    Not all documents are published in every version, so in the case of a specific document some subversions may be skipped, this is normal behavior.

  • Content changes are tracked in our jira system.
    After publishing a version of our public documents, we immediately open a further jira epic for the next document version.
    In case of any change request or any issue that arises, we open a jira ticket in this epic where we describe the issue in detail.
    Tickets are commented on by members of the EHSZ group, consisting the EHSZ management and employees dedicated to compliance and process control issues.
    If necessary, the EHSZ management makes a decision on proposed change.
    When urgently needed, but at least once in a year, a new version is published.
    Based on the importance and urgency of the collected tasks, a decision is made as to which changes will be included in the next release and which will be postponed to later releases.

  • The LATEX source is modified by using specific filters that indicate the changes.
    Internal draft versions are translated and revised in several turns until the modified wording is approved.

  • This process ensures that no any proposed change can be forgotten and that each change is approved by the responsible people.

Monitoring external requirements

In order to better understand what happened, we briefly explain how Microsec handles the changes in the external requirements

  • Microsec regularly monitors changes in external requirements. When a new version of a relevant requirement is published, we study it carefully.

  • The entire process is also managed in our jira system.
    Different types of requirements are monitored in various epics.
    When a new version of a specific requirement is released, we open a new jira task in the corresponding epic and document the research result in it.
    These jira tasks are managed by our EHSZ compliance team.

  • The investigation focuses primarily on changes to the specification.
    We examine the version of the requirement published with change tracking, but we also make our own comparison with the previous version of the given document.
    Each change is classified in the document according to its relevance to us.

  • During the investigation, we mark the changes by different colors in the document according to their relevance to us, and we also make comments on it.
    If changes are necessary, we open the necessary jira tickets to modify our public documents, our processes, request developments or anything else.

  • This practice generally ensures that no requirements changes can be overlooked or forgotten.

Root of the problem

  • In this case, the problem was that the entire CABF BR Chapter 7 was completely rewritten and the entire chapter was reorganized with many more overlapping details that could not be fully covered by the method we used.
  • Of course, we also used dual control now, and even more people reviewed the new CABF BR requirement and described what needed to be changed.
    However, due to the complexity of the task, this change for DV certificates evaded our attention.
  • We see that our standard process does not guarantee success with such a large change in a requirement.

Improvemets in our processes

  • In order to prevent similar mistakes, we have decided that in the future, in the event of changes of similar complexity, beyond our normal process, we will compile a separate list of requirements for each type of certificate, where the requirements applicable to the given type of certificate are more transparent.

Action Items Status

Action Item Kind Due Date Status
Issuing new certificates using the corrected certificate profiles Repair 2024-04-04 Done
Contacting all involved Subscribers and informing them of the necessary measures Repair 2024-04-04 Done
Revoking the misissued certificates Repair 2024-04-05 Done
Supervision of our practice in case of changes in requirements Prevent 2024-04-30 Done

Is this Bugzilla incident report ready to be closed?

Flags: needinfo?(szoke.sandor)

We do not have any open issues regarding this incident report.

Flags: needinfo?(szoke.sandor)

I will close this on or about Wed. 28-Aug-2024.

Flags: needinfo?(bwilson)
Status: ASSIGNED → RESOLVED
Closed: 3 months ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: