1889699 - Microsec: Disallowed subject attribute field in DV certificate

Status: ASSIGNED

(CA Program :: CA Certificate Compliance, task)

Description

[dr. Sándor SZŐKE]

Incident Report

Summary

It was reported on an MELASZ's internal mailing list that the Digicert linter (https://github.com/digicert/pkilint) already supports the current CABF BR requirements.
Three misissued DV certificates was reported to Microsec.
The problem is that the reported DV certificates contain the SerialNumber extension (2.5.4.5), which is not allowed in DV certificates.
The next day, Microsec received another email from Sectigo reporting a misissued DV certificate with the same problem.

Impact

The misissued DV certificates contain the SerialNumber (2.5.4.5) extension, which is not allowed in DV certificates.
The presence of this extra field has no impact on the usability or security of the certificate, but it unnecessarily increases the size of the certificate.

Timeline

1. Processing MELASZ email

2024-04-03

• 15:31 UTC (07:31 CETS)

• some of our colleagues received an email on the MELASZ's internal mailing list about a potential problem with our DV certificates

• 08:30 UTC (10:30 CETS)
  • Microsec began the investigation of the issue.
  • Microsec studied the current CABF BR requirements and verified that the presence of SerialNumber (2.5.4.5) extension is not allowed in DV certificates, and NOT RECOMMENDED in IV and OV certificates.
  • Microsec recognized that the problem exists in all issued DV certificates, so all valid DV certificates must be replaced
• 10:27 UTC (12:27 CETS)
  • the director of Microsec was informed of the result of the investigation with the preliminary findings and proposals for solving the problem
  • Microsec decided to remove this extension from DV, IV, and OV certificate profiles, but leave it in the EV profiles
  • Microsec decided to modify its public documents with urgency
• 14:00 UTC (16:00 CETS)
  • development and testing of modified profiles is complete, new profiles are installed on the live system
  • 1 user domain and 3 test domains are affected as follows:
    • www.taxipay.hu
    • edvtlsca2023-valid.e-szigno.hu
    • edvtlsca2023-expired.e-szigno.hu
    • edvtlsca2023-revoked.e-szigno.hu
• 14:25 UTC (16:25 CETS)
  • version 3.12 CP and CPS were issued
  • 4 new DV certificates were issued for the 4 domains

2. Processing Sectigo email

2024-04-03

• 10:54 UTC (12:54 CETS)
  • Microsec received an email from Sectigo to the general purpose email address info@... provided in CCADB reporting a misissued certificate.
  • Microsec's OTRS system received the email messages and sent an automatic notification email with a registration number: Ticket#2024040384005781
• 12:54 UTC (14:59 CETS)
  • OTRS ticket was moved from "standard INBOX" to "High Priority" waiting list

2024-04-04

• 07:22 UTC (09:22 CETS)
  • ticket was assigned to the head of Customer Service
• 07:48 UTC (09:48 CETS)
  • JIRA ticket was opened to manage the issue
  • MSTeams notification was sent to Sándor Szőke, dep. director
• 09:05 UTC (11:05 CETS)
  • OTRS ticket was assigned to Sándor Szőke
• 09:14 UTC (11:14 CETS)
  • Sándor Szőke sent a manual notification email to the sender.
  • Microsec began investigating the issue and discovered that it was the same issue as managed the previous day

3. Further processing the issue

2024-04-04

• 09:20 UTC (11:20 CETS)

  • Microsec contacted the owner of www.taxipay.hu, who asked one day to change the certificate on his site

• 13:30 UTC (15:30 CETS)

  • Microsec begun revoking the misissued DV certificates

• 14:10 UTC (16:10 CETS)

  • Microsec finished revoking the misissued DV certificates, except the one which will be revoked next day

• 17:00 UTC (19:00 CETS)

  • This incident report was opened in Bugzilla.

Root Cause Analysis

2023-08-29

• Microsec assigns a unique OID to each Client and places this OID into this field for each certificate type to easily identify the Client. The only exception was EV certificates, where this field is used to store other information.
• Microsec made a mistake when it failed to recognize that this field is not allowed in DV certificates
• Due to the small number of DV certificates issued, this problem has remained unexplored until now
• Microsec uses two linters prior the issuance (certlint and zlint), but none of them could indicate this problem.

Lessons Learned

• We need to study more carefully the requirements for any type of certificates, especially if it is not issued in quantity

What went well

• we responded to the notification in a timely manner
• we found the root of the problem
• We have changed our policies and certificate profiles
• we have collected the affected certificates
• we issued the new certificates
• most of the misissued certificates have been revoked

What didn't go well

• we should speed up our process of managing High Priority messages

Where we got lucky

• We were lucky because this problem does not affect the intended use and security of the certificates.

Action Items

Action Item	Kind	Due Date	Status
Issuing new certificates using the corrected certificate profiles	Repair	2024-04-04	Done
Contacting all involved Subscribers and informing them of the necessary measures	Repair	2024-04-04	Done
Revoking the misissued certificates	Repair	2024-04-05	In Progress

Appendix

Details of affected certificates

There are 1 user domain and 3 Microsec domains used for test web sites, the below list contains all the available precertificates:

crt.sh ID	Not Before	Subject Name	serial	Revocation time
https://crt.sh/?id=12588022553	2024.04.03 14:54:52 UTC	C=HU, CN=www.taxipay.hu	04:a8:e4:83:cb:81:f5:cc:c4:c7:c6:17:ab:0a	valid
https://crt.sh/?id=12588014992	2024.04.03 14:53:39 UTC	C=HU, CN=www.taxipay.hu	04:a8:e3:76:81:1c:18:91:9a:f5:3f:f8:cd:0a	2024-04-03 14:54:20 UTC
https://crt.sh/?id=11182230554	2023.11.22	C=HU, CN=www.taxipay.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.2550	04:82:90:1d:63:b2:5e:b7:89:bd:99:39:6d:0a	to be revoked
https://crt.sh/?id=12587853169	2024.04.03 14:35:00 UTC	C=HU, CN=edvtlsca2023-expired.e-szigno.hu	04:a8:d7:25:66:2e:cf:5e:e8:5c:fd:81:4f:0a	expired
https://crt.sh/?id=12587909922	2024.04.03 14:35:00 UTC	C=HU, CN=edvtlsca2023-expired.e-szigno.hu	04:a8:d7:25:66:2e:cf:5e:e8:5c:fd:81:4f:0a	expired
https://crt.sh/?id=12587878558	2024.04.03 14:34:34 UTC	C=HU, CN=edvtlsca2023-revoked.e-szigno.hu	04:a8:da:71:1f:d7:27:d3:d0:ba:f1:7e:c6:0a	2024-04-03 14:35:09 UTC
https://crt.sh/?id=12587909087	2024.04.03 14:25:52 UTC	C=HU, CN=edvtlsca2023-valid.e-szigno.hu	04:a8:d2:05:a5:2f:d7:07:02:48:3e:01:91:0a	valid
https://crt.sh/?id=12587810999	2024.04.03 14:25:52 UTC	C=HU, CN=edvtlsca2023-valid.e-szigno.hu	04:a8:d2:05:a5:2f:d7:07:02:48:3e:01:91:0a	valid
https://crt.sh/?id=12577744401	2024.04.02	C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560	04:a8:34:87:6c:29:20:c7:e7:65:5a:20:c4:0a	2024-04-04 13:33:03 UTC
https://crt.sh/?id=12577607644	2024.04.02	C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560	04:a8:34:87:6c:29:20:c7:e7:65:5a:20:c4:0a	2024-04-04 13:33:03 UTC
https://crt.sh/?id=12515508437	2024.03.27	C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560	04:a7:02:a2:cf:5d:ed:8f:cb:5a:37:41:2c:0a	2024-04-04 14:08:02 UTC
https://crt.sh/?id=12498961907	2024.03.27	C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560	04:a7:02:a2:cf:5d:ed:8f:cb:5a:37:41:2c:0a	2024-04-04 14:08:02 UTC
https://crt.sh/?id=12514262556	2024.03.27	C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560	04:a6:ac:8f:1b:62:af:08:26:22:e1:dd:c6:0a	2024-04-04 14:08:05 UTC
https://crt.sh/?id=12496767101	2024.03.27	C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560	04:a6:ac:8f:1b:62:af:08:26:22:e1:dd:c6:0a	2024-04-04 14:08:05 UTC
https://crt.sh/?id=12520787537	2024.03.27	C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560	04:a6:a9:58:80:ce:e9:19:88:11:4a:21:58:0a	2024-04-04 14:03:02 UTC
https://crt.sh/?id=12513919538	2024.03.27	C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560	04:a6:a3:62:5b:99:dc:e4:1b:4c:68:a7:0b:0a	2024-04-04 14:03:06 UTC
https://crt.sh/?id=12513834167	2024.03.27	C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560	04:a6:9a:6b:13:22:a9:eb:63:fc:e6:79:fb:0a	2024-04-04 14:03:09 UTC
https://crt.sh/?id=12513827024	2024.03.27	C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560	04:a6:97:da:92:97:9e:5c:19:5b:87:b0:c2:0a	2024-04-04 13:53:03 UTC
https://crt.sh/?id=12508856594	2024.03.26	C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560	04:a6:81:0f:1c:0f:c6:df:d3:65:89:a6:d3:0a	2024-04-04 13:53:06 UTC
https://crt.sh/?id=12508815531	2024.03.26	C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560	04:a6:7c:d8:94:70:d5:2d:7e:3e:90:64:a2:0a	2024-04-04 13:53:09 UTC
https://crt.sh/?id=12495568331	2024.03.26	C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560	04:a6:7c:d8:94:70:d5:2d:7e:3e:90:64:a2:0a	2024-04-04 13:53:09 UTC
https://crt.sh/?id=12291399508	2024.03.06	C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560	04:9e:e2:ed:bc:a6:92:e0:0a:5e:30:72:eb:0a	2024-04-04 13:38:03 UTC
https://crt.sh/?id=12291337592	2024.03.06	C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560	04:9e:e2:ed:bc:a6:92:e0:0a:5e:30:72:eb:0a	2024-04-04 13:38:03 UTC
https://crt.sh/?id=12291093513	2024.03.06	C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560	04:9e:db:5c:c1:98:1c:cf:48:04:a6:fc:4d:0a	2024-03-06 09:34:22 UTC
https://crt.sh/?id=12290628172	2024.03.06	C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560	04:9e:cb:c8:91:8f:0a:a7:bc:9b:e1:d0:67:0a	2024-03-06 09:33:51 UTC
https://crt.sh/?id=11693263276	2024.01.08	C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560	04:8d:48:f5:da:33:8c:18:f9:7f:47:ab:35:0a	2024-04-03 14:46:59 UTC
https://crt.sh/?id=11680562040	2024.01.08	C=HU, CN=edvtlsca2023-valid.e-szigno.hu, serialNumber=1.3.6.1.4.1.21528.2.3.2.3560	04:8d:48:f5:da:33:8c:18:f9:7f:47:ab:35:0a	2024-04-03 14:46:59 UTC

https://crt.sh/?sha256=[sha256 fingerprint of the certificate]

Based on Incident Reporting Template v. 2.0

Comment 1

[Mathew Hodson]

(In reply to dr. Sándor SZŐKE from comment #0)

Impact

The misissued DV certificates contain the SerialNumber (2.5.4.5) extension, which is not allowed in DV certificates.
The presence of this extra field has no impact on the usability or security of the certificate, but it unnecessarily increases the size of the certificate.

CCADB is clear what information belongs in the Impact section. It's not an opportunity to downplay the importance of an incident. It should contain the important information upfront for a reader to asses the size and scope of misissuance. From CCADB:

The Impact section should contain a short description of the size and nature of the incident. For example: how many certificates, OCSP responses, or CRLs were affected; whether the affected objects share features (such as issuance time, signature algorithm, or validation type); and whether the CA Owner had to cease issuance during the incident.

These questions were not answered in the report:

• How many certificates were misissued?
• When was the first misissuance and the start of the incident?
• Did Microsec cease issuance during the incident?

Additionally the timeline included doesn't meet expectations. It should include events from before Microsec became aware of the issue (e.g. certificate issuance, policy reviews, software changes, policies coming into effect). From CCADB:

The Timeline section must include a detailed timeline of all events and actions leading up to and taken during and after the incident. The timeline must include not just the actual discovery of the incident and subsequent events, but also relevant events occuring beforehand (e.g. something changed or was introduced).

Comment 2

[dr. Sándor SZŐKE]

Thank you for your comments, we collect the requested information and respond soon.

Comment 3

[Ryan Dickson]

Hi Sándor,

Section 4.9.1.1 of the TLS BRs states the following:

With the exception of Short-lived Subscriber Certificates, the CA SHOULD revoke a certificate within 24 hours and MUST revoke a Certificate within 5 days and use the corresponding CRLReason (see Section 7.2.2) if one or more of the following occurs:
...
...
12. The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement (CRLReason #4, superseded);
...
...

I noticed a few of these certificates were revoked with a reason of cessationOfOperation (e.g., https://crt.sh/?id=12577744401, https://crt.sh/?id=12577607644, and https://crt.sh/?id=12515508437). Can you help us understand this discrepancy and how it should be considered against the language found in BRs (above) and the corresponding CPS?

Thanks,
Ryan

Comment 4

[dr. Sándor SZŐKE]

Hi Ryan,

I checked the revoked certificates and unfortunately you are rigth, 10 misissued certificates were revoked with a reason code of cessationOfOperation yesterday afternoon.

We started the investigation for this problem.

Comment 5

[dr. Sándor SZŐKE]

Attachment: 20240403--Misissued_DV_certificates_Microsec.xlsx

Comment 6

[dr. Sándor SZŐKE]

Incident Status Report 2024-04-06

Impact - further information

• In the Appendix of the incident report we provided a list of misissued certificates. We added more information to that list and loaded a detailed excel table with the relevant information to this ticket
• Microsec issued faulyt certificates for 1 user domain and 3 Microsec domains used for test websites
• the total number of misissued certificates is 23 and there were 4 precertificates whithout leaf certificates issued
• the incident did not affect the issuance of the OCSP responses and CRLs
• after receiving the notification about the misissuance, the problem was resolved within half a day and during that time there was no further DV certificate issuance

Timeline - further information

2023-04-11

• Baseline Requirements for the Issuance and Management of Publicly Trusted Certificates Version 2.0.0 was published
  • section 7. CERTIFICATE, CRL, AND OCSP PROFILES has been totally rewritten with several new requirements
  • the use of the SerialNumber extension (2.5.4.5) in DV certificates was not previously prohibited, but according to the new requirements, the use of this extension is
    • NOT RECOMMENDED in case of IV and OV certificates
    • not allowed in case of DV certificates
    • still mandatory in case of EV certificates according to EV Guidelines
  • the deadline to update the certificate profiles was 2023-09-15

2023-08-30

• version 3.7 of our CP and CPS were published with the following main changes
  • Global revision
  • Revocation reasons
  • Certificate validity times
  • Key Usage
  • Identity validation by a third party
  • Managing security incidents
  • Supported cryptographic algorithms
  • New, dedicated TLS hierachy
• the use of SerialNumber extension is still mandatory for DV, IV and OV certificates in our CP and CPS
• it was our fault not to recognize the change in the CABF BR requirement for DV certificates

2023-10-30

• version 3.10 of our CP and CPS were published with the following main changes
  • Identity validation of natural persons
  • CA hierarchies
• the use of SerialNumber extension is still mandatory for DV, IV and OV certificates

2023-12-22

• first misissued certificate for www.taxipay.hu domain

2023-12-19

• version 3.11 of our CP and CPS were published with the following main changes
  • Revision, some small changes
• the use of SerialNumber extension is still mandatory for DV, IV and OV certificates

2024-04-03

• first notification email received about the issue
• version 3.12 of our CP and CPS were published with the following main changes
  • Removing SerialNumber extension
• certificate policies upgraded according to the CABF BR requirement

2024-04-04

• This incident report was opened in Bugzilla
• Comment in this Bugzilla ticket from Mathew Hodson requesting more information

2024-04-05

• Comment in this Bugzilla ticket from Ryan Dickson indicating the use of an incorrect revocation reason code in some cases
  • Microsec realized that during the revocation of the misissued certificates on 2024-04-04 afternoon, not the correct revocation reason code was set
  • the fault affected only the domain edvtlsca2023-valid.e-szigno.hu, which is used for our test site in accordance with the Root Program Requirements
  • the revocation for this test domain was made manually and the operator set an incorrect reason code value
  • to solve the issue, Microsec decided to retire the affected key
    • the valid certificate was revoked with reason code keyCompromise
    • the affected revocation reason codes were upgraded from cessationOfOperation to keyCompromise
    • new key was generated for that test site
    • new certificate was issued for that test site
  • currently we have only one revoked DV certificate with reason code cessationOfOperation, it belongs to another test site from edvtlsca2023-revoked.e-szigno.hu
    • this revocation happend only for test purposes, so we left this case as it is

2024-04-06

• we collected all our DV certificates issued after 2023-09-15
  • from crt.sh
  • from our internal certificate database
• we merged the 2 lists and uploaded the result to this Bugzilla ticket
• we also collected the requested additional information and upload it to Bugzilla as part of this status report

Action Items

Action Item	Kind	Due Date	Status
Issuing new certificates using the corrected certificate profiles	Repair	2024-04-04	Done
Contacting all involved Subscribers and informing them of the necessary measures	Repair	2024-04-04	Done
Revoking the misissued certificates	Repair	2024-04-05	Done

Appendix

Details of affected certificates uploaded in an excel file

Comment 7

[dr. Sándor SZŐKE]

Incident Status Report - 2024-04-09

Timeline

2024-04-09

• 2 PSD2 certificates revoked, see in Bug#2: https://bugzilla.mozilla.org/show_bug.cgi?id=1887110
• revocation of all misissued certificates finished

Action Items

Action Item	Kind	Due Date	Status
Issuing new certificates using the corrected certificate profiles	Repair	2024-03-20	Done
Contacting all involved Subscribers and informing them of the necessary measures	Repair	2024-03-20	Done
Revoking 44 of 46 misissued certificates	Repair	2024-03-24	Done
Revoking 2 misissued PSD2 certificates	Repair	2024-04-09	Done
Supervision of our practice in case of changes in requirements	Prevent	2024-04-30	Started

Appendix

Details of affected certificates

Misissued certificate (or precertificate)	Revocation time / OCSP answer
https://crt.sh/?id=10727629590	2024-04-09 12:08:03 UTC
https://crt.sh/?id=12240328933	2024-04-09 12:58:03 UTC

Comment 8

[dr. Sándor SZŐKE]

Please ignore my previous comment, it belongs to another bug.