Closed
Bug 1886257
Opened 11 months ago
Closed 6 months ago
Microsec: Misissuance an EV TLS certificate without CPSuri
Categories
(CA Program :: CA Certificate Compliance, task)
CA Program
CA Certificate Compliance
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: szoke.sandor, Assigned: szoke.sandor)
Details
(Whiteboard: [ca-compliance] [ev-misissuance])
Incident Report
Summary
It was reported by email to info@... , that Microsec misissued an EV certificate.
The problem is that the certificate does not contain the CPSuri link.
Microsec did not react in time, so a second email was sent to info@... and also to the Microsec contact persons CCADB.
Due to the delay, 3 separate incident reports will be created as follows:
- Bug #1 must focus on the certificate misissuance reported in the problem report.
- Bug #2 must focus on the delayed revocation of the misissued certificates described in the problem report.
- Bug #3 must focus on the failure to respond to a certificate problem report in a complete and/or timely manner.
The current bug focuses on the certificate misissuance (Bug 1).
Other issues will be presented in separate bugs.
Impact
The missing CPSuri information has no impact on the usability or security of the certificate, but it makes it more difficult for users to find the policy information.
The misissued certificate is:
https://crt.sh/?id=12302329269
Timeline
2024-03-07
- Microsec received an email to the general purpose email address info@... provided in CCADB reporting a potentially misissued certificate.
- Microsec's OTRS system received the email and sent an automatic notification email with a registration number: Ticket#2024030884001045
2024-03-18
- 13:06 UTC (14:06 CET)
- Microsec received a second email to the general purpose email address info@... provided in CCADB reporting a potentially misissued certificate. This second email was also sent directly to the contact persons of Microsec provided in CCADB.
- Microsec's OTRS system received the email messages and sent an automatic notification email with a registration number: Ticket#2024031884006012
- 15:00 UTC (16:00 CET)
- Sándor Szőke, the main contact person in CCADB, sent a manual notification email to the sender.
- Microsec began the investigation of the issue.
- 15:22 UTC (16:22 CET)
- Microsec opened internal jira tickets for the management of the issue.
- 15:30 UTC (16:30 CET)
- Microsec studied the current CABF BR and EVG requirements and verified that the presence of CPSuri is mandatory in the case of EV TLS certificates.
- Microsec decided to modify the problematic EV TLS certificate profiles and issue a new version of its Certificate Profiles document.
- Microsec realized that the problem starts with version 2023-08-29 of the certificate profiles. Each EV certificate issued after this date shall be considered as misissued and has to be replaced.
- 16:15 UTC (17:15 CET)
- Microsec modified all (7) EV TLS certificate profiles in the open working area.
- 16:40 UTC (17:40 CET)
- A new TAG was created in the SVN configuration management system for the new certificate profiles to be activated.
- 16:45 UTC (17:45 CET)
- The Customer was informed about the changes and the necessary measures. There was no problem with revocing the misissued certificate.
- 17:03 UTC
- A new EV TLS certificate was issued with CPSuri: https://crt.sh/?id=12426691140
- 17:05 UTC
- The misissued certificate was revoked.
- 18:38 UTC (19:38 CET)
- Microsec officially released the new version of the Certificate Profiles document.
2024-03-19
- 10:00 UTC (11:00 CET)
- Microsec discussed the issue at its regular management meeting and developed a detailed action plan.
- Microsec discovered that a total of 45 EV TLS certificates were misissued, including some test certificates. These certificates shall be replaced and revoked within 5 days in accordance with CABF requirements.
-18:10 UCT - This incident report was opened in Bugzilla.
Root Cause Analysis
2023-08-29
- Microsec released a new version of its Certificate Profiles. One of the most important changes was the update of the TLS certificate profiles according to the CABF Baseline Requirements ver 2.2.2. In this CABF BR version, the whole section 7 was overwritten and there were many specific and detailed requirements.
In section 7.1.2.10.5 CA Certificate Certificate Policies Table 68 says that the presence of the policyQualifiers extension is NOT RECOMMENDED.
Based on this requirement, Microsec decided to remove the policyQualifiers information from the TLS certificates. - This was an error on Microsec's part, as the CABF EVG requires this information to be present in EV TLS subscriber certificates.
- Microsec uses two linters before the issuance (certlint and zlint), but none of them could find this problem.
- From this date until 2024-03-18, all issued EV TLS certificates were missing this CPSuri information.
Lessons Learned
We learned that many times there are several different requirements for a given certificate type, which are not necessarily match.
In case of any change in a requirements, we also have to check whether the requested change is consistent with other requirements.
What went well
- Thanks to our well-documented configuration management system, we could quickly find the root of the problem and we could also find all the infected certificates.
What didn't go well
- The problem was that we changed each TLS profile based on the new requirements of the CABF BR and did not check whether this change met the requirements of the CABF EVG or not.
Where we got lucky
- We were lucky because this problem does not affect the intended use and security of the certificates.
Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Issuing new certificates using the corrected certificate profiles | Repair | 2024-03-20 |
| Contacting all involved Subscribers and informing them of the necessary measures | Repair | 2024-03-20 |
| Revoking the misissued certificates | Repair | 2024-03-24 |
| Supervision of our practice in case of changes in requirements | Prevent | 2024-04-30 |
Appendix
Details of affected certificates
There are 45 misissued certificates, the full list will be published soon in the next status report.
https://crt.sh/?sha256=[sha256 fingerprint of the certificate]
Based on Incident Reporting Template v. 2.0
|
||
Updated•11 months ago
|
Assignee: nobody → szoke.sandor
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] [ev-misissuance]
|
Assignee | |
Comment 1•11 months ago
|
||
Incident Status Report - 2024-03-22
Timeline
2024-03-19
- Microsec issued 44 of 45 infected certificates
- One domain was protected by CAA, the customer was asked to change its domain configuration
2024-03-20
- Microsec contacted each affected customers
2024-03-21
- One PSD2 service provider reported that it is not possible to replace the certificate in the whole network within 5 days and asked the extension of the deadline
- Microsec contacted the Root Program operators asking for extension for this specific customer
- Questions regardind deadline will be managed in separate Bug #2 to be opened soon
2024-03-22
- Problematic customer changed the CAA setting on its domain
- Microsec issued the missing certificate
- List of affected certificates created with crt.sh links
- Bug#3 incident report created: https://bugzilla.mozilla.org/show_bug.cgi?id=1886998
- Bug#1 status report with the certificate list
Action Items
| Action Item | Kind | Due Date | Status |
|---|---|---|---|
| Issuing new certificates using the corrected certificate profiles | Repair | 2024-03-20 | Done |
| Contacting all involved Subscribers and informing them of the necessary measures | Repair | 2024-03-20 | Done |
| Revoking the misissued certificates | Repair | 2024-03-24 | Scheduled |
| Supervision of our practice in case of changes in requirements | Prevent | 2024-04-30 | Planned |
Appendix
Details of affected certificates
|
|
Assignee | |
Comment 2•11 months ago
|
||
Incident Status Report - 2024-04-05
Timeline
2024-03-24
- 44 misissued certificates revoked latest that day
- 2 PSD2 certificates will be revoked later, see in Bug#2: https://bugzilla.mozilla.org/show_bug.cgi?id=1887110
Action Items
| Action Item | Kind | Due Date | Status |
|---|---|---|---|
| Issuing new certificates using the corrected certificate profiles | Repair | 2024-03-20 | Done |
| Contacting all involved Subscribers and informing them of the necessary measures | Repair | 2024-03-20 | Done |
| Revoking the misissued certificates | Repair | 2024-03-24 | Done |
| Supervision of our practice in case of changes in requirements | Prevent | 2024-04-30 | Planned |
Appendix
Details of affected certificates
Based on Incident Reporting Template v. 2.0
|
|
Assignee | |
Comment 3•10 months ago
|
||
Incident Status Report - 2024-04-09
Timeline
2024-04-09
- 2 PSD2 certificates revoked, see in Bug#2: https://bugzilla.mozilla.org/show_bug.cgi?id=1887110
- revocation of all misissued certificates finished
Action Items
| Action Item | Kind | Due Date | Status |
|---|---|---|---|
| Issuing new certificates using the corrected certificate profiles | Repair | 2024-03-20 | Done |
| Contacting all involved Subscribers and informing them of the necessary measures | Repair | 2024-03-20 | Done |
| Revoking 44 of 46 misissued certificates | Repair | 2024-03-24 | Done |
| Revoking 2 misissued PSD2 certificates | Repair | 2024-04-09 | Done |
| Supervision of our practice in case of changes in requirements | Prevent | 2024-04-30 | Started |
Appendix
Details of affected certificates
| Misissued certificate (or precertificate) | Revocation time / OCSP answer |
|---|---|
| https://crt.sh/?id=10727629590 | 2024-04-09 12:08:03 UTC |
| https://crt.sh/?id=12240328933 | 2024-04-09 12:58:03 UTC |
|
|
Assignee | |
Comment 4•10 months ago
|
||
Incident Status Report - 2024-04-24
Timeline
2024-04-24
- based on Wayne's recent post we give the serials of the affected certificates to be easier to check the revocation status on CRL
Action Items
| Action Item | Kind | Due Date | Status |
|---|---|---|---|
| Issuing new certificates using the corrected certificate profiles | Repair | 2024-03-20 | Done |
| Contacting all involved Subscribers and informing them of the necessary measures | Repair | 2024-03-20 | Done |
| Revoking 44 of 46 misissued certificates | Repair | 2024-03-24 | Done |
| Revoking 2 misissued PSD2 certificates | Repair | 2024-04-09 | Done |
| Supervision of our practice in case of changes in requirements | Prevent | 2024-04-30 | In progress |
Appendix
Details of affected certificates
| Misissued certificate (or precertificate) | CA | serial | Revocation time | Revocation reason |
|---|---|---|---|---|
| https://crt.sh/?id=10295679628 | Qualified e-Szigno TLS CA 2018 | 0256ab4375cf2548df356ec909 | 2024-03-24 13:29:25 UTC | Revoked (superseded) |
| https://crt.sh/?id=10310939595 | Qualified e-Szigno TLS CA 2018 | 0256f93386cd2bce64857cd8ac0a | 2024-03-24 13:31:56 UTC | Revoked (superseded) |
| https://crt.sh/?id=10414730504 | Qualified e-Szigno TLS CA 2018 | 025808dff06ee65be4386b3f8f0a | 2024-03-24 13:26:47 UTC | Revoked (superseded) |
| https://crt.sh/?id=10616175576 | Qualified e-Szigno TLS CA 2018 | 025a0ac1b55bf54c69f1959f180a | 2024-03-24 13:08:44 UTC | Revoked (superseded) |
| https://crt.sh/?id=10727629590 | Qualified e-Szigno TLS CA 2018 | 025ae29b4f3e1ccde97563ab09 | 2024-04-09 12:08:03 UTC | Revoked (superseded) |
| https://crt.sh/?id=10832687312 | Qualified e-Szigno TLS CA 2018 | 025bf5f6428d28086266953c5f0a | 2024-03-24 13:09:26 UTC | Revoked (superseded) |
| https://crt.sh/?id=10842169527 | Qualified e-Szigno TLS CA 2018 | 025c1e7637c7861a15f5ddcb380a | 2024-03-24 13:31:12 UTC | Revoked (superseded) |
| https://crt.sh/?id=10883139748 | Qualified e-Szigno TLS CA 2018 | 025c57f3fe6b483a42ea61019c0a | 2024-03-20 08:09:58 UTC | Revoked (superseded) |
| https://crt.sh/?id=11036887243 | Qualified e-Szigno TLS CA 2018 | 025d89455d24e1c36e270ee3e00a | 2024-03-24 13:12:01 UTC | Revoked (superseded) |
| https://crt.sh/?id=11045293257 | Qualified e-Szigno TLS CA 2018 | 025d95ca11b31ec30ad223bf09 | 2024-03-24 13:30:37 UTC | Revoked (superseded) |
| https://crt.sh/?id=11056313661 | Qualified e-Szigno TLS CA 2018 | 025dc08a50a778b121adc58ddd0a | 2024-03-24 13:12:45 UTC | Revoked (superseded) |
| https://crt.sh/?id=11116868963 | e-Szigno Qualified TLS CA 2018 | 025f0c729119fb926af70d79ff0a | 2024-03-24 13:10:47 UTC | Revoked (superseded) |
| https://crt.sh/?id=11117329791 | e-Szigno Qualified TLS CA 2018 | 025f11d1e3d759bfa4ded4c8890a | 2024-03-24 13:11:24 UTC | Revoked (superseded) |
| https://crt.sh/?id=11165482725 | Qualified e-Szigno TLS CA 2018 | 026021c6b2cd52e672743ba5790a | 2024-03-24 13:10:06 UTC | Revoked (superseded) |
| https://crt.sh/?id=11195680877 | e-Szigno Qualified TLS CA 2018 | 0260a9db7a1768a27d132a2c680a | 2024-03-24 13:28:44 UTC | Revoked (superseded) |
| https://crt.sh/?id=11324522638 | Qualified e-Szigno TLS CA 2018 | 0262ef0872d37de24be93df0980a | 2024-03-24 13:04:54 UTC | Revoked (superseded) |
| https://crt.sh/?id=11324522905 | Qualified e-Szigno TLS CA 2018 | 0262f0bdeedca06a0a1a052ee90a | 2024-03-24 13:05:42 UTC | Revoked (superseded) |
| https://crt.sh/?id=11324536254 | Qualified e-Szigno TLS CA 2018 | 0262f2294ae451e41e092911e10a | 2024-03-24 13:06:30 UTC | Revoked (superseded) |
| https://crt.sh/?id=11324601663 | Qualified e-Szigno TLS CA 2018 | 0262f5eff60cc879bc33474a480a | 2024-03-24 13:07:19 UTC | Revoked (superseded) |
| https://crt.sh/?id=11421879782 | Qualified e-Szigno TLS CA 2018 | 0265a1f7c66544b5ff654d8cde0a | 2024-03-24 13:02:55 UTC | Revoked (superseded) |
| https://crt.sh/?id=11637983162 | Qualified e-Szigno TLS CA 2018 | 026611d66a7f62e4f48b847ded0a | 2024-03-24 13:28:04 UTC | Revoked (superseded) |
| https://crt.sh/?id=11829872636 | Qualified e-Szigno TLS CA 2018 | 02680a43fb257527bc5258b009 | 2024-03-24 13:27:26 UTC | Revoked (superseded) |
| https://crt.sh/?id=11831444455 | e-Szigno Qualified TLS CA 2018 | 02681ba3b6b844a5a6d3f874e60a | 2024-03-24 13:13:24 UTC | Revoked (superseded) |
| https://crt.sh/?id=11905563510 | e-Szigno Qualified TLS CA 2018 | 02695fcfaa90d175c4730ba92e0a | 2024-03-20 08:10:23 UTC | Revoked (superseded) |
| https://crt.sh/?id=12018686671 | Qualified e-Szigno TLS CA 2018 | 026b6c6239b2ac06f77f190c09 | 2024-03-24 13:08:02 UTC | Revoked (superseded) |
| https://crt.sh/?id=12240196909 | e-Szigno Qualified TLS CA 2018 | 026f7cd8ec097404aea4001809 | 2024-03-24 13:30:02 UTC | Revoked (superseded) |
| https://crt.sh/?id=12240328933 | Qualified e-Szigno TLS CA 2018 | 026f7ff68c70ce49a410e7e95490e00c | 2024-04-09 12:58:03 UTC | Revoked (superseded) |
| https://crt.sh/?id=12302329269 | e-Szigno Qualified TLS CA 2018 | 0270676a26e0d2d16eebebd5a80a | 2024-03-18 17:05:01 UTC | Revoked (superseded) |
| https://crt.sh/?id=12302352224 | e-Szigno Qualified TLS CA 2018 | 027068f792c05d21e98fb939f80a | 2024-03-24 13:13:59 UTC | Revoked (superseded) |
| https://crt.sh/?id=12302356668 | e-Szigno Qualified TLS CA 2018 | 0270698fc52c662bf6af5db7580a | 2024-03-24 13:14:36 UTC | Revoked (superseded) |
| https://crt.sh/?id=12314334149 | e-Szigno Qualified TLS CA 2018 | 0270c846c47b53c1e7d9339f190a | 2024-03-24 13:15:17 UTC | Revoked (superseded) |
| https://crt.sh/?id=12314334030 | e-Szigno Qualified TLS CA 2018 | 0270c9b06b5c0268ceff2228890a | 2024-03-24 13:15:57 UTC | Revoked (superseded) |
| https://crt.sh/?id=12314340807 | e-Szigno Qualified TLS CA 2018 | 0270ca5770677f979bd17b79330a | 2024-03-24 13:16:36 UTC | Revoked (superseded) |
| https://crt.sh/?id=12314349359 | e-Szigno Qualified TLS CA 2018 | 0270cc7813888409b7cfad07050a | 2024-03-24 13:17:14 UTC | Revoked (superseded) |
| https://crt.sh/?id=12314353230 | e-Szigno Qualified TLS CA 2018 | 0270cd482bb3eae372424b7ee90a | 2024-03-24 13:17:53 UTC | Revoked (superseded) |
| https://crt.sh/?id=12314353231 | e-Szigno Qualified TLS CA 2018 | 0270ce722926f9dc592d6861510a | 2024-03-24 13:18:26 UTC | Revoked (superseded) |
| https://crt.sh/?id=12314360223 | e-Szigno Qualified TLS CA 2018 | 0270cf238fcb7224c33e956e1b0a | 2024-03-24 13:19:03 UTC | Revoked (superseded) |
| https://crt.sh/?id=12314368545 | e-Szigno Qualified TLS CA 2018 | 0270d0538fbae9f17f43c3e25a0a | 2024-03-24 13:19:44 UTC | Revoked (superseded) |
| https://crt.sh/?id=12314368627 | e-Szigno Qualified TLS CA 2018 | 0270d10743549a78fe4cbed3820a | 2024-03-24 13:20:16 UTC | Revoked (superseded) |
| https://crt.sh/?id=12314376645 | e-Szigno Qualified TLS CA 2018 | 0270d288a3a614fc5af76583010a | 2024-03-24 13:20:53 UTC | Revoked (superseded) |
| https://crt.sh/?id=12314386237 | e-Szigno Qualified TLS CA 2018 | 0270d326cfe16f633a65b90a7f0a | 2024-03-24 13:21:36 UTC | Revoked (superseded) |
| https://crt.sh/?id=12314394907 | e-Szigno Qualified TLS CA 2018 | 0270d476b1db937dc522b857ca0a | 2024-03-24 13:22:11 UTC | Revoked (superseded) |
| https://crt.sh/?id=12314395075 | e-Szigno Qualified TLS CA 2018 | 0270d5a043d429a69e10574bc90a | 2024-03-24 13:22:45 UTC | Revoked (superseded) |
| https://crt.sh/?id=12314402503 | e-Szigno Qualified TLS CA 2018 | 0270d66d7c6dd1d5cf486264e90a | 2024-03-24 13:23:20 UTC | Revoked (superseded) |
| https://crt.sh/?id=12314410487 | e-Szigno Qualified TLS CA 2018 | 0270d7ca0475cce5375091fce60a | 2024-03-24 13:23:55 UTC | Revoked (superseded) |
| https://crt.sh/?id=12314410326 | e-Szigno Qualified TLS CA 2018 | 0270d8a841ee43a51d806633680a | 2024-03-24 13:24:38 UTC | Revoked (superseded) |
|
|
Assignee | |
Comment 5•10 months ago
|
||
Incident Status Report - 2024-04-30
Action item: Supervision of our practice in case of changes in requirements
Certificate profile change management
In order to better understand what happened, we briefly explain how Microsec handles the changes in its certificate profiles.
-
Certificate profiles are editable text files that our CA software uses during issuance. We have more than 150 active user certificate profiles.
Profile files are organized into folders according to which CA group they belong to. -
Certificate profiles are managed in our SVN system.
We make modifications in TRUNKs, and after testing and approving the new version, a TAG marked with time values is created. -
Before new certificate profiles are deployed in a live system, an official certificate profile policy is created by using LATEX, that contains all certificate profiles.
The certificate profile policy is officially approved by the director of the EHSZ.
New profiles will be activated based on the profile TAG information contained in the certificate profile policy document. -
Changes to both the certificate profiles and the profile policy are monitored in our jira system.
-
After publishing a version of our certificate profile policy, we immediately open a further jira epic for the next policy version.
In case of any change request or any issue that arises, we open a jira ticket in this epic where we describe the issue in detail.
Tickets are commented on by members of the EHSZ group, consisting the EHSZ management and employees dedicated to compliance and process control issues.
If necessary, the EHSZ management makes a decision on proposed change. -
When necessary, a new version of the certificate profile policy is published and corresponding changes are made to the certificate profiles.
Based on the importance and urgency of the collected tasks, a decision is made as to which changes will be included in the next release and which will be postponed to later releases.
This process ensures that no any proposed change can be forgotten and that each change is approved by the responsible people.
Root of the problem
-
The problem with the faulty change was that the changes of the different certificate types were handled in one jira task, although different requirements were in force for them.
-
We opened a joint task to remove the UserNotice and CPSuri extensions from all certificate profiles based on the new CABF BR requirement/recommendation.
We made a mistake when we didn't notice that there was a different requirement for EV certificates.
Although we use dual control, none of our compliance team members noticed this issue.
Improvements in our processes
- To prevent similar errors from occurring in the future, we decided to improve our management process in jira by splitting tasks into several subtasks based on different certificate types.
Action Items
| Action Item | Kind | Due Date | Status |
|---|---|---|---|
| Issuing new certificates using the corrected certificate profiles | Repair | 2024-03-20 | Done |
| Contacting all involved Subscribers and informing them of the necessary measures | Repair | 2024-03-20 | Done |
| Revoking 44 of 46 misissued certificates | Repair | 2024-03-24 | Done |
| Revoking 2 misissued PSD2 certificates | Repair | 2024-04-09 | Done |
| Supervision of our practice in case of changes in requirements | Prevent | 2024-04-30 | Done |
|
|
||
Comment 6•6 months ago
|
||
Is there anything remaining to be done regarding this incident report?
Flags: needinfo?(szoke.sandor)
|
|
Assignee | |
Comment 7•6 months ago
|
||
We do not have any open issue regarding this incident report.
Flags: needinfo?(szoke.sandor)
|
|
||
Updated•6 months ago
|
Status: ASSIGNED → RESOLVED
Closed: 6 months ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•