SSL.com: S/MIME certificates issued prior to validation
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: secauditor, Assigned: secauditor)
Details
(Whiteboard: [ca-compliance] [smime-misissuance])
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/116.0
Steps to reproduce:
This is a preliminary incident report concerning the issuance of 9 S/MIME certificates before the validation process was completed.
Actual results:
- How your CA first became aware of the problem (e.g., via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP or CCADB public mailing list, a Bugzilla bug, or internal self-audit), and the time and date.
An internal ticket was generated by the Validation team, who noticed the issuance of the certificates while they were performing their normal validation tasks.
-
A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a requirement became applicable, a document changed, a bug was introduced, or an audit was performed.
2023-05-01 Update of our RA Portal to integrate human review/approval of identity validations performed by software (automated validations)
2023-07-03 S/MIME IV+OV certificate request no.1 by a customer
2023-08-17 S/MIME IV+OV certificate request no.2 by a customer
2023-08-17 12:22:22 CDT - First automated IV attempt for certificate request no.2
2023-08-17 12:25 CDT - Automated IV for certificate request no.2 successfully completed
2023-08-17 12:37 CDT - Certificate no.2 issued with Serial: 757eb8b9a3fdae5e8a453e9dde58dc76.
2023-08-18 19:08:57 CDT - First automated IV attempt for certificate request no.1
2023-08-21 01:19 CDT - Automated IV for certificate request no.1 successfully completed
2023-08-21 01:42 CDT - Certificate no.1 issued with Serial: 33512dde56ccc96815eb779324b6251c.
2023-08-21 03:00 CDT - Validation team noticed that Certificate no.2 was issued before the completion of the OV step. A review of other orders began to understand the issue, confirming that only S/MIME IV+OV orders are affected.
2023-08-21 04:10 CDT - Validation team notified Software Engineering and Compliance about the issue via an internal ticket. Instructions are given to the Validation Specialists to pause processing new S/MIME IV+OV certificate orders until a bug fix was deployed.
2023-08-21 09:30 CDT - Software Engineering identified the offending code and deployed an emergency fix to ensure OV are completed, if they are required, before certificate generation email is sent to the customer.
2023-08-21 09:32 CDT - Compliance registers a Security Event ticket to manage the issue in accordance with our Incident Management Policy.
2023-08-21 15:25 CDT - Per request by Compliance, an investigation of all S/MIME IV+OV certificates issued since 2023-05-01 was initiated to produce the population of all affected certificates (retrospection).
2023-08-22 09:45 CDT - Compliance completes initial analysis and formally declares a Security Incident. Information is requested from all teams involved.
2023-08-22 10:56 CDT - Retrospection completed; 7 additional certificates were found to be affected, resulting in a population of 9 affected certificates in total.
2023-08-22 11:15 CDT - Software Engineering deployed an additional fix that prevents the re-issuance/replacement of the affected certificates.
2023-08-23 3:44 CDT – Completed revocation of all affected certificates.
2023-08-23 - Compliance started compiling the preliminary incident report.
2023-08-25 - Per review of the applicable Root Store Policies, Compliance updated the internal procedure for the disclosure of incidents related to S/MIME certificates.
2023-08-25 - Filed preliminary incident report in Bugzilla (this document).
-
Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.
The issue has been remediated and no such mis-issuances are possible since 2023-08-22 (see timeline).
In particular:
All teams involved (Compliance, Validation, Software Engineering) were immediately informed upon discovery of the issue. The validation team was immediately instructed to pause processing new S/MIME IV+OV orders until a bug fix was deployed.
A code check was performed once the issue was identified, and a fix was deployed to ensure no new mis-issuances were possible. A subsequent update prevented re-issuance/replacement of S/MIME IV+OV certificates that were issued during the incident (see timeline).
- In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g., OCSP failures, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help measure the severity of each problem.
Nine (9) certificates were affected by this issue, all of them S/MIME IV+OV, issued between 2023-05-22 and 2023-08-22.
The issue was that due to a bug which was introduced in the code, once the IV part was completed and approved by both parties (SSL.com and customer), the email with the link to generate the certificate was being sent to the applicant regardless of any OV validation. This only affected S/MIME IV+OV certificates, and in particular their OV portion.
See Section 6 of this report for more details about the issue.
- Certificate data for the problematic certificates
- Serial Number - SHA256 hash - Not Before
1 - 640c633d7debb96ecc38e14ccaa8977a - 3A:8A:8F:99:3E:94:80:2C:08:40:4E:D8:CC:E7:A0:FD:54:3C:7E:D6:24:73:AD:4F:A6:B8:88:E8:B4:9D:B2:E7 - May 22 17:32:19 2023 GMT
2 - 5015e341c7f7f02637ff11c3a0dbbb19 - E7:47:38:83:25:D3:87:BB:92:19:04:B9:13:E7:07:BB:59:AA:39:5D:9E:5B:04:A0:00:70:B6:F1:B1:08:5F:4C - May 22 17:42:03 2023 GMT
3 - 2006643e5b6718d5fbce521ce6fc06c9 -
C4:C1:A0:F4:61:1C:96:61:CC:86:9D:51:03:4D:2E:B5:00:1B:CA:1B:DC:D6:AE:DB:93:1D:ED:C8:C9:11:88:DC - Jun 22 16:29:17 2023 GMT
4 - 7c847ab6f439a181e3bb8955286abf2f - 81:FD:EF:C1:F7:11:A8:68:24:E6:45:F4:A5:BE:BC:50:B6:C2:FA:3F:A3:3F:F8:F5:9A:C8:56:13:17:50:EF:CD - Jul 31 09:18:28 2023 GMT
5 - 48dd42e8ad7dbf131dd0f50c505ebdf8 - BC:70:2C:6C:19:11:AF:5A:7C:E8:9D:DC:32:80:AC:47:79:44:7D:49:98:D8:D0:FE:A4:11:E2:D4:6A:4B:E3:E0 - Jul 31 22:57:00 2023 GMT
6 - 44a57684e9d8d4ce516eaefb590bf8ae - 9A:0D:FD:F4:8E:57:3F:58:99:C8:7B:F4:64:E7:3C:5A:3E:AC:1A:4E:0B:C5:E0:4C:01:A5:D3:BF:23:C8:24:66 - Aug 13 18:12:22 2023 GMT
7 - 757eb8b9a3fdae5e8a453e9dde58dc76 7F:DF:E7:E9:FB:D2:C2:78:84:61:B2:E3:47:AF:C7:95:93:C8:17:3A:E1:2E:3D:81:60:DE:22:14:D6:E4:38:7B - Aug 17 17:27:04 2023 GMT
8 - 33512dde56ccc96815eb779324b6251c - BF:2F:E8:C3:34:5D:36:D9:72:99:BD:7D:A1:D7:3E:F3:CB:8B:72:AC:B4:58:EC:5F:4C:7F:68:50:EA:19:E9:A2 - Aug 21 06:32:32 2023 GMT
9 - 5b7c9918121dc41cc14b0febb392885a - 9D:6E:35:FA:75:82:29:7A:20:8C:27:53:1A:24:9D:3D:3B:AC:A3:AA:43:88:1A:AB:2B:A3:8C:76:4D:2D:58:30 - Aug 22 00:10:31 2023 GMT
- Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
The code that introduced the bug was a large change focused on integrating manual approval into an IV process. The bug was introduced when IV+OV certificates were included in the process. There was an oversight with the changes that were required to include IV+OV certificates. Specifically, a random token that was sent at the completion of the IV process, was not accounting for the validation of the organization.
This bug avoided detection due to the low volume of these types of certificates. The certificate was allowed to be issued, which removed the order from the validation process and out of view of the validation team.
This section will be updated based on the results of the ongoing investigation (see section 7 below).
- List of steps your CA is taking to resolve the situation and ensure that such a situation or incident will not be repeated in the future. The steps should include the action(s) for resolving the issue, the status of each action, and the date each action will be completed
The development team has already identified the cause of the issue and has rectified the problem.
In accordance with our Incident Management Policy, a root cause analysis shall take place by our internal audit team after completion of the investigation, in order to identify any underlying issues and introduce any remediating measures needed.
We will update this bug next week with the findings of our investigation.
Expected results:
Ideally, the certificate generation email would not have been sent until after the completion of the validation process and approval of all supporting evidence by a Validation Specialist.
|
||
Updated•2 years ago
|
|
|
||
Updated•2 years ago
|
Currently there is no new information to report. However, our investigation into this issue is ongoing, and we will provide another update by the end of next week.
A Root Cause Analysis has been performed by our development team and a preliminary report has been filed internally. The ultimate cause for the incident was a reliance on a single developer to perform the acceptance testing on the code. After this code was introduced, but prior to the discovery of the issue, an internal review within the development team led to the correction of this process, and we now require 2 or more developers to perform acceptance testing on new code.
|
|
||
Comment 3•2 years ago
|
||
Hi Thomas,
Can you let us know if/when this issue has been fully rectified to prevent this type of misissuance from reoccurring?
Thanks,
Ben
Hi Ben,
The issue has been fully rectified by our software engineering team. The root cause has been addressed and preventative measures have been fully implemented.
Thank you for inquiring,
Tom
|
|
||
Comment 5•2 years ago
|
||
Good. I plan to close this on Friday, 29-Sept-2023.
|
|
||
Updated•2 years ago
|
Description
•