Sectigo: Certificate issuance delayed for more than 398 days after DCV was completed
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: martijn.katerbarg, Assigned: martijn.katerbarg)
Details
(Whiteboard: [ca-compliance] [ev-misissuance])
Attachments
(2 files)
1. How your CA first became aware of the problem
We received an email from our auditor inquiring about the date Domain Control Validation (DCV) was performed on a sample certificate.
2. Timeline
January 10, 2020 – 08:52 UTC
We receive an order for an EV certificate for presentephemere.com.
January 10, 2020 – 08:55:26 UTC
DCV is performed and marked as completed in our systems.
January 10, 2020 – 12:25 UTC
We start the EV vetting process for this order.
January 10, 2020 – 12:34 UTC
The vetting agent requires additional information from the Applicant, and an email is sent to the Applicant. After this email is sent, the vetting agent moves on to processing a different order.
July 10, 2020 – 03:00 UTC
Our automated systems put the order in a “SUSPENDED” state due to 6 months of inactivity.
May 24, 2022 – 08:53 UTC
We receive an email from the Applicant regarding this order asking for us to continue processing. We take the order out of the “SUSPENDED” state, so that the Subject Identity Information validation process may occur.
May 24, 2022 – 09:35:50 UTC
Our vetting team concludes the vetting and second approval process for this order. Our system conducts the pre-issuance CAA check, pre-issuance linters are run on the tbsCertificate, and the precertificate and final certificate are issued and submitted to CT logs.
April 17, 2023 – 14:13 UTC
We receive an email from our auditor asking to confirm the DCV timestamp we disclosed for this order during the audit process.
April 17, 2023 – 15:18 UTC
We start reviewing the order in question, and preliminary investigation shows that the DCV timestamp is more than 398 days prior to the issuance of the certificate. We schedule a revocation of the certificate.
April 18, 2023 – 10:11 UTC
We get confirmation that our preliminary investigation is correct and confirm a misissuance.
April 18, 2023 – 10:55 UTC
We request a report from our DBA team to identify if there are any additional certificates with a similar issue. Generation of this report is expected to take several days.
April 18, 2023 – 13:01:14 UTC
The certificate is revoked.
April 18, 2023 – 14:00 UTC
During our twice-weekly WebPKI Incident Response (WIR) team call, we further discuss this incident and potential remediation steps. As we have a deployment window coming up within a few days, we decide to add an emergency fix to the planned deployment, which will remediate the issue.
April 19, 2023 – 11:05 UTC
We complete development of our remediation changes.
April 19, 2023 – 14:48 UTC
We deploy the patch on our QA systems. QA testing starts.
April 20, 2023 – 19:22 UTC
We complete QA testing.
April 22, 2023 – 23:36 UTC
We deploy the patch to our production systems, remediating the issue.
3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem.
We have stopped issuance of certificates with this problem.
4. Summary of the problematic certificates
At present, 1 certificate has been identified and revoked. We are investigating if more certificates are affected and will confirm once this investigation has been concluded.
5. Affected certificates
| Serial Number | Certificate | Precertificate |
|---|---|---|
| 7628C95404EE7AC99AD04B1A6424B095 | Certificate | Precertificate |
6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now
Most certificate requests get issued within a short time window, resulting in only a very small portion of requests that linger for a long period of time. In general, these are cases where we’ve asked customers for additional information or confirmation of details and not heard back from them. As one can imagine, the longer it has been pending, the smaller the chance of a customer coming back to us to continue with the vetting process. This is why we automatically move any order that is older than 6 months into the “SUSPENDED” state.
We want to make it clear that this is not a case of DCV reuse, which is where DCV performed for one certificate request is later relied on for further certificate request(s) for the same domain, and which has been the topic of previous Sectigo incident reports within the last 2 years.
Rather, this is a case of delayed issuance, where DCV was performed for the specific request but outside of the allowed timeframe. Due to the unlikelihood of a customer coming back to us after more than a year in order to proceed with an old certificate request, neither our CA implementation nor our DCV Review had dealt with this delayed issuance scenario.
When a vetting agent moves a certificate from the “SUSPENDED” state back to the “ACTIVE” state so that they can process a delayed request, they do not review anything to do with DCV, since our DCV processes are entirely automated.
7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future
We have developed, reviewed, and deployed a pre-issuance check that will make sure that the completed DCV is still sufficiently fresh at the moment of issuance, remediating this issue.
|
||
Updated•1 year ago
|
|
|
||
Comment 1•1 year ago
|
||
As part of our response to this incident, we reviewed the relevant language in BR 4.2.1:
"The CA MAY use the documents and data provided in Section 3.2 to verify certificate information, or may reuse previous validations themselves, provided that the CA obtained the data or document from a source specified under Section 3.2 or completed the validation itself no more than 825 days prior to issuing the Certificate. For validation of Domain Names and IP Addresses according to Section 3.2.2.4 and 3.2.2.5, any reused data, document, or completed validation MUST be obtained no more than 398 days prior to issuing the Certificate." (emphasis mine)
The 825 day requirement clearly applies to both the initial Use and any subsequent Reuse of data, documents, and validations; however, we noticed that the stricter 398 day requirement only applies to Reuse - that is, it seems to us that the most natural reading of the last part of this language is "...any reused data, [reused] document, or [reused] completed validation MUST be obtained no more than 398 days prior to issuing the Certificate." Taken to its logical conclusion, this would mean that the BRs impose only an 825 day deadline on the initial Use of a completed DCV check. We believe that this is a loophole in the BRs rather than the intended requirement, and we intend to raise this matter at CABForum and to propose clarifying text. We mention it here because we found it interesting that the BRs currently have the same blind spot that we had in relation to delayed issuance.
When we looked at the full set of applicable policy requirements, we found that this BR loophole is a moot point for Sectigo’s CA operations due to MRSP 2.1, which specifies a 398 day requirement that encompasses both initial Use and subsequent Reuse of completed DCV checks:
"5.1. for server certificates issued on or after October 1, 2021, each dNSName or IPAddress in a SAN or commonName MUST have been validated in accordance with section 3.2.2 of the CA/Browser Forum's Baseline Requirements within the preceding 398 days;"
|
Assignee | |
Comment 2•1 year ago
|
||
Our investigation into this matter continues, looking at certificates issued in the past.
Due to the size of the certificate base, generating the reports is a time-consuming task. Once we have received and reviewed the reports, we will provide an update.
|
|
Assignee | |
Comment 3•1 year ago
|
||
On May 3, 2023 – 16:19 UTC we confirmed findings on the first generated report looking at our existing certificate base, confirming that 21 EV certificates are affected by the details outlined in this bug.
We sent out notifications to Subscribers and scheduled a revocation event for May 4, 2023 – 14:00 UTC.
The following certificates were included in this revocation event:
Revocation of these certificates was completed by May 4, 2023 – 14:10 UTC.
We continue to run several reports and will update the community with further results.
|
|
Assignee | |
Comment 4•1 year ago
|
||
|
|
Assignee | |
Comment 5•1 year ago
|
||
On May 9, 2023 – 13:41 UTC we confirmed findings on the second generated report looking at our existing certificate base, confirming 168 affected OV and DV certificates.
We sent out notifications to Subscribers and scheduled a revocation event for May 10, 2023 – 11:00 UTC.
Revocation of these certificates was completed by May 10, 2023 – 11:22 UTC. All revoked certificates are listed in attachment 9333084 [details].
Additional reports are being run which may yield further results.
|
|
Assignee | |
Comment 6•11 months ago
|
||
Our final reports are still running. We expect to be in a position to present the final results by this time next week.
|
|
Assignee | |
Comment 7•11 months ago
|
||
|
|
Assignee | |
Comment 8•11 months ago
|
||
Yesterday, May 24, 2023 at 16:11 UTC we received and confirmed the final reports resulting in a final 45 additional certificates affected by the details outlined in this bug.
We sent out notifications to Subscribers and scheduled a revocation event for May 25, 2023 – 14:10 UTC. The affected certificates are listed in attachment 9335973 [details].
Revocation of these certificates was completed by May 25, 2023 – 13:28 UTC.
This revocation completes our investigation into this incident. Remediation was completed on April 22, 2023, as outlined in comment 0.
|
|
Assignee | |
Comment 9•11 months ago
|
||
We have completed investigation and remediation of this incident. It appears there are no further questions or comments. As such we would like to request closing this bug.
|
||
Comment 10•11 months ago
|
||
I will close this on or about Friday, 2-June-2023, unless there are questions or issues to discuss further.
|
|
||
Updated•11 months ago
|
Description
•