1878139 - Sectigo: Failure to invalidate Email DCV Random Values after 30 days

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Martijn Katerbarg]

Preliminary Incident Report

Summary

Through our ongoing code-review process, we discovered that the Random Values used in our email-based domain control method did not get invalidated after 30 days of being created by our systems.

At 03:00 UTC this morning, February 1st, 2024, we deployed a patch, remediating the possibility of our system considering Random Values “valid for use in a confirming response” beyond the 30 day deadline.

We are currently further investigating this incident, including running reports to identify if any non-compliant certificate issuance has occurred, and ensuring that incorrectly confirmed responses will not be relied on for new certificate issuance.

We will post a full incident report no later than February 9th 2024.

Comment 1

[Martijn Katerbarg]

Attachment: Random_Value_affected_certificates.csv

Comment 2

[Martijn Katerbarg]

Incident Report

Summary

Through our ongoing code-review process, we discovered that the Random Values used in our email-based domain control validation methods did not get invalidated after 30 days of being created by our systems.

Impact

2577 unexpired certificates, issued between 2023-01-08 and 2024-02-01, were issued based on domain control validation that used a Random Value that should no longer have been considered valid.

Timeline

All times are UTC.

2024-01-29:

• 12:10 A member of our product development team reaches out to our compliance team, inquiring if the Random Value generated for email-based DCV methods needs to be invalidated after a certain time. Our compliance team confirms it does.
• 12:16 Further review reveals that when a resend request for a DCV email is received, a new Random Value is generated, and the previous Random Value is correctly invalidated. When no resend is requested, the existing Random Value does not expire after 30 days.
• 12:34 We create a development ticket to implement compliant Random Value expiration.
• 16:44 A member of our QA team asks R&D to confirm that our other DCV methods are unaffected by this bug.
• 17:04 Our R&D team confirms that our other DCV methods that use Random Values are unaffected by this bug.

2024-01-30:

• 12:31 We finalize a patch and request code review.
• 12:51 We deploy the patch to our QA environment.
• 19:58 Our patch is amended with additional code changes based on feedback from our QA team.

2024-01-31:

• 10:12 We deploy the updated patch to our QA environment.
• 10:16 With remediation code complete, we start work on writing scripts and queries to identify any potential misissuance due to this bug.
• 14:04 An emergency deployment is approved pending signoff for pre-deployment testing.
• 14:13 Maintenance window is scheduled for February 1, 2024 from 03:00 to 05:00. Notifications are sent out to customers.
• 16:33 Our QA team completes pre-deployment testing.

2024-02-01:

• 03:00 We start our maintenance window and deploy the code changes.
• 03:20 Deployment is completed, and services return to normal.

2024-02-02:

• Work continues throughout the day on identifying which certificates may have been affected, with queries and scripts being finalized.

2024-02-05:

• 13:04 A list of suspect certificates is sent to the compliance team for investigation and verification.
• 18:16 Out of an abundance of caution, we disable reuse of the domain control validations associated with our list of suspect certificates.

2024-02-06:

• 08:45 The compliance team requests an additional report to help confirm the final list of affected certificates.
• 12:00 The additional report is sent to the compliance team.
• 14:30 We confirm a total of 2577 certificates to have been misissued due to this bug.
• 14:35 We schedule a revocation event for February 7, 2024 at 14:00 UTC.
• 14:42 Our validation and support teams are notified of the upcoming revocation event. We start sending notifications to affected customers.

2024-02-07:

• 14:00 All affected certificates have been revoked.

Root Cause Analysis

CA/B Forum ballot 169, which passed in 2016, added a limitation that each Random Value used for email-based DCV ”SHALL remain valid for use in a confirming response for no more than 30 days from its creation”. This requirement was subsequently removed in ballot 181 and re-added in ballot 190, which both passed in 2017.

Reviewing the available internal communication from around those times has shown us that while these ballots were discussed and it was identified that changes to Comodo’s existing DCV methods would be required, that team’s focus appears to have been on how ballot 169 impacted DCV methods that are not email-based. Unfortunately, none of the individuals responsible for that particular Comodo R&D project are current Sectigo employees, so we only have limited references available on the discussions and processes followed 8 years ago, from which we have not been able to produce any useful Timeline entries. This limited view has somewhat obscured the root cause of this incident. All we can tell at this point is that timely expiration of the Random Value for email-based DCV was not properly implemented. Other DCV methods were not affected by this bug.

Starting with the formation of our WebPKI Incident Response team in 2020, we have vastly improved how we respond to incidents and to changes in compliance requirements from CA/B Forum and the root programs, so going forwards we have high confidence that our current team will not make the kind of mistake that led to this incident.

Code-review of our inherited Certificate Systems, including the DCV components, has been and continues to be an ongoing process. The discovery of this bug and our subsequent remediation shows that the process is working as intended and worth the effort.

Lessons Learned

What went well

• The non-compliance was discovered through our internal code review process, which shows the process in place is working as intended.

What didn't go well

• Back in 2016/2017, one of the new requirements introduced by a CA/B Forum ballot was not properly actioned, and this oversight was not spotted until 2024.

Where we got lucky

• Usage of email-based DCV methods are in a rapid decline. In a random sample, we observed that less than 1% of all certificates issued through our platform rely on email-based DCV. Amongst our Subscribers that use email-based DCV, most will provide a confirming response containing the Random Value shortly after they receive it via email; only a small minority will attempt this action after 30 days or more.

Action Items

Action Item	Kind	Due Date
Review processes in place to review and action changes to requirements	Prevent	Completed
Expand internal certificate audit processes to check the Random Value age when the confirming response was received	Detect	2024-03-31

Appendix

Details of affected certificates

The list of affected certificates can be found in attachment 9379098 [details]

Comment 3

[Martijn Katerbarg]

With our complete incident report available in comment #2 and remediation completed, all that remains is the expansion of our internal certificate audit process.

Ben, could we set a next-update for March 31st to track this final item?

Comment 4

[Martijn Katerbarg]

Our final action item, incorporating a check against the Random Value age, was completed within our internal audit and will be run starting with the next audit period.

Ben, as this completes our actions on this bug and there have been no further comments, we would like to request closing this bug.

Comment 5

[Ben Wilson]

OK - thanks. I will close this on Wed. 3-Apr-2024.