Closed Bug 1653504 Opened 1 year ago Closed 1 year ago

Sectigo: Certificates with RSA keys where modulus is not divisible by 8

Categories

(NSS :: CA Certificate Compliance, task)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: nick, Assigned: nick)

References

Details

(Whiteboard: [ca-compliance])

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Related to: https://bugzilla.mozilla.org/show_bug.cgi?id=1653475
At 5am BST on Friday July 17th, Jeremy emailed our SSL Abuse reporting email with details of a number of certificates issued with non-divisible-by-8 RSA key sizes.

We assessed the list and found 32 certificates total.

However, the majority of the list was disclosed over a year ago in: https://bugzilla.mozilla.org/show_bug.cgi?id=1518553

We re-checked the list and found 2 certificates that had incorrect key sizes but were NOT on the list in 1518553:
https://crt.sh/?id=2310092134
https://crt.sh/?id=2310032071

These certificates have now been revoked (9:30 BST).

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

Jan 8, 2019: https://bugzilla.mozilla.org/show_bug.cgi?id=1518553 was opened, covering issuance of certificates with the p521 curve, and also RSA keys of incorrect size (comments 11 and 12).
Jul 17, 2020: Jeremy (Digicert) informed us of a number of certificates that used incorrect RSA key sizes after their own investigations (https://bugzilla.mozilla.org/show_bug.cgi?id=1653475)
We verified the list, and discovered the 2 missed certificates.
Those certificates were revoked.

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

We blocked issuance of certificates with invalid RSA key sizes back in 2019. We also reported a list of certificates issued before the block.
Two certificates were missed from that list.

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

https://crt.sh/?id=2310092134
https://crt.sh/?id=2310032071

  1. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

As #4.

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The two missed certificates were in a state ('replaced') in our system. This meant they were not considered by the reporting query that generated the list in bug 1518553 (which looked only explicitly for 'issued').

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We have immediately revoked the two affected certificates.
We are re-running a scan of our corpus of certificates for any more missed from 1518553, and will update this bug once that is completed.

Assignee: bwilson → nick
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Whiteboard: [ca-compliance]

Thanks Nick.

I'm actually quite concerned about this, and hoping that the explanation in Sections 6 and 7 are just placeholders for more thoughtful and thorough analysises.

Specifically, Comment #6 identifies a bug that has already been flagged with other CAs, some repeatedly. Have you analyzed other CA incidents discussing this sort of issue? For example, Bug 1526154, Bug 1524815, Bug 1462844 were all discussed on mozilla.dev.security.policy in February 2019. While this is after the original January 2019 response in Bug 1518553, it's well before the updated list in Comment #11

To what extent did or does Sectigo follow incidents from other CAs? What steps does Sectigo take to institutionalize and distribute the lessons from those incidents? What processes does Sectigo have to learn from the mistakes of other CAs and re-evaluate whether it made those same mistakes?

My hope is you'll similarly revisit the answer in Section 2 to more thoroughly document, as a "date-and-time-stamped sequence of all relevant events", the surrounding context and factors here. Given that Section 6 proposes to identify a root cause at Sectigo, it's entirely reasonable to evaluate whether other CA incidents shared that root cause, and determine what could have been used to prevent Sectigo from being affected.

This is why I'm quite concerned: If Sectigo isn't learning from other CAs' incidents, this is quite worrying. Understanding a holistic set of processes designed to review and detect these is critical to this incident.

I want to emphasize: there's no detail too small to be considered including in Section 6 / Section 7. Your goal, as much as possible, is to demonstrate holistically to anyone how thoroughly mitigated this issue is. Consider the feedback to other CAs and What makes a good incident response

Flags: needinfo?(nick)

Ryan: I have a more detailed update to post early tomorrow morning (UK), as a result of further reporting Rob has done against our certificate body, with more details for sections 6 and 7.

Flags: needinfo?(nick)

We've continued doing some analysis here and re-running reports over our body of issued certificates.
As such, I can expand on sections 6 and 7:

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

As explained above, the report in https://bugzilla.mozilla.org/show_bug.cgi?id=1518553#c11 was generated by Robin following his comment at https://bugzilla.mozilla.org/show_bug.cgi?id=1518553#c7:
"We will follow up with analysis of the prevalence of subscriber keys in issued certificates whose RSA moduli are not divisible by 8."

After discussing the history of bug #1518553 in more depth with Robin and Rob yesterday, I now understand that that report was generated from CT log data (using crt.sh) and NOT, as I had stated earlier in this bug, from our internal dataset. Robin had wanted to look at and share data about disallowed RSA moduli in the wider WebPKI ecosystem, not just Sectigo's own failings in this regard. Robin's chosen approach was also influenced by the fact that, at the time, querying our internal CA database for disallowed RSA moduli was a slower and more complex process than querying crt.sh.

My statement about the certificates being 'replaced' was true, but was not the cause of the problem. The two certificates mentioned here did not appear in any CT logs before early 2020, having been issued prior to Chrome's CT enforcement date. The report that Robin generated was based on an incorrect understanding that the crt.sh dataset would cover all Sectigo-issued certificates. While CT compliance was mandatory for newly-issued certificates at the time of the report generation, it was overlooked that some older certificates issued by Sectigo had not been submitted to CT logs by Sectigo and may not have been found and submitted to CT logs by GoogleBot or other third-parties.

We neglected to double-check the reporting and that it covered all Sectigo-issued certificates. Consequently, the requirement to 'scan your corpus of certificates' [https://wiki.mozilla.org/CA/Responding_To_An_Incident#Follow-Up_Actions] was missed, and this resulted in 12 Sectigo-issued certificates with RSA moduli not divisible by 8 remaining undetected until Jeremy's report reached us.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

From the list provided by Jeremy (Digicert) we found two (2) certificates that were not in Robin's original report, as detailed above.

We then performed a scan over the body of certificates in our own CA system, looking for any certificates we issued whose RSA moduli are not divisible by 8.
Upon completion of the report, filtering down to unexpired, unrevoked certificates that were not on the list Robin originally reported, we had an additional ten (10) certificates.
As with the previous two, rather than suggest they should have been part of the original report and remain valid, our team is already processing revocations.
They will be revoked on or before 2-Aug-2020 6am UTC.
The crt.sh links for these ten are:
https://crt.sh/?id=1271591244
https://crt.sh/?id=1262339801
https://crt.sh/?id=3150234739
https://crt.sh/?id=2309988066
https://crt.sh/?id=3150234754
https://crt.sh/?id=3150235847
https://crt.sh/?id=3150235863
https://crt.sh/?id=3150235734
https://crt.sh/?id=3150235806
https://crt.sh/?id=3150235881

As will be detailed in an (imminent) update to bug #1563579, changes have and are being made to the processes by which incidents are handled now and going forward. The change of most relevance to this bug is that we will be introducing a more formal process of peer review for each phase of each incident response, to reduce the likelihood that one person's incorrect understanding can remain undetected.

As for ensuring that "such issuance will not be repeated in the future", it's worth noting that the defect in Sectigo's CA system (that permitted RSA moduli not divisible by 8) was successfully addressed by our response to bug #1518553.

This spreadsheet shows the result of our "scan over the body of certificates in our own CA system" that Nick just mentioned:
https://docs.google.com/spreadsheets/d/1R0r_tp9ymDj_kgVxHCK_Vy_f-dJ-08hvnJLJlCKRSsA/edit?usp=sharing

We're also in the process of confirming that the externally-operated CAs that inherit trust from Sectigo-operated roots have been operated in accordance with the "RSA keys whose modulus size in bits is divisible by 8" requirement. There's a separate tab in the spreadsheet for each externally-operated CA organization. We're currently awaiting a response from Apple and D-TRUST. I'll update this bug once this further analysis is complete.

Except for the SUMMARY tab, this spreadsheet lists every noncompliant certificate and precertificate found, including the ones that Robin previously disclosed in bug #1518553. (Note that certificate/precertificate pairs were deduplicated in the list of crt.sh links provided above by Nick).

(In reply to Rob Stradling from comment #4)

We're currently awaiting a response from Apple and D-TRUST. I'll update this bug once this further analysis is complete.

Apple have not issued any certificates that fall foul of the "RSA keys whose modulus size in bits is divisible by 8" requirement. We're expecting a response from D-TRUST this week.

(In reply to Rob Stradling from comment #5)

We're expecting a response from D-TRUST this week.

D-TRUST have not issued any certificates that fall foul of the "RSA keys whose modulus size in bits is divisible by 8" requirement.

Ben, can this bug be closed now?

Flags: needinfo?(bwilson)
Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Duplicate of this bug: 1725041
You need to log in before you can comment on or make changes to this bug.