Open Bug 1914383 Opened 1 month ago Updated 19 minutes ago

Telekom Security: CRL-Entries with wrong CRL Reason Codes

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
ASSIGNED

People

(Reporter: Arnold.Essing, Assigned: Arnold.Essing)

Details

(Whiteboard: [ca-compliance] [crl-failure])

Attachments

(1 file)

wrong_revocation_reasoncodes.csv
25.90 KB, text/csv
Details

Preliminary Incident Report

Summary
As part of our bug https://bugzilla.mozilla.org/show_bug.cgi?id=1875820 “Telekom Security: TLS certificates with basicConstraints not marked as critical”, certificates also had to be revoked.
During the revocation incorrect CRL reason codes were also used. This does not comply with section 4.9.1.1 of the TLS BRs.

Impact
The required reason code CRLReason #4, 'superseded,' was not set for all of the affected certificates.

Next steps
A full incident report including the root cause analysis, lessons learned, and action items will be posted.

Assignee: nobody → Arnold.Essing
Status: NEW → ASSIGNED
Whiteboard: [ca-compliance] [crl-failure]

Final Incident Report

Summary
As part of our Bug 1875820 "Telekom Security: TLS certificates with basicConstraints not marked as critical", 762 TLS certificates were revoked, partially by Telekom Security as CA, partially by the Enterprise RAs of the affected customers.
Since the certificates had to be revoked because they were not issued in accordance with the BR, the reason for revocation "superseded" according to the BR#4.9.1.1 (12) should have been chosen.
However, 298 of the affected certificates were revoked with different revocation reasons (166 unspecified, 1 keyCompromise, 123 affiliationChanged, 8 privilegeWithdrawn).

Impact
The CRL of the “TeleSec Business TLS-CA 2022” contains 298 entries with wrong resp. missing revocation reasons.
Since this bug does not affect the issuance of certificates, the issuance has not been stopped.

Timeline
All times are UTC.
2022-06-01: Mozilla Root Store Policy 2.8 becomes effective
2022-09-09: All Systems have been updated to support only the reasonCodes allowed by Mozilla
2022-10-01: The usage of reasonCodes according to the Mozilla Root Store Policy 2.8 becomes effective
2023-07-15: Ballot SC61 “New CRL Entries must have a Revocation Reason Code” becomes effective (adoption of the usage of reasonCodes according to MRSP 2.8)
2024-01-22, 11:37: 816 certificates have been identified that are affected by the above mentioned bug. Of these, 762 certificates had to be revoked, 54 certificates had already been revoked
2024-01-22, 16:37: The affected customers were informed and were asked to exchange and revoke the affected certificates as soon as possible but within 5 days at the latest
2024-01-23 - 2024-02-06: Revocation of all certificates, partially by customers themselves, partially by Telekom Security
2024-02-06, 15:41: All affected certificates are replaced and revoked
2024-08-21, 16:16: Mail from member Chrome CA Program with a reference to the CRL Reason Codes used
2024-08-22, 04:30: Investigation started by a member of the Compliance-Team
2024-08-22, 09:00: Meeting of the Compliance-Team with the result that incorrect revocation reasons were chosen
2024-08-22, 12:00: Management Call
2024-08-22, 12:56: This bug was filed with a preliminary incident report
2024-08-22, 14:03: Feedback to the Chrome CA program, subsequently to the Apple and Microsoft CA programs
2024-08-23, 06:20: The list of affected certificates resp. the revocations with wrong revocation reasons has been finalized

Root Cause Analysis
The requirement to select the revocation reason “superseded” according to MRSP 2.8 or Ballot SC61 was not included in our bug process description, so we did not point out this requirement to the customers when we wrote the request to revoke the certificates affected by Bug 1875820.
Since also the instructions in the customer frontend for selecting a revocation reason did not explicitly address the case of misissued certificates and the Enterprise RAs were focused on meeting the revocation deadlines, incorrect reasons for revocation were chosen in 298 cases, also due to time pressure.

Lessons Learned

What went well
Both our internal RAs and Solution Managers were aware of the need to use the revocation reason “superseded”, so they pointed this out in telephone conversations with some customers following the written request.

What didn't go well
The above mentioned discussions did not take place with all customers. In the end, not all customers were explicitly informed of the use of the correct revocation reason.
Following the revocation, we did not check the selection of revocation reasons and did not detect the Bug ourselves.

Where we got lucky
n/a

Action Items
| Action Item | Kind | Due Date |
| Internal training of all employees of the solutions that issue public certificates| Prevent | 2024-08-30 |
| Information to customers regarding the use of the correct revocation reasons with reference to the descriptions in CP and CPS | Prevent | 2024-08-30 |
| Adaptation of internal process descriptions for Bug handling including a subsequent check of the revocations carried out | Prevent | 2024-08-30 |
| Adjustment of the Terms of Use regarding revocation in the case of misissuance| Prevent | 2024-10-01 |

Appendix
Details of affected certificates
For the list of affected certificates respective revocations, see Appendix (wrong_revocation_reasoncodes.csv)

Stefan,

Both your root cause analysis as well as the Lessons Learned, seem to put the burden on revocation with the correct reasoncode on your customers. To that end, I have a few questions:

  • Given that your original bug (bug 1875820) affected 816 certificates, yet this bug affects a total of 298, could you confirm the revocation of the other certificates were initiated by Telekom Security itself, and with the correct reasoncode?
  • You mention one certificate has the keyCompromise reasonCode. My initial thought here is that the customer revoked the certificate themselves specifying this reasoncode. Is that correct?
    • If that is indeed correct, can you explain why you are in this bug indicating it is an incorrect use of reasonCode?

Both the BRs and the Mozilla Root Store Policy allow this reasonCode if it is specified by the customer. It would seem to me that, if the certificate is scheduled to be revoked with reasonCode “superseded”, but prior to the execution of that revocation the customer requests revoking it with “keyCompromise”, then the use of keyCompromise would be warranted and compliant.

The same question goes for the 166 marked as “unspecified”. If these customers requested the certificate to be revoked without a reasoncode, prior to the execution of the scheduled “superseded” revocation, are they really incorrect?

This question is also for the Root Stores and larger community: The BRs state that a CA MUST revoke a certificate within 24 hours with CRLReason "unspecified (0)" if requested by the Subscriber. I do not believe there’s a requirement to change a reasoncode if new information becomes available. While I believe that might be a good thing, especially for keyCompromise, I don’t believe there currently is a requirement to change reasoncodes after revocation has occurred.

Hello Martijn,
Thank you for your comments.

Given that your original bug (bug 1875820) affected 816 certificates, yet this bug affects a total of 298, could you confirm the revocation of the other certificates were initiated by Telekom Security itself, and with the correct reasoncode?

As a note: The original bug affected 816 certificates that have been issued since 2023-09-15. Only 762 of these were still active when the original bug was discovered, i.e. only 762 of the 816 certificates had to be revoked due to the bug.
Regarding your question: The remaining 464 certificates have been revoked with the correct reasonCode. These revocations were made by Telekom Security as well as by some of the customers who simply chose the correct reasonCode by themselves or who had been informed incidentally as described in “What went well”.

You mention one certificate has the keyCompromise reasonCode. My initial thought here is that the customer revoked the certificate themselves specifying this reasoncode. Is that correct?
If that is indeed correct, can you explain why you are in this bug indicating it is an incorrect use of reasonCode?
Both the BRs and the Mozilla Root Store Policy allow this reasonCode if it is specified by the customer. It would seem to me that, if the certificate is scheduled to be revoked with reasonCode “superseded”, but prior to the execution of that revocation the customer requests revoking it with “keyCompromise”, then the use of keyCompromise would be warranted and compliant.
The same question goes for the 166 marked as “unspecified”. If these customers requested the certificate to be revoked without a reasoncode, prior to the execution of the scheduled “superseded” revocation, are they really incorrect?

Yes, the certificates have been revoked by the customers and the reasonCodes have been specified (or “unspecified”) by them as well. We also confirmed with the customer that the keyCompromise was an oversight / miss click by the customer and that no actual key compromise occurred. We considered the chosen reasonCodes to be technically incorrect because the revocations performed by our customers are most likely still based on our bug and therefore BR 4.9.1.1 (12).

This question is also for the Root Stores and larger community: The BRs state that a CA MUST revoke a certificate within 24 hours with CRLReason "unspecified (0)" if requested by the Subscriber. I do not believe there’s a requirement to change a reasoncode if new information becomes available. While I believe that might be a good thing, especially for keyCompromise, I don’t believe there currently is a requirement to change reasoncodes after revocation has occurred.

We do not think that the CRL reasonCodes should be changed retroactively and did not intend to change them (unless there would be a key compromise).

We do not think that the CRL reasonCodes should be changed retroactively and did not intend to change them (unless there would be a key compromise).

I disagree with that statement. The CABF has historically accepted changes to reasonCode value for past revocations. This was discussed in MDSP in 2020 after ballot SC31 for reasonCodes associated with CA Certificates, became effective.

A CA should be allowed to change the reasonCode for a specific revocation as new evidence becomes available.

FWIW - For CAs trusted by Mozilla, it's currently not a "requirement". Instead, it's a "should" -- "When the CA operator obtains verifiable evidence of private key compromise for a certificate whose CRL entry does not contain a reasonCode extension or has a reasonCode extension with a non-keyCompromise reason, the CA operator SHOULD update the CRL entry to enter keyCompromise as the CRLReason in the reasonCode extension. Additionally, the CA operator SHOULD update the revocation date in a CRL entry when it is determined that the private key of the certificate was compromised prior to the revocation date that is indicated in the CRL entry for that certificate."

However, as further guidance, https://wiki.mozilla.org/CA/Revocation_Reasons#Updating_CRL_Entries says,

  • The CRLReason may only be changed from a non key compromise reason to the keyCompromise reason.
  • The exception to the best practice described in RFC 5280 (section 5.3.2) only applies when the CRLReason is keyCompromise.
    • When setting the date for an initial revocation for CRLReason keyCompromise the revocationDate may be in the past.
    • Backdating for any other CRLReason is not endorsed by Mozilla.

Our current Action Items status is as follows:
| Action Item | Kind | Due Date |
| Internal training of all employees of the solutions that issue public certificates | Prevent | 2024-08-30 (completed) |
| Information to customers regarding the use of the correct revocation reasons with reference to the descriptions in CP and CPS | Prevent | 2024-08-30 (completed) |
| Adaptation of internal process descriptions for Bug handling including a subsequent check of the revocations carried out | Prevent | 2024-08-30 (completed) |
| Adjustment of the Terms of Use regarding revocation in the case of misissuance | Prevent | 2024-10-01 |

(In reply to Ben Wilson from comment #6)

FWIW - For CAs trusted by Mozilla, it's currently not a "requirement". Instead, it's a "should" -- "When the CA operator obtains verifiable evidence of private key compromise for a certificate whose CRL entry does not contain a reasonCode extension or has a reasonCode extension with a non-keyCompromise reason, the CA operator SHOULD update the CRL entry to enter keyCompromise as the CRLReason in the reasonCode extension. Additionally, the CA operator SHOULD update the revocation date in a CRL entry when it is determined that the private key of the certificate was compromised prior to the revocation date that is indicated in the CRL entry for that certificate."

However, as further guidance, https://wiki.mozilla.org/CA/Revocation_Reasons#Updating_CRL_Entries says,

  • The CRLReason may only be changed from a non key compromise reason to the keyCompromise reason.
  • The exception to the best practice described in RFC 5280 (section 5.3.2) only applies when the CRLReason is keyCompromise.
    • When setting the date for an initial revocation for CRLReason keyCompromise the revocationDate may be in the past.
    • Backdating for any other CRLReason is not endorsed by Mozilla.

Ben,

We should not be using this bug to discuss MRSP issues but FWIW, for TLS Certificates, backdating a specific revocation entry does not make sense in practice because the status of a certificate is checked at connection time. Changing the CRLReason when the CA determines a more appropriate entry, should be allowed IMO, just like it's allowed for the status of CA Certificates.

As a separate matter, does the fact that there is a statement on a wiki that says that a certain practice "is not endorsed by Mozilla", make it policy enforceable? If there are strong feelings about this being considered a very bad practice, I believe it should be discussed and added to the TLS BRs.

We are on track with our remaining Action Item. Please let us know if there are any further comments or questions.

We are on track with our remaining Action Item. Please let us know if there are any further comments or questions.

We are on track with our remaining Action Item. Please let us know if there are any further comments or questions.

You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: