1651461 - DigiCert: Failure to revoke within 7 days: OCSP EKU issue

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Brenda Bernal]

We acknowledge that DigiCert will not be able to revoke all CAs impacted by the issue referenced under this bugzilla: https://bugzilla.mozilla.org/show_bug.cgi?id=1649951.

We will update with our full incident report before the end of the week as we are continuing our investigations.

Comment 1

[Martin Sullivan]

copy and pasting here to keep things consistent:

Just a quick Update,

We are working with Microsoft on a plan to reduce their timeline we are hoping to have something we can post publicly by the end of this week.

IEXTCA-SSL.ibechtel.com - https://crt.sh/?id=9400462
Are preparing a formal letter describing their destruction process that was completed some time ago, they are expecting this to be provided by the 24th July.

VZ Cybertrust Client CA - https://crt.sh/?id=135970330
KPN Class 2 CA - https://crt.sh/?id=341594698
Both are on track with their timelines.

Comment 2

[Ben Wilson]

An update is needed.

Comment 3

[Brenda Bernal]

Hi Ben, the last update was posted on the other bug here: https://bugzilla.mozilla.org/show_bug.cgi?id=1649951#c7
The other CAs impacted are still working towards the original timelines posted (https://bugzilla.mozilla.org/show_bug.cgi?id=1649951#c4).

Comment 4

[Ryan Sleevi]

Brenda/Martin/Jeremy: I think one of the things we're looking for is called out in https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation and was already requested once from DigiCert in Comment 5 of Bug 1649951

I'm hoping you can use this issue to substantively is to respond, here, to the concern raised:

On the details of this incident report alone, one might conclude that, in the future, if there were any future issues with intermediates, it would also take 7 months to remediate, rather than 7 days, because no detail is given here about the delay and its mitigation. That would be unacceptable then, and that's why it should be unacceptable now. Detail is what helps inform the balanced tradeoff of saying "We accept the risk now, because in the future, we believe there's a clear and viable strategy to reduce the risk, and we can see this as an opportunity to learn".

As called out in https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation, which I realize DigiCert has continued to show some confusion, despite the lengthy discussions about this specific page, is:

You will perform an analysis to determine the factors that prevented timely revocation of the certificates, and include a set of remediation actions in the final incident report that aim to prevent future revocation delays.

Comment 5

[Martin Sullivan]

Full Incident Report:

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

After being alerted of the problem report (reference: https://bugzilla.mozilla.org/show_bug.cgi?id=1649951 ), we started contacting impacted parties and recognized that not all of the CAs would be revoked within the 7 days.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

1-July-2020 original post on MDSP https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/EzjIkNGfVEE
2-July 2020 This bug was created (reference: https://bugzilla.mozilla.org/show_bug.cgi?id=1649951 )
2-July 2020 Scope list of ICA’s is generated. Customer notifications sent.
2-July 2020 through 7-July 2020 – Phone calls with users trying to determine how to optimize shut down
7-July-2020 through 8-July-2020 – Revoked ABB, and confirmed that IEXTCA-SSL.ibechtel.com was already revoked, which left us with 3 additional parties that would extend beyond the 7 days: KPN, Microsoft and VZ Client CA.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

DigiCert no longer issues any new certificates with the OCSP EKU.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

VZ Cybertrust Client CA - https://crt.sh/?id=135970330
Microsoft IT TLS CA 1 - https://crt.sh/?id=21606064
Microsoft IT TLS CA 2 - https://crt.sh/?id=21606056
Microsoft IT TLS CA 4 - https://crt.sh/?id=21606070
Microsoft IT TLS CA 5 - https://crt.sh/?id=21606058
KPN Class 2 CA - https://crt.sh/?id=341594698

Their planned revocation and key destruction timeline were noted here: https://bugzilla.mozilla.org/show_bug.cgi?id=1649951#c4

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

See #4

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

As stated in our original bug, we recognize the security risk and are replacing the certificates. DigiCert doesn’t primarily use delegated responders, meaning there is no reason to create new ICAs with the OCSPSigning EKU going forward.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

As required by Incident Reporting for CAs for delayed revocations, we will provide our decision and rationale for delaying revocation and our analysis that prevented timely revocation. We will have an update on this before the end of next week as we are still finalizing some plans currently.

Comment 6

[Martin Sullivan]

Quick update, we waiting on a reply from Bechtel to see if there is evidence of the Key Destruction

Verizon are currently Scheduled for their Witnessed Destruction by the end of next week.

ABB is revoked and are working on getting a witness for Key Destruction.

KPN and Microsoft are both working towards their current timelines.

Comment 7

[Martin Sullivan]

This weeks update:

• Bechtel's key destruction in 2017 when we terminated our relationship was witnessed by another Bechtel employee, not by an external auditor.

• Verizon completed the witnessed Key destruction this week; the auditors report will be available shortly.

• ABB is revoked. Due to travel restrictions by their Company and with the Crypto material being in a separate country from their PKI operations team, the key destruction is looking to be completed around Q1 2021.

• KPN and Microsoft are both working towards their current timelines.

Comment 8

[Martin Sullivan]

Every one is still on the time line above.

I am hoping in the next update that we can have certifcate counts moving forward to show how this is tracking towards this being closed out.

Comment 9

[Martin Sullivan]

Find below the Verizon Key destruction report.
we should have the Outstanding numbers for MS and KPN CA's next week.

Comment 10

[Martin Sullivan]

Attachment: Verizon-Key-Destruction-Report-of-Independent-Certified-Public-Accountant-signed.pdf

Comment 11

[Martin Sullivan]

Microsoft have informated us they have 631,247 outstanding Certs are on track with their time frame and we will be updating this as we go ahead.

We still on track with the original plan for the Revocation of the KPN ICA's this month. Which will just leave the above Microsoft ones.

I plan to give the next update 1st Nov where I expect we will have updated numbers for Microsoft showing the trend towards 0 and confirmation all the close out tasks for KPN are complete.

Comment 12

[Martin Sullivan]

The KPN CA (and it's children) have been revoked this week as per our schedule.

We are working to get the Key destruction witnessed and report generated next.

The next update tracking Microsoft's cert should still be start of Nov.

Comment 13

[Martin Sullivan]

Microsoft are now down to 259,997 Active Certs.

as can be seen this is trending well for the expected timeline to revoke.

I am hoping to have the confrimed date for the Witnessed Key Destruction for the KPN CA in the next update.

Comment 14

[Martin Sullivan]

Nothing much to update currently.

Microsoft are still trending down as expected.

The Key destruction for the KPN CA will be either then 2nd or 9th of Dec depending on avaliblity of Witness.

Comment 15

[Martin Sullivan]

Microsoft are down to 100,715 active certs trending well for the revocation on schedule early 2021.

Comment 16

[Martin Sullivan]

we had a slight delay lining up Auditors and Staff at the diffrent sites for the KPN Key Destruction.

this is now booked in for this Wed 16th.

Next update will be confirming this is completed.

Comment 17

[Martin Sullivan]

I can confirm the key destruction for KPN was completed and witnessed last Wednesday and the report is being prepared.

We are now just waiting on the Microsoft CA's, the next update will be on their current active numbers and we shall then lock in the Key destruction for that one as well.

Comment 18

[Martin Sullivan]

Microsofts last update is they are down to 30890 active certificates.
They are epecting to revoke on the 16th Feb with the Key destruction happening shortly thereafter.

we are still waiting on the KPN report from auditors and will post as soon as we have that.

Comment 19

[Martin Sullivan]

We have recived the draft Key destruction report and will post up the final once signed off.

The revoke for MS CA's is confrimed for 16th Feb. with Key Destruction happening prior to March 1st

Comment 20

[Martin Sullivan]

Microsoft are down to 6407 active certs, these ICA are still on track for revocation on the 16th of this month.

the Key destruction report has been accepted and we are just waiting on signatures.

Comment 21

[Martin Sullivan]

Please find the Key destruction report for the KPN CA:
https://bugzilla.mozilla.org/attachment.cgi?id=9202662

we are still on track for the revocation of the Microsoft CA's next week

Comment 22

[Martin Sullivan]

I can confirm the revoke happened on schedule on the 16th

the Key destruction audit/report is currently in process.

Comment 23

[Martin Sullivan]

link below to the final key destruction:

https://bugzilla.mozilla.org/show_bug.cgi?id=1649951#c31

this has remediated all the issues on this case and I ask that it is closed.