Closed Bug 1816806 Opened 2 years ago Closed 2 years ago

DigiCert: OCSP not responding issue

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: brenda.bernal, Assigned: brenda.bernal)

Details

(Whiteboard: [ca-compliance] [ocsp-failure])

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

During an audit of our production systems in Europe, our auditors (BDO) found that the OCSP responder was not updating as expected. The last update was 13-January-2023.

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

All times are MST
17:16 13-January-2023 DigiCert production team deployed changes to web servers that were related to OCSP operation in our European data centre.
15:24 09-Feb-2023 Auditor notifies DigiCert Compliance informing us that they found an issue with the OCSP responder.
15:45 09-Feb-2023 Notification was reviewed, and investigation started to confirm the issue.
16:25 09-Feb-2023 Issue confirmed and escalated to Engineering.
18:11 09-Feb-2023 Issue resolved and tested.
09:27 10-Feb-2023 Added monitoring to the new systems located in Europe.

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

This is impacting OCSP responses only with no issues with certificates.

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

There are no impacted certificates since these are OCSP response issues only.

  1. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

There are no impacted certificates since these are OCSP response issues only.

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The cause was a failure of the API. The configured API key kept failing the permission check after the CA was updated to 64-bit. The update to 64-bit occurred on 13-January-2023 as per the above timeline on 1). The failed API was being used to transfer the OCSP responses from the origin servers from the CA.

This was on a new system located in Europe that was still in testing phase. No customers were onboarded yet. The alerts for this system were not yet configured so DigiCert production team were not notified of the OCSP update issue.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Once this issue was confirmed, a fix was identified which was to generate and install a new API key. This successful installation returned the service to a working state. We have already deployed the alerts in production in Europe for our OCSP responder.

We have added a step to audit other alerts in our production environment to ensure coverage. This step will be worked on ahead of each production release. The enablement and testing of real-time alert and monitoring is an added checklist item required for tickets to be reviewed and approved in the weekly change control meeting to ensure completeness. This change is effective immediately.

Flags: needinfo?(benwilsonusa)
Assignee: nobody → brenda.bernal
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Flags: needinfo?(benwilsonusa)
Whiteboard: [ca-compliance] [ocsp-failure]

We have not further updates on this issue. Does anyone have any questions? If not, can I request Ben that we close the bug?

As there does not seem to be any questions, Ben can we close this?

Flags: needinfo?(bwilson)

Ben is there anything else you need from us?

I'll close this on or about Wed. 8-March-2023.

Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.