1627152 - DigiCert: OCSP NextUpdate

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Brenda Bernal]

Incident Report – Mozilla Policy Violation

1.How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

DigiCert was working with our non-TLS SubCAs to address and update their CPS to reflect the changes to the Mozilla Root Store Program Requirements, given the April 1st deadline. One of those Sub CAs sent an e-mail requesting more information about the requirement “responses MUST have a defined value in the nextUpdate field, and it MUST be no more than ten days after the thisUpdate field.” They were unsure if this applied to s/MIME since they do real-time OCSP signing for sMIME. We informed them the language applies to all end-entity certs as written.

After the report, we investigated further and found that the software used by the customer (DigiCert provided software) does not permit customers to include a thisUpdate value in their OCSP responses. The issue is limited to our legacy Symantec on prem CA platform (MPKI7) used for sMIME and private certs.

For clarity, the issue is limited solely to s/MIME, and does impact both technically constrained ICAs and non-technically constrained ICAs operated through the software.

2.A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

March 6, 2020 – SubCA requested clarification about a Mozilla policy requirement in relation to OCSP responses for S/MIME certificates. Digicert researches issues and begins root cause analysis.
March 28, 2020 – DigiCert confirmed that the customer is responding to OCSP responses without a nextUpdate field. The root cause is that the software was developed to meet 6960 (where nextUpdate is optional), but not the Mozilla requirement.
March 30-31, 2020 - Internal discussions occurred confirming issue and planning a response / software fix.
April 1, 2020 – Internal team scoping project.
April 2, 2020 – DigiCert files initial incident report.

3.Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

We have not stopped issuance. The issue applies to revocation information, which is still being provided.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

We are unsure at this point. We know the issue impacts OCSP information for a subset of sMIME certificates issued through this platform.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Not sure the best way to report this in this particular case. Could you provide guidance on what to do with this for client certs?

6.Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The software that some of our SubCAs use does not pre sign OCSP responses and provides OCSP responses in real-time to requests. This avoided detection as the developers believed it was compliant based on RFC6960, which states that “If nextUpdate is not set, the responder is indicating that newer revocation information is available all the time.” (https://tools.ietf.org/html/rfc6960#section-4.2.2.1) The Mozilla requirement was missed by the prod/dev team responsible for the legacy software during their review.

7.List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

DigiCert Development is rolling out a fix to set a nextUpdate field that is compliant with Mozilla Root Store Policy. We will provide an update once scoping is finished.

Comment 1

[Ryan Sleevi]

(In reply to Brenda Bernal from comment #0)

March 6, 2020 – SubCA requested clarification about a Mozilla policy requirement in relation to OCSP responses for S/MIME certificates. Digicert researches issues and begins root cause analysis.
March 28, 2020 – DigiCert confirmed that the customer is responding to OCSP responses without a nextUpdate field. The root cause is that the software was developed to meet 6960 (where nextUpdate is optional), but not the Mozilla requirement.

Why such a long gap?

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Not sure the best way to report this in this particular case. Could you provide guidance on what to do with this for client certs?

Can you provide an attachment as to which certificates are configured to use this particular (set of?) OCSP responders?

OCSP cases are understandably more complex than general misissuance, but the overall objective remains to assess the scope of the issue and the potential impact to clients.

Comment 2

[Brenda Bernal]

With regards to the time gap, we had multiple CPS reviews occurring simultaneously given the April 1st deadline. We needed to research internally with the software team responsible to confirm how the OCSP is functioning with this platform since it is legacy.

I can work to get that file attachment with the certificate information you requested, but will need a few days to collect that information. I'm hoping to provide something by next Wed (April 8th).

Comment 3

[Brenda Bernal]

See below to update on CA certs in scope:
https://crt.sh/?id=484579148
https://crt.sh/?id=93941261
https://crt.sh/?id=945961436
https://crt.sh/?id=12722281
https://crt.sh/?id=56158747
https://crt.sh/?id=1786693519
https://crt.sh/?id=572219617
https://crt.sh/?id=198542882
https://crt.sh/?id=572219616
https://crt.sh/?id=198542881
https://crt.sh/?id=1435339078
https://crt.sh/?id=1435339079
https://crt.sh/?id=1639470764
https://crt.sh/?id=1639472075

Also an update on 7) for the initial report above:
The patch has been completed. It is currently going through QA for testing. We will update once this is complete and we have released to Customers or in 7 days, whichever is sooner.

Comment 4

[Brenda Bernal]

The patch was released to customers early this week (Monday). One of the customers has already tested and deployed to production. The other customer is in testing phase. We will update once they are out of testing and have successfully deployed to production.

Comment 5

[Brenda Bernal]

Update: the last customer has deployed the OCSP fix into production successfully today. If there are no other questions, we request the bug to be closed. Thank you.

Comment 6

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.