Closed Bug 1686966 Opened 5 years ago Closed 5 years ago

Camerfirma: Delayed revocations related to certificates without CABForum OV Reserved Policy Identifier

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: ana.lopes, Assigned: ana.lopes)

Details

(Whiteboard: [ca-compliance] [leaf-revocation-delay])

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36

Steps to reproduce:

Actual results:

This bug is opened because of the delay in the revocation of the certificates affected for the bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1685557

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
    We were aware of the problem on January 13th because of the comment 7 in the bug 1685557 - Camerfirma: Certificates without CABForum OV Reserved Policy Identifier (mozilla.org).

  2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
    All the certificates were revoked some minutes after the deadline. The last one was revoked on 2021-01-13 at 16:42:00 UTC.
    From now on, we are going to include the time in the bugs that require revocations, and we will control the time in minutes instead of complete days, as we had done so far.

  3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.
    N/A.

  4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.
    See the list of certificates in the bug 1685557 - Camerfirma: Certificates without CABForum OV Reserved Policy Identifier (mozilla.org)

  5. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.
    See the list of certificates in the bug 1685557 - Camerfirma: Certificates without CABForum OV Reserved Policy Identifier (mozilla.org)

  6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
    We had established all our revocation deadlines considering natural days and not minutes and seconds, so we considered that the revocation of certificates had been performed properly until we received the comment 7 in the bug 1685557 - Camerfirma: Certificates without CABForum OV Reserved Policy Identifier (mozilla.org).
    The revocation process started on 2021-01-13 at 16:24:29 UTC and the last certificate was revoked on 2021-01-13 at 16:42:00 UTC.
    According to the exact time when the incident report was published, all of certificates had to be revoked by 2021-01-08 at 15:51 UTC, that means a delay of 51 minutes.

  7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.
    From now on we will specify future deadlines including hours, minutes and seconds and we will provide our operations and development department with that information to avoid non-compliance cases like this.

Assignee: bwilson → ana.lopes
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Whiteboard: [ca-compliance] [delayed-revocation-leaf]

(In reply to Ana Lopes from comment #0)

The revocation process started on 2021-01-13 at 16:24:29 UTC and the last certificate was revoked on 2021-01-13 at 16:42:00 UTC.
According to the exact time when the incident report was published, all of certificates had to be revoked by 2021-01-08 at 15:51 UTC, that means a delay of 51 minutes.

This is incorrect. 2021-01-08 at 15:51 UTC would be the latest deadline (and enough to establish that revocation was delayed). Making the revocation deadline dependent on publication would entail that there is no revocation deadline if publication of the incident report is not possible (e.g., if Bugzilla is in maintenance).

The real deadline according to BR 4.9.1.1 is 5 days after the CA knows about (or is made aware of) the misissuance. The time of this event has not been yet been disclosed by Camerfirma. It also seems that Camerfirma is not following relevant discussions about revocation deadlines at https://bugzilla.mozilla.org/show_bug.cgi?id=1665763 and https://bugzilla.mozilla.org/show_bug.cgi?id=1649880 .

This whole incident further puts Camerfirma's stated "reasons" for delayed revocation in https://bugzilla.mozilla.org/show_bug.cgi?id=1668331 into question (even before arguments became more bizarre in https://bugzilla.mozilla.org/show_bug.cgi?id=1668331#c20 ). If Camerfirma does not have knowledge (or established a process!) about the BR's revocation timelines, they, obviously, cannot make ANY informed decision about delaying revocation.

(In reply to paul.leo.steinberg from comment #1)

We agree with you in the part that the deadline should not depend on what time you report the error, but the normal procedure for us is to publish the information as soon as we finish our internal investigations to make sure we have detected the affected cases and have enough knowledge about the problem, and that is the moment when we consider that we must start counting the time to comply with the requirements. We cannot establish deadlines before knowing the problem or certificates affected.

We have registered delays in revocation in other cases, but there were no delays for minutes or hours like in this case and a plan with the explanations has been scheduled for all those situations.

(In reply to Ana Lopes from comment #2)

We agree with you in the part that the deadline should not depend on what time you report the error, but the normal procedure for us is to publish the information as soon as we finish our internal investigations to make sure we have detected the affected cases and have enough knowledge about the problem, and that is the moment when we consider that we must start counting the time to comply with the requirements. We cannot establish deadlines before knowing the problem or certificates affected.

BR 4.9.5 makes it pretty clear that the time frame is based on when the CA gets the report, and it doesn't matter when the CA finishes their investigation: "The period from receipt of the Certificate Problem Report or revocation-related notice to published revocation MUST NOT exceed the time frame set forth in Section 4.9.1.1."

(In reply to Mathew Hodson from comment #3)

BR 4.9.5 makes it pretty clear that the time frame is based on when the CA gets the report, and it doesn't matter when the CA finishes their investigation: "The period from receipt of the Certificate Problem Report or revocation-related notice to published revocation MUST NOT exceed the time frame set forth in Section 4.9.1.1."

Although I agree with your sentiment that this took too long, a Certificate Problem Report that reports a class or category of problematic certificates, without the identifiers of those certificates (such that the CA must by themselves find out which certificates are problematic) makes it very difficult to revoke in 24 hours due to the sheer amount of valid certificates at any moment for certain CAs.

Similarly, in this case, I think "it takes >24 hours to determine which certificates of all our issued unexpired certificates are problematic according to this generic problem report" could be a reasonable explanation for exceeding the 24-hour timespan, as having all issued certificates in a ready-to-query format is (afaik) not a written requirement of root stores.

(In reply to Ana Lopes from comment #2)

We cannot establish deadlines before knowing the problem or certificates affected.

You can. I'll give some examples:

  • If you manually verify certificates, each day you detect some amount of incorrect certificates. You can set the internal deadline for revocation of problematic certificates that have been detected that day at the start of that day, plus 120 hours; that way a revocation will definately not be too late for internally discovered problematic certificates.
  • If you have an authorative query that detects problematic certificates, you can put the deadline at 120 hours after each result of that query was returned (or the start of the minute/hour/day that query returned that result if the moment is not clear)
  • ... etc.

I can understand that an exact moment of discovery cannot be determined, but revoking a certificate on time is really not difficult if you're working with the 5-day window, and a 24-hour window is also quite reasonable (e.g. "certificates found to be problematic in the next hour are to be revoked 24 hours from now").

We have registered delays in revocation in other cases, but there were no delays for minutes or hours like in this case and a plan with the explanations has been scheduled for all those situations.

I'm not certain what you mean by this statement. Could you elaborate?

We appreciate your comments.
(In reply to Mathias from comment #4) We are aware that – by a small margin – we did not revoke in time, but we are working a lot on improving the processes, considering also all the useful deadlines information that you provided.

We do not have more updates for this bug.

We do not have more updates for this bug.

We do not have more updates for this bug.

We do not have more updates for this bug.

We do not have more updates for this bug.

We do not have more updates for this bug.

Flags: needinfo?(bwilson)

We do not have more updates for this bug.

Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] [delayed-revocation-leaf] → [ca-compliance] [leaf-revocation-delay]
You need to log in before you can comment on or make changes to this bug.