Closed Bug 1649880 Opened 1 year ago Closed 1 year ago

QuoVadis: Failure to provide a preliminary report within 24 hours.

Categories

(NSS :: CA Certificate Compliance, task)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: fozzie, Assigned: stephen.davidson)

Details

(Whiteboard: [ca-compliance] Next Update 1-October 2020)

I sent a problem report to compliance@quovadisglobal.com and the preliminary report was not sent within 24 hours.

Tuesday 30th June 17:22 UTC - I sent a report concerning https://misissued.com/batch/122/ and "Wuppertal" not being a province in Germany.
Wednesday 1st July 20:37 UTC - I recieved a preliminary report from QuoVadis stating that they will revoke the certificates.

The preliminary report was not sent within 24 hours as mandated by section 4.9.5 of the Baseline Requirements.

As of Monday 6th July 15:51 UTC (5 days, 22 hours after the initial report) the certificates still haven't been revoked.

Therefore, QuoVadis has also failed to revoke these certificates within 5 days as mandated by section 4.9.1.1 of the Baseline Requirements.

Assignee: bwilson → stephen.davidson
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance]
  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

A problem report was filed by George to compliance@quovadisglobal.com.

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

30 June 2020, 12:22 UTC – problem report filed by George.
30 June 2020, 17:31 UTC – acknowledgement sent by QuoVadis to George.
1 July 2020, 20:37 UTC – confirmation sent by QuoVadis to George that the certificates would be revoked.
6 July 2020, 17:07 UTC – certificates revoked within 5 days.

  1. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

See below.

  1. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

Date of first certificate issued: Sept-07-2017
Date of last certificate issued: Oct-20-2017

The certificates included:
L = Remscheid
ST = Wuppertal
C = DE

Properly the certificates should have included:
L = Remscheid
ST = Nordrhein-Westfalen
C = DE

  1. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

https://crt.sh/?id=237733387
https://crt.sh/?id=307551562

A review was made; no other similar valid certificates were identified.

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

See below.

George reports a failure to communicate in a timely manner. QuoVadis communicated with George 8 times on 30 June/1 July on this and other reports. Working from home, my final response on 1 July was delayed due to an internet outage in my neighborhood (where I am sheltering in place) which required me to go to the office to send the response confirming the revocation.

  1. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

Since these certificates were issued in 2017, QuoVadis adopted ISO 3166-2 as a baseline for determining appropriate stateOrProvinceName.
A review has been made of all approved Organization details in our certificate management system for compliance against ISO 3166-2.

Just a few things:

(In reply to Stephen Davidson from comment #2)

30 June 2020, 12:22 UTC – problem report filed by George.
30 June 2020, 17:31 UTC – acknowledgement sent by QuoVadis to George.
1 July 2020, 20:37 UTC – confirmation sent by QuoVadis to George that the certificates would be revoked.
6 July 2020, 17:07 UTC – certificates revoked within 5 days.

This shows the certificates were not revoked within 5 days? More than 6 days had passed since my report and when you revoked the certificates, can we expect another incident to be opened on the delay in revocation as well?

QuoVadis communicated with George 8 times on 30 June/1 July on this and other reports.
I personally don't see how this is relevant to this specific report. You only sent two responses on this one, the initial acknowledgement and the delayed preliminary report.

QuoVadis believes this matter has been properly resolved, and requests this bug be closed.

You still haven't either said how you've revoked the certificates within 5 days or opened an incident report about a delay in revocation, can you clarify further on this?

Flags: needinfo?(stephen.davidson)

The certificate was revoked within 5 days of our determination that information appearing in the cert was inaccurate, which is compliant with BR 4.9.1.1.

Flags: needinfo?(stephen.davidson)

The certificate was revoked within 5 days of our determination that information appearing in the cert was inaccurate, which is compliant with BR 4.9.1.1.

BR 4.9.5:

The period from receipt of the Certificate Problem Report or revocation-related notice to published revocation MUST NOT exceed the time frame set forth in Section 4.9.1.1.

I've seen many CAs fail to argue that the timeframe set forth in 4.9.1.1 starts at the point in time the problem report was validated by the CA (e.g. Bug 1640805 Comment 8), only to be pointed out that section 4.9.5 invalidates this claim. Please stop making this mistake. If the problem report was received at 30 June 2020, then the revocation should not have been later than 5 July 2020.

The circumstances here are different from your example. The language from BR 4.9.1.1 is: “The CA determines or is made aware that any of the information appearing in the Certificate is inaccurate”. This differs from the other items in that section which simply say “The CA is made aware.” This is because a CA typically must take steps to determine that that information is inaccurate.
As noted earlier in this bug, this report was one of several discussions that was taking place with the reporter at that time; in other cases the certificates were found to be correct.
Proper investigation is the correct approach unless specific evidence of a certificate being inaccurate is provided in the report. Similarly, one can’t submit that a cert has compromised keys and expect 24hr revocation without providing evidence of keycompromise.

Under 4.9.1.1 it states

The CA SHOULD revoke a certificate within 24 hours and MUST revoke a Certificate within 5 days if one or
more of the following occurs:

  1. The CA determines or is made aware that any of the information appearing in the Certificate is
    inaccurate;

Under 4.9.5 Time within which CA Must Process the Revocation Request it states

The period from receipt of the
Certificate Problem Report or revocation-related notice to published revocation MUST NOT exceed the time
frame set forth in Section 4.9.1.1.

This seems pretty clear to me.

(In reply to Stephen Davidson from comment #8)

The circumstances here are different from your example. The language from BR 4.9.1.1 is: “The CA determines or is made aware that any of the information appearing in the Certificate is inaccurate”. This differs from the other items in that section which simply say “The CA is made aware.” This is because a CA typically must take steps to determine that that information is inaccurate.

Whoa! Can you please supply any supporting evidence for that reading of the BR? It might be helpful to determine if there is a systemic problem in the procedures used to interpret the BR at QuoVadis.

That exact line was last change by SC6 and discussed in https://lists.cabforum.org/pipermail/servercert-wg/2018-August/000163.html, which suggest the exact opposite of your statement and describes indeed the exact situation we are now in.

I do not see how QuoVadis can interpret the BR without researching its history and accompanying discussions and wonder how the community should trust QuoVadis with interpreting the BR correctly.

Flags: needinfo?(stephen.davidson)

think what Stephen is referring to is that a lot of these bug reports allege some vague badness that can't be determined from the initial report. Later clarification provides a copy of the key or some evidence that something went wrong. The CA isn't made aware until there is some pointed evidence that the CA can investigate to determine whether something went wrong. Otherwise, I can allege a "CA A is not following the BRs" and then claim they failed to revoke within 24 hours/7 days every time an issue is found. This doesn't make sense since I could just email every CA now and double the number of incident reports required for any future issue.

What should happen though (and didn't happen here) is that the CA should report back that they are unaware of an wrongness with the current supplied information and ask for clarification.

From 4.9.5:

Within 24 hours after receiving a Certificate Problem Report, the CA SHALL investigate the facts and
circumstances related to a Certificate Problem Report and provide a preliminary report on its findings to both
the Subscriber and the entity who filed the Certificate Problem Report.

Note this is not an acknowledgement, it's a report on the findings. Ie - is the certificate incorrectly issued or cannot be determined (in which case, I think the certificate problem report is rejected).

After reviewing the facts and circumstances, the CA SHALL work with the Subscriber and any entity reporting
the Certificate Problem Report or other revocation-related notice to establish whether or not the certificate
will be revoked, and if so, a date which the CA will revoke the certificate. The period from receipt of the
Certificate Problem Report or revocation-related notice to published revocation MUST NOT exceed the time
frame set forth in Section 4.9.1.1.

Agreed that this says that the revocation period must be within the 24 hours/5 days of when the problem report is filed if the CA is made aware of an issue. There is plenty of discussion on this on both the Forum and other bugs. If the CA rejects the certificate problem report as non-actionable (won't revoke based on the cert problem report), then it's hard to argue that the 5 day revocation will start because the CA was not made aware of any problem (4.9.1.1.). However, I think this needs to be clearly spelled out in the response to the certificate problem report (will revoke or wont' revoke). I think this clarity is required under 4.9.5.

Looking at this particular bug and the response from Stephen, I think Stephen agreed that the certificate should be revoked, which means that the five day rule applied effective when John sent the notice. This does indeed mean we need to file a delayed revocation bug for this incident and refine the Quovadis revocation process.

To summarize so we can get closure on this issue:
The problem report filed on 30 June 2020, 12:22 UTC was acknowledged on 30 June 2020, 17:31 UTC (same day) by QuoVadis. However, the problematic part is the acknowledgement to action the certificate (for revocation) was confirmed on 1 July 2020, 20:37 UTC (~32hours after the problem report was received.) As Jeremy noted above from 4.9.5, the prelim report would need to be provided within 24 hours of receipt of the problem report. We acknowledge the need to automate the certificate problem reporting and revocation process so we default to compliance to prescribed times frames as described here: https://bugzilla.mozilla.org/show_bug.cgi?id=1639801#c14

Ben - anything you want us to do on this bug? Should we leave it open until we get the automated revocation process complete or just track progress on this bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1639801#c14?

Since this is Quovadis, we'll need to integrate them after the revocation system's scheduled Oct 1 release.

This bug was originally opened by George based on section 4.9.5 of the Baseline Requirements, which states, "Within 24 hours after receiving a Certificate Problem Report, the CA SHALL investigate the facts and circumstances related to a Certificate Problem Report and provide a preliminary report on its findings to both the Subscriber and the entity who filed the Certificate Problem Report." Mozilla states, "An incident arises any time a CA fails to comply with an applicable requirement found in the CA/Browser Forum’s Baseline Requirements ... an incident can arise from ... procedural or operational issues ...." https://wiki.mozilla.org/CA/Responding_To_An_Incident
This appears to be a procedural issue that DigiCert/QuoVadis continues working on -- timing re: compliance with responses to problem reporting.
Nobody has addressed what is considered a "preliminary report" and I don't think it's defined in the Baseline Requirements.
From the stated facts, QuoVadis sent an "acknowledgement" to George within 24 hours, but that wasn't a "preliminary report"? When QuoVadis sent the confirmation that the certificates would be revoked, did that constitute the "preliminary report"?
Is DigiCert addressing the "preliminary report" step as part of its "automated revocation process"?
For now I'll leave this open and set a "next update" for October 1.

Whiteboard: [ca-compliance] → [ca-compliance] Next Update 1-October 2020

Hey Ben. We decided it wasn't worth arguing about a preliminary report and what that means when the bug was opened. There was an acknowledgment of the email so you could say there was a preliminary report since it isn't defined. However, I wanted to do something better in the automated revocation system. Specifically, I want a report to go out automatically for key compromise after confirmation of the compromise. We are addressing the preliminary report step as part of the automated process.

Still working on the automated key ceremony and reporting tool. The key compromise tool is on track for Oct 1 with Quovadis integration shortly afterwards. We will then expand it to other revocation reasons.

We are currently testing the MVP of the key ceremony tool. Once this is live, we'll be hooking it up to Quovadis. We'll have Quovadis update their CPS when they start hooking up their systems to the key compromise tool.

The DigiCert portal for automated key compromise revocation is live. We will start looking at how to integrate this with Quovadis so they can use it.

Quovadis is now integrated with the automated revocation system. They also updated their CPS to specify how to report key compromise using the system. Same process as DigiCert certs.

Then can this bug be closed, I believe?

Yes - this is finished. Please close the bug. Thanks Ben

Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Flags: needinfo?(stephen.davidson)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.