1590171 - QuoVadis: failure to reply to CPR in a timely manner

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Cynthia Revström]

On 2019-10-20 14:30 UTC, I sent the following report to [e-mail], as noted in CCADB.

---

Hello,

I am relying party who believes that there may have been a number of EV certs that have been improperly issued.

There has been a number of EV certs with businessCategory being "Non-Commercial Entity" despite not being intergovernmental organization as specified by 8.5.5. of the CA/Browser Forum EV Guidelines.

I have listed a few certificates below that I believe will illustrate the issue.

https://crt.sh/?q=32349b5b0be2ff3ba61615cbdf61445265a35d571f4bceb73933adcf551702c1
https://crt.sh/?q=a67d0ffe53c33f241e7929a9830c276f7755ec68b93f4ed2ab79e4c39ce1f3d2
https://crt.sh/?q=f5941a645ee81777434837a59cea7b13bb9d2085e2ce4fad0edf97956e33b518
https://crt.sh/?q=f38f7312b5d4ac7d34f1ff7df213fcac7ec51d5e7663d19d859da59db7a45f53

If this was improperly issued, please report the incident as described here: https://wiki.mozilla.org/CA/Responding_To_An_Incident

I have not received a response within 24 hours, as required by BRs 4.9.5.

Comment 1

[Ryan Sleevi]

Stephen: Can you provide an update about the delay?

It also sounds like there's a separate incident report to open, as it would certainly appear those certificates are misissued. Please open a new one regarding any misissuance for "Non-Commercial Entities"

Comment 2

[Stephen Davidson]

Hi Ryan - this feels like an effort to unreasonably proliferate disclosures. Is the Mozilla requirement that every problem report or revoked certificate result in a Bugzilla disclosure? Every crt.sh error?

This problem report was sent to QuoVadis yesterday at 13:30 UTC.
The report was received yesterday at 14:30 UTC by QuoVadis, and a response was sent yesterday to Cynthia Revstrom at 15:06 UTC.

Our team is working on it.

A prior report was received last week from Cynthia Revstrom relating to a single certificate. This did not come through the QuoVadis problem reporting channels but rather indirectly via a DIgiCert contact. I am not sure if Cynthia Revstrom received our response, however that certificate was investigated and revoked within the 5 days.
In the process, we believe we have other certificates with the same issue - including the below. We are investigating the extent of this issue and will report separately.

I request that this bug be marked invalid.

Comment 3

[Cynthia Revström]

I have looked through all the emails and spam again to make sure.
I have not received a reply for the issue I reported yesterday and the one 5 days ago was not much better.
All I got was:

---

Sorry for the slow response here. At first glance, this does look odd. It’s possible the entity is an agency under a non-commercial entity, I suppose, but I’d need to dig into the government records to confirm. I’ll share this with QV and have them send you an update with what they find.

---

I understand if you were unable to find my contact details for the one 5 days ago as that was sent through DigiCert, but you should have been able to reply to the one I sent yesterday.

Comment 4

[Ryan Sleevi]

(In reply to Stephen Davidson from comment #2)

Hi Ryan - this feels like an effort to unreasonably proliferate disclosures. Is the Mozilla requirement that every problem report or revoked certificate result in a Bugzilla disclosure? Every crt.sh error?

Could you highlight what is unreasonable about expecting CAs to report violations of the BRs/EVGs/Root Program requirements?

The proposed Policy 2.7 makes it clearer what the expectations are - https://github.com/mozilla/pkipolicy/blob/2.7/rootstore/policy.md - although I would hope that QuoVadis/DigiCert would be looking to rise above the "absolute minimum required".

Note that the inclusion within the report is wholly in-line with existing Mozilla expectations and requirements.

This problem report was sent to QuoVadis yesterday at 13:30 UTC.
The report was received yesterday at 14:30 UTC by QuoVadis, and a response was sent yesterday to Cynthia Revstrom at 15:06 UTC.

We've seen CAs provide such information in the past, and it revealed problems with their CA problem reporting process and sending e-mails. Could you please include the e-mail headers to help diagnose the error?

In the process, we believe we have other certificates with the same issue - including the below. We are investigating the extent of this issue and will report separately.

As noted in https://wiki.mozilla.org/CA/Responding_To_An_Incident , providing an incident report as soon as a CA understands there is an issue is strongly recommended (and under 2.7, clearly required). Your DigiCert partners are fully familiar with the expectations here, and so understanding if and when they were brought into the loop on this compliance matter is similarly useful in understanding (on that new issue).

Comment 5

[Stephen Davidson]

I apologise - I was having issues with webmail yesterday (as our email systems are in an awkward transition stage). My reply all response to Cynthia Revstrom seems to have only gone to the QuoVadis aliases, not to her. The fault is entirely mine. For what it's worth that email is attached.

From: Stephen Davidson
Sent: Sunday, October 20, 2019 12:06 PM ADT
To: Stephen Davidson (EXT) <[e-mail]>
Cc: [e-mail] <[e-mail]>
Subject: Re: Potential EVG violation
 
Hello Cynthia:

Thank you for your report.  Did you receive our reply to your earlier report regarding the National Library of Scotland certificate you reported to Client Wilson at DigiCert?

We acknowledge that this should have been categorized as Government Entity, and are replacing that certificate and others that were also incorrectly issued to the National Library of Scotland.

On Friday we commenced a review of the use of the Non-Commercial Entity category in QuoVadis certificates, as we know there are other certificates miss-classified, as you have indicated below.  

We will investigate the certificates you reported below within 24 hours and if verified replace these certificates within 5 days.

Regards, Stephen
QuoVadis

Comment 6

[Ryan Sleevi]

Thanks for the investigation.

I think, related to this, while human error is understandable, it's important to understand what processes may change / how incidents are responded to in the future, to prevent repeats of this issue. We've seen multiple CAs adopt multi-party reviews or checklists for incidents.

Comment 7

[Stephen Davidson]

Hi Ryan - I think you will acknowledge that QuoVadis has historically been proactive in making disclosures and in responding to reports. The reason I ask about disclosures, is there is a recent trend where security researchers are contacting us with individual problem reports and specifically requesting Bugzilla disclosure. All's fair, but your desire to have a complete resolution within 5 days is unreasonable. In the investigation of the original reported issue, a wider issue may be identified as is the case here. The immediate focus is to halt the proliferation of new certs with the issue, and then to identify and replace other affected certs. That normally will not happen in the tight 5 day cycle of the original problem report.

On a related point, last week I looked at crt.sh's issues list. There were more than 100 "error" certs spanning 14 different error zLint errors from 10 different CAs. I rarely see these showing up in Bugzilla disclosures nor being revoked.

Comment 8

[Ryan Sleevi]

(In reply to Stephen Davidson from comment #7)

Hi Ryan - I think you will acknowledge that QuoVadis has historically been proactive in making disclosures and in responding to reports. The reason I ask about disclosures, is there is a recent trend where security researchers are contacting us with individual problem reports and specifically requesting Bugzilla disclosure. All's fair, but your desire to have a complete resolution within 5 days is unreasonable. In the investigation of the original reported issue, a wider issue may be identified as is the case here. The immediate focus is to halt the proliferation of new certs with the issue, and then to identify and replace other affected certs. That normally will not happen in the tight 5 day cycle of the original problem report.

I think this is conflating a few things.

The BRs require the initial incident report in 24 hours. You either need to make a determination it is a problem (and planning for revocation) or make a determination it's not a problem (and that you'll not be revoking). 5 days is the timeframe for revocation (except in the cases where 24 hours is required).

So to recap:

• Consistent expectation that all problem reports will be responded to within 24 hours.
• Once you've determined an issue, consistent expectation that all certs identified (in the problem report || by the CA) will be revoked within 24 hours / 5 days

Now, because you already have a requirement to respond to an incident report within 24 hours, the expectation is you'll notify Mozilla/Root Programs via m.d.s.p. post/bug that there is an issue, and the immediate steps you're taking - for example, halting issuance until you can determine the scope of the issue. You're similarly expected to provide updates as you investigate and analyze the root cause.

It's not expected that you'll have all the details available within 24 hours, but the notification that provides sufficient insight into understanding the scope and risk is critical, along with ensuring that there is, by the end of things, a complete incident report that explains and understands root causes.

Your DigiCert partners are quite familiar with this and the expectations, and I would encourage QuoVadis to work closely, since these decisions similarly reflect DigiCert. It's not trying to set an unreasonable burden, but to ensure timely disclosure and scope of incidents, and timely responsiveness to externally-reported issues.

On a related point, last week I looked at crt.sh's issues list. There were more than 100 "error" certs spanning 14 different error zLint errors from 10 different CAs. I rarely see these showing up in Bugzilla disclosures nor being revoked.

You can and should file issues for CAs trusted in Mozilla's program, and/or report directly to the CA, which would then file issues. The goal is to ensure systemic resolution. Of course, not all CAs that show in crt.sh are trusted by Mozilla products, and in those cases, you should follow-up with the applicable root programs.

Comment 9

[Jeremy Rowley]

There's a whole lot going on in this thread, and I'll respond with as complete knowledge as I have.

1. Per the BRs, the expectation is that we'll respond to reporters within 24 hours of receiving incident reports and revoke certs within 24 hours/5 days of discovery of the mis-issuance. That's true for both Quovadis and DigiCert.

2. There is currently no data sharing between the companies which means I don't have information readily available (other than what they can easily share here) to address how many certificates are impacted by Cynthia's report or whether a timeline was met. However, I do acknowledge that we need to start merging the companies rather than running them as independent orgs. We were planning on doing that after finishing the legacy Symantec shut-down, but given then number of Quovadis incidents being filed recently, I'll look for ways to accelerate at least the TLS and process integration. We're working on an integration plan now. I'll share something hopefully by the end of the this month.

3. In the meantime, I'll have myself and other DigiCert people added to the Quovadis mailing list so we can help respond to incident reports. This will ensure better coverage of incidents than just Stephen Davidson and ensure against something going wrong with his email. This will also help us start tracking Quovadis problem report responses and revocation timelines in a more organized manner.

4. Brenda and I will work with Stephen to write figure out the root cause on what Cynthia reported. Stephen has all the data, but we'll work on writing up the report and filing the bug. We've already had him kick off an investigation on the number of certificates impacted.

5. What other CA's are doing is irrelevant to whether Quovadis has mis-issued a certificate or failed to respond to a reporter. We should send cert problem reports on what we see (or post the results of our investigation on a bug) but it doesn't excuse our own mis-issuance.

Comment 10

[Stephen Davidson]

Hi Ryan: As noted before, I responded to the problem report within an hour of receiving it. There was a mixup across internal email systems, and neither I nor others copied on the email caught it. As noted before, the original certificate reported by Cynthia Revstrom led to a greater investigation described below.

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

We received a problem report from Clint Wilson at DigiCert who had received an email from Cynthia Revstrom regarding a certificate issued to the National Library of Scotland with possible incorrect use of “Non-Commercial Entity” in businessCategory field.

https://crt.sh/?q=362705555bb4fa780a0050e96cbf6b682f2796c5335dc8edd6fd843618a4b003

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

14 Oct 2019 20:30 GMT Received problem report from Clint Wilson at DigiCert. Ideally problem reports should be directed to [e-mail].
15 Oct 2019 21:30 GMT Following investigation, instructions given to replace and revoke certificate. Response was sent to Cynthia Revstrom (discovered to have not been delivered following the opening of this bug).
16 Oct 2019 Additional investigation identifies more certificates at National Library of Scotland; instructions given to replace and revoke.
18 Oct 2019 Research started into larger issue with EV clients miscategorized as “Non-Commercial Entity.”
20 Oct 2019 14:30 GMT additional problem report received from Cynthia Revstrom to [e-mail].
20 Oct 2019 15:06 GMT Response sent by QuoVadis (discovered to have not been delivered following the opening of this bug). I have reached out to Cynthia Revstrom to apologise.
21 Oct 2019 12:53 GMT Initial certificate for National Library of Scotland confirmed revoked.
21 Oct 2019 23:30 GMT Following investigation, an additional 66 certificates were identified for revocation, customers contacted starting morning of 22 Oct 2019. The investigation continues.
22 Oct 2019 all 20 certificates at National Library for National Library of Scotland confirmed revoked.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

QuoVadis has reviewed all clients with the “Non-Commercial Entity” for EV businessCategory. We stopped issuing certificates resulting from the miscategorisation on 18 Oct 2019.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

There were 20 problem certificates issued to the National Library of Scotland certificates. The first was issued on 18 Feb 2018 and the last on 11 Jan 2019.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

The attached batch 1 includes the initial certificate reported by Cynthia Revstrom and others at the same customer. In our batch revalidation of clients tagged as “Non-Commercial Entity” we have identified another 66 certificates, attached as batch 2. This includes the certificates identified by Cynthia Revstrom in her problem report of 20 Oct 2019. The investigation continues.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

RAs believed that “Non-Commercial Entity” applied to charities and other non-commercial entities. This was not the specification laid out in QuoVadis training manuals.

We notice that this same “Non-Commercial Entity” miscategorisation is widespread amongst EV issuing CAs internationally.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

As part of the integration of QuoVadis into DigiCert, the QuoVadis compliance team now reports to DigiCert. In addition, an additional experienced internal auditor has been hired by DigiCert whose role includes the 3% review of QuoVadis TLS issuance.

Previously TLS validation was performed by separate national subsidiaries of QuoVadis for their respective client bases. Since the acquisition by DigiCert, QuoVadis validation has been centralized into one team with a group leader, and reports into the larger DigiCert validation group. Three highly experienced validation staff from DigiCert have been seconded to the group.

Short term, updated training has been provided to QuoVadis RAs relating to EV JOI and businessCategory fields.

Longer term, QuoVadis TLS issuance will move to DigiCert’s platforms, benefiting from DigiCert’s significant investment in automation and validation tools. A roadmap will be provided at a later date.

Comment 11

[Stephen Davidson]

Attachment: Batch 1

Comment 12

[Stephen Davidson]

Attachment: Batch 2

Comment 13

[Ryan Sleevi]

(In reply to Stephen Davidson from comment #10)

Hi Ryan: As noted before, I responded to the problem report within an hour of receiving it. There was a mixup across internal email systems, and neither I nor others copied on the email caught it. As noted before, the original certificate reported by Cynthia Revstrom led to a greater investigation described below.

Apologies if it was unclear, but the expectation here is that there are two bugs.

This bug should only deal with the failure to respond to Cynthia's 2019-10-20 report. The response in Comment #9 was useful to understanding mitigations, and Comment #5 was useful in understanding root cause.

A separate issue should be open for the incident discussed in the remainder of Comment #10 (and Comment #11 and Comment #12).

That is, there are two distinct incidents here:

• A failure to respond in the time required by the BRs
• A failure with validation of Non-commercial entities.

Each of these deserve distinct incident reports and analysis about root causes and mitigations.

Comment 14

[Stephen Davidson]

For the sake of consistency, QuoVadis confirms here the revocation of the certificates identified in Batch 2 by end of day 10/27. A further update will be provided this week.

Comment 15

[Stephen Davidson]

Attachment: Batch 3

For the sake of consistency, QuoVadis continues this disclosure here. As noted above we continued our investigation, and a third batch of certificates was identified. The issue was communicated to affected customers on Friday 25 October, and completion of the revocation of the certificates was completed by end of 30 October. We continue to investigate EV fields in QV-issued certificates using the enhanced tools and methodologies used by DigiCert.

Comment 16

[Ryan Sleevi]

Stephen: Please review Comment #13, which requested QuoVadis open a distinct issue regarding the failure to validate non-commercial entity as required. There should be two distinct issues here, and this one is about understanding the steps being taken to ensure timely responsiveness. I want to hold off closing this one out until it's clear that QuoVadis has filed an incident report regarding the underlying incident (the one whose response was delayed)

Comment 17

[Stephen Davidson]

Hi Ryan: Per your request we have opened a new bug at https://bugzilla.mozilla.org/show_bug.cgi?id=1593357.
We request the closure of this bug.

Comment 18

[Ryan Sleevi]

Wayne: Kicking this one to you.

We don't really have an incident report in a consistent form for the failure to respond, but we've got bits and pieces of what happened and what went wrong, mostly through Comment #5 and Comment #9.

In terms of remediations, Comment #9 spells out - DigiCert folks have been added to the QuoVadis problem reporting email, to make sure that DigiCert is in the loop on incident responses.

In terms of next steps, it was unclear if it's reasonable to request the standard Incident Response template for the failure to respond issue (Comment #10 isn't that). That's what I'm passing to you.

For the issue that QuoVadis failed to respond to, we'll move to Bug #1593357, and that's what Comment #10 covers.

Comment 19

[Wayne Thayer]

(In reply to Ryan Sleevi from comment #18)

Wayne: Kicking this one to you.
...
In terms of next steps, it was unclear if it's reasonable to request the standard Incident Response template for the failure to respond issue (Comment #10 isn't that). That's what I'm passing to you.

I do think it is reasonable and useful to request an incident report specific to the failure to respond within the timeline specified in the BRs.

Stephen: will you please do that?

For the issue that QuoVadis failed to respond to, we'll move to Bug #1593357, and that's what Comment #10 covers.

Comment 20

[Stephen Davidson]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

This bug was filed the day after Cynthia Revstrom’s email to us. We were already investigating the issue following an earlier communication between Cynthia Revstrom and Clint Wilson of DigiCert. A confirmation email was sent to Cynthia Revstrom within an hour of receipt, but due to user error she was dropped from the response.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

20 Oct 2019 11:29 ADT Problem report received from Cynthia Revstrom to [e-mail].
20 Oct 2019 12:07 ADT Response sent by myself to Cynthia Revstrom from @digicert.com address (discovered to have not been delivered following the opening of this bug).
21 Oct 2019 14:17 ADT This bug created.
21 Oct 2019 16:59 ADT Apology sent to Cynthia Revstrom including original response, with additional detail received from her at 17:03 ADT, and further response sent to her at 20:25 ADT.

See https://bugzilla.mozilla.org/show_bug.cgi?id=1590171#c10 and https://bugzilla.mozilla.org/show_bug.cgi?id=1593357 for more detail.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

NA

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

NA

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

NA

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

NA

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Cynthia Revstrom’s email was received on a Sunday morning to a QuoVadis email address. Due to a transition between mail systems, I forwarded the email from my QuoVadis mobile device to a DigiCert email address in order to respond in better detail from a laptop. To be clear, an attempt was made to respond within the hour of her problem report. Neither I, nor others copied on the email, realised that Cynthia Revstrom had been dropped from the CC list until her bug was filed the next day.

As reported elsewhere, QuoVadis is gradually transitioning from its own legacy systems to DigiCert systems. This transition includes a change from QuoVadis' legacy system for support ticketing to the DigiCert system. This project was in its closing stages during this occurrence and was completed on 28 Oct. This system is now in use for problem reports submitted via the QuoVadis website forms.

Comment 21

[Ryan Sleevi]

As far as incident responses go, this is... less than ideal.

This system is now in use for problem reports submitted via the QuoVadis website forms.

The problem report was sent via e-mail. So while this is useful to show a systemic approach (even though it's not stated as such), it's also problematic in that it doesn't address the root cause of the underlying issue.

The only saving grace here is Jeremy's Comment #9, which at least shows an attempt to mitigate the e-mail issue, when he stated:

In the meantime, I'll have myself and other DigiCert people added to the Quovadis mailing list so we can help respond to incident reports. This will ensure better coverage of incidents than just Stephen Davidson and ensure against something going wrong with his email. This will also help us start tracking Quovadis problem report responses and revocation timelines in a more organized manner.

From a systemic prevention point of view, one can easily see low-hanging fruit such as having a formalized incident response process, including someone responsible for developing and responding to the report, along with independent review /and/ independent confirmation that the steps were followed. This would, for example, have involved independent review to make sure this was sent to Cynthia, and perhaps involve a separate follow-up to confirm response.

I'm... not impressed by the incident report in Comment #20. It appears to be a bit of 'paint by numbers' and even then, forgot to color in three whole colors. But we know what the picture is supposed to look like, we know that art school may be a bit of a stretch, we know this isn't going up on the fridge or getting any gold stars, but... we also probably know enough to close this out and hope for better next time.

Comment 22

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.