Open Bug 1573937 Opened 2 months ago Updated 26 days ago

DigiCert/Verizon: Qualified 2019 Audit Statements

Categories

(NSS :: CA Certificate Compliance, task)

task
Not set

Tracking

(Not tracked)

People

(Reporter: kwilson, Assigned: brenda.bernal)

Details

(Whiteboard: [ca-compliance] - Next Update - 20-September 2019)

Attachments

(3 files)

Verizon received the following qualified audit statements.

WebTrust CA:
https://bug1479561.bmoattachments.org/attachment.cgi?id=9084779

WebTrust BR:
https://bug1479561.bmoattachments.org/attachment.cgi?id=9084780

WebTrust EV:
https://bug1479561.bmoattachments.org/attachment.cgi?id=9084781

The purpose of this bug is to track closure of all of the qualifications, and for the CA to provide incident reports for them.

For each qualification that was already reported in Bugzilla, it is fine to just reference that Bugzilla Bug number, and finish resolving in that bug.

Thanks Kathleen. We will shortly post a timeline of our actions for remediation and our plan forward on this audit issue with Verizon.

Incident report - Due to the ongoing efforts to work this issue, we will provide an update as frequently as we get new information. Below is the timeline known up to this point:

2019-07-01: DigiCert reminded Verizon that their audit was due by 28 July. Verizon acknowledged.
2019-07-10: Verizon notified DigiCert that the audit will likely be qualified. We also learned that their audit team lead was unavailable due to an urgent personal matter.
2019-07-22: Verizon explained that evidence gathering was affected by the audit team lead's absence and proactively proposed to get a letter from EY that states that the report will be delivered on 9 August.
2019-07-26: Verizon provided the letter from EY, DigiCert posted it to CCADB as a public comment on the CAs that were valid.
2019-07-26: DigiCert revoked 5 Verizon CAs, reducing the count of CAs in Verizon’s control to two. The two CAs are no longer issuing.
2019-08-12: Verizon's audits were received and posted to CCADB. DigiCert commented on Verizon's audit attachment Bugzilla acknowledging the findings and confirming that a remediation plan was required. Similar comments were posted to the CAs still valid in CCADB.
2019-08-12: DigiCert reiterated its Third Party CA Minimum Requirements Policy to Verizon. This policy states that we will revoke CAs that are the subject of qualified audits unless a prescribed plan is followed. Verizon is required to schedule a meeting with DigiCert to discuss the failure and the detailed plan and schedule for remediation. Within no greater than 45 days of notice of an Audit Failure, Verizon must produce a signed engagement letter for a “Point-in-Time” audit with a scope that explicitly covers each element of the remediation plan and is conducted in accordance with the audit requirements stated in our policy, which is a union of Mozilla, Microsoft, and CABF requirements. The Point-in-Time audit MUST be started no greater than 30 days from the scheduled remediation completion date.
2019-08-12: Verizon provides a draft remediation plan to DigiCert.
2019-08-13: Verizon schedules a meeting with DigiCert as required by our policy, to be held Monday, 19 August.
2019-08-14: This bug was opened.
2019-08-16: This draft timeline, which is not intended to substitute for a full incident report, was posted to share the timeline so far and the milestones.
2019-08-19: Verizon and DigiCert scheduled to meet to discuss the findings and the remediation plan, ensuring that the findings will not repeat and documenting proof of how Verizon will prevent new findings until these CAs are revoked.

Brenda: It's been 11 days since the last update, what happened with the 2019-08-19 meeting?
Jeremy: https://bugzilla.mozilla.org/show_bug.cgi?id=1566162#c9 and https://bugzilla.mozilla.org/show_bug.cgi?id=1566162#c10 indicated DigiCert was going to be providing weekly updates, based on Thursday meeting reviews. https://bugzilla.mozilla.org/show_bug.cgi?id=1566162#c12 suggests as well that you or Brenda are going to provide weekly updates here.

Flags: needinfo?(jeremy.rowley)
Flags: needinfo?(brenda.bernal)

As posted in the other bug on CAs (https://bugzilla.mozilla.org/show_bug.cgi?id=1566162), Verizon agreed to a revocation date of March 2020. However, this does not re-mediate the current audit findings. Verizon currently proposed an 11 step plan that solves their remediation by October 31st with milestones along the way. We do not feel this is aggressive enough considering historically the CA has not operated compliantly. We've asked for a more aggressive timeline to revoke the certificates or at least fix the compliance issues. I was hoping that we'd have an earlier date for revocation before needing to post an update, but I don't have anything yet. We're going to agree on a date this week.

The meeting went okay, but the timeframe for fixing some of these issues was disappointing with the first remediation not planned until mid-September.

Flags: needinfo?(jeremy.rowley)

Hey Ryan - separate thread to reply to you since I'd started that post before your comment. We assigned this one to me, and I should have assigned it to Brenda for updating the bug. All of these will be updated by her. Our meetings do happen weekly and we assign people to ensure the bugs are updated every 7-14 days, depending on the bug. In fact, we assign two people. I ended up working on the EV issues and trying to figure out the crt.sh/mozilla-disclosure issues instead, so bad prioritization. I'm going to have Brenda post more details than I shared above about the Verizon call since I realize that's simply not enough information.

As a follow up to Jeremy's update, we have notified Verizon of our intent to either revoke their intermediate certificates by September 10, 2019 or alternatively place the CAs on OneCRL by this Thursday, August 29. We will provide an update on which plan we are moving forward with by the end of this week.

Flags: needinfo?(brenda.bernal)

Assigning to Brenda per comment #5

Assignee: steve.medin → brenda.bernal

Verizon executives have reached out to us. We have received and are reviewing a counter proposal from Verizon for acceleration of their audit remediation and revocation timeline from what was noted in Comment 4. We will post another update once we can come to an agreement.

To confirm: You plan to revoke Verizon sooner than September 10, as mentioned in Comment #6?

If so, that sounds like a great plan, given the issues identified, the long-standing set of issues with the Verizon hierarchy, and considering Verizon’s lack of engagement here or proactive incident reporting regarding these issues. The risk of continued trust seems to far outweigh the benefits.

Given that it’s a US holiday on Monday, is it reasonable to expect DigiCert plans to revoke these next week, in order to address the risks?

Flags: needinfo?(brenda.bernal)

The counter proposal that we are evaluating does not involve revocation sooner than September 10th. Apologies if there was a misunderstanding. We are still in discussion with Verizon. They are working on aggressively completing their audit remediation within this month. They have also accelerated their revocation timeline to January (not September) from March 2020 for the full shutdown. We will post an update shortly after the long weekend.

Flags: needinfo?(brenda.bernal)

Could you help me understand why DigiCert would not revoke these outright? The set of issues are deeply concerning, both for Verizon and for DigiCert overseeing Verizon. Based on the information Verizon itself has shared to date - which is zero, which cannot be remedied after the fact (hence, “to date”) - this is certainly grounds for clear, decisive, and industry leading action. It would be a significant, if not impossible, bar to justify DigiCert not revoking.

Flags: needinfo?(brenda.bernal)

We are coordinating closely with Verizon’s senior leadership and they have reiterated to us that they understand the importance of addressing these issues quickly and completely. They have committed to have all issues remediated no later than September 19, 2019 and also to submit proof of the remediations to the auditor by that date. Verizon has committed to us that they are pushing to close the issues even sooner and will provide us with reports on their progress weekly, so they are openly sharing information. Verizon has stated that they are working towards a completed and clean point-in-time audit by September 19, 2019 as well. Further, Verizon is no longer issuing new certificates and had stopped issuing such certificates prior to May 1, 2018, which is the start of the audit period in scope. We also have discussed and agreed on a plan for the accelerated decommissioning of the SSL OnDemand platform, including revocation of the related CAs, by the end of January 2020, which will allow Verizon’s customers to have an orderly transition of services off of these CAs without adverse customer and end-user impact. To ensure adequate monitoring of progress, DigiCert will conduct on-site reviews of the Verizon PKI environment leading up to the final shut down date and receive a weekly status report from Verizon, effective immediately.

Regarding impact of immediate revocation, Verizon has informed us that as of September 10, it will have roughly 6,750 certificates active from these two CAs, which include Sub Alt Name including about 13,000 hosts. Roughly 750 certificates are expiring each month. Verizon is no longer issuing new certificates and had stopped issuing such certificates prior to May 1, 2018, which is the start of the audit period in scope. Due to the volume of hosts involved, replacing certificates in all hosts is a complex project, as Verizon and/or its customers would need to coordinate with the host and owner of the host, and the owner would need to generate Certificate Signing Requests, submit orders for replacement certificates, go through the required validation processes, retrieve signed certificates, and install them into each relevant host. Given the extensive efforts and coordination involved, this is not a task that can be completed in a few days, no matter how expeditiously all parties want to get this resolved.

We hope this update provides the community the level of detail that explains the situation with Verizon. Thank you.

Flags: needinfo?(brenda.bernal)

Brenda: thank you for this information. Please update this bug as you receive new information from Verizon, and no later than 20-September.

Whiteboard: [ca-compliance] Qualified Audits → [ca-compliance] - Next Update - 20-September 2019

Thanks Wayne. Per our quick synch yesterday with the Verizon team, the remediation effort is moving along as planned. We will provide a next update on the 20th when the PIT audit should be completed.

Attachment #9094063 - Attachment description: WTCA PIT audit letter 092019 → 19 - Verizon - WebTrust CA - PIT - Report of Independent Certified Publi....pdf

Verizon has successfully completed their Point-in-Time audit to confirm that their last annual audit's findings were fully remediated. I am attaching the signed copies of the PIT audit letters here.

We can also share in this update that as per Comment 12, we have agreed to a mid-November date for DigiCert's on-site review of Verizon’s PKI environment. This will help us gain assurance that the environment is operating as expected leading up to the final shut down date in January 2020. Verizon continues to keep on pace with the estimate of roughly 750 active certificates expiring per month; their count is at 614 certificates that have expired so far from the beginning of September to as of yesterday, Sept 18th.

We can provide the next update by December 2nd on the outcome of our onsite review.

You need to log in before you can comment on or make changes to this bug.