1573937 - DigiCert / Verizon: Qualified 2019 Audit Statements

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Kathleen Wilson]

Verizon received the following qualified audit statements.

WebTrust CA:
https://bug1479561.bmoattachments.org/attachment.cgi?id=9084779

WebTrust BR:
https://bug1479561.bmoattachments.org/attachment.cgi?id=9084780

WebTrust EV:
https://bug1479561.bmoattachments.org/attachment.cgi?id=9084781

The purpose of this bug is to track closure of all of the qualifications, and for the CA to provide incident reports for them.

For each qualification that was already reported in Bugzilla, it is fine to just reference that Bugzilla Bug number, and finish resolving in that bug.

Comment 1

[Brenda Bernal]

Thanks Kathleen. We will shortly post a timeline of our actions for remediation and our plan forward on this audit issue with Verizon.

Comment 2

[Brenda Bernal]

Incident report - Due to the ongoing efforts to work this issue, we will provide an update as frequently as we get new information. Below is the timeline known up to this point:

2019-07-01: DigiCert reminded Verizon that their audit was due by 28 July. Verizon acknowledged.
2019-07-10: Verizon notified DigiCert that the audit will likely be qualified. We also learned that their audit team lead was unavailable due to an urgent personal matter.
2019-07-22: Verizon explained that evidence gathering was affected by the audit team lead's absence and proactively proposed to get a letter from EY that states that the report will be delivered on 9 August.
2019-07-26: Verizon provided the letter from EY, DigiCert posted it to CCADB as a public comment on the CAs that were valid.
2019-07-26: DigiCert revoked 5 Verizon CAs, reducing the count of CAs in Verizon’s control to two. The two CAs are no longer issuing.
2019-08-12: Verizon's audits were received and posted to CCADB. DigiCert commented on Verizon's audit attachment Bugzilla acknowledging the findings and confirming that a remediation plan was required. Similar comments were posted to the CAs still valid in CCADB.
2019-08-12: DigiCert reiterated its Third Party CA Minimum Requirements Policy to Verizon. This policy states that we will revoke CAs that are the subject of qualified audits unless a prescribed plan is followed. Verizon is required to schedule a meeting with DigiCert to discuss the failure and the detailed plan and schedule for remediation. Within no greater than 45 days of notice of an Audit Failure, Verizon must produce a signed engagement letter for a “Point-in-Time” audit with a scope that explicitly covers each element of the remediation plan and is conducted in accordance with the audit requirements stated in our policy, which is a union of Mozilla, Microsoft, and CABF requirements. The Point-in-Time audit MUST be started no greater than 30 days from the scheduled remediation completion date.
2019-08-12: Verizon provides a draft remediation plan to DigiCert.
2019-08-13: Verizon schedules a meeting with DigiCert as required by our policy, to be held Monday, 19 August.
2019-08-14: This bug was opened.
2019-08-16: This draft timeline, which is not intended to substitute for a full incident report, was posted to share the timeline so far and the milestones.
2019-08-19: Verizon and DigiCert scheduled to meet to discuss the findings and the remediation plan, ensuring that the findings will not repeat and documenting proof of how Verizon will prevent new findings until these CAs are revoked.

Comment 3

[Ryan Sleevi]

Brenda: It's been 11 days since the last update, what happened with the 2019-08-19 meeting?
Jeremy: https://bugzilla.mozilla.org/show_bug.cgi?id=1566162#c9 and https://bugzilla.mozilla.org/show_bug.cgi?id=1566162#c10 indicated DigiCert was going to be providing weekly updates, based on Thursday meeting reviews. https://bugzilla.mozilla.org/show_bug.cgi?id=1566162#c12 suggests as well that you or Brenda are going to provide weekly updates here.

Comment 4

[Jeremy Rowley]

As posted in the other bug on CAs (https://bugzilla.mozilla.org/show_bug.cgi?id=1566162), Verizon agreed to a revocation date of March 2020. However, this does not re-mediate the current audit findings. Verizon currently proposed an 11 step plan that solves their remediation by October 31st with milestones along the way. We do not feel this is aggressive enough considering historically the CA has not operated compliantly. We've asked for a more aggressive timeline to revoke the certificates or at least fix the compliance issues. I was hoping that we'd have an earlier date for revocation before needing to post an update, but I don't have anything yet. We're going to agree on a date this week.

The meeting went okay, but the timeframe for fixing some of these issues was disappointing with the first remediation not planned until mid-September.

Comment 5

[Jeremy Rowley]

Hey Ryan - separate thread to reply to you since I'd started that post before your comment. We assigned this one to me, and I should have assigned it to Brenda for updating the bug. All of these will be updated by her. Our meetings do happen weekly and we assign people to ensure the bugs are updated every 7-14 days, depending on the bug. In fact, we assign two people. I ended up working on the EV issues and trying to figure out the crt.sh/mozilla-disclosure issues instead, so bad prioritization. I'm going to have Brenda post more details than I shared above about the Verizon call since I realize that's simply not enough information.

Comment 6

[Brenda Bernal]

As a follow up to Jeremy's update, we have notified Verizon of our intent to either revoke their intermediate certificates by September 10, 2019 or alternatively place the CAs on OneCRL by this Thursday, August 29. We will provide an update on which plan we are moving forward with by the end of this week.

Comment 7

[Wayne Thayer]

Assigning to Brenda per comment #5

Comment 8

[Brenda Bernal]

Verizon executives have reached out to us. We have received and are reviewing a counter proposal from Verizon for acceleration of their audit remediation and revocation timeline from what was noted in Comment 4. We will post another update once we can come to an agreement.

Comment 9

[Ryan Sleevi]

To confirm: You plan to revoke Verizon sooner than September 10, as mentioned in Comment #6?

If so, that sounds like a great plan, given the issues identified, the long-standing set of issues with the Verizon hierarchy, and considering Verizon’s lack of engagement here or proactive incident reporting regarding these issues. The risk of continued trust seems to far outweigh the benefits.

Given that it’s a US holiday on Monday, is it reasonable to expect DigiCert plans to revoke these next week, in order to address the risks?

Comment 10

[Brenda Bernal]

The counter proposal that we are evaluating does not involve revocation sooner than September 10th. Apologies if there was a misunderstanding. We are still in discussion with Verizon. They are working on aggressively completing their audit remediation within this month. They have also accelerated their revocation timeline to January (not September) from March 2020 for the full shutdown. We will post an update shortly after the long weekend.

Comment 11

[Ryan Sleevi]

Could you help me understand why DigiCert would not revoke these outright? The set of issues are deeply concerning, both for Verizon and for DigiCert overseeing Verizon. Based on the information Verizon itself has shared to date - which is zero, which cannot be remedied after the fact (hence, “to date”) - this is certainly grounds for clear, decisive, and industry leading action. It would be a significant, if not impossible, bar to justify DigiCert not revoking.

Comment 12

[Brenda Bernal]

We are coordinating closely with Verizon’s senior leadership and they have reiterated to us that they understand the importance of addressing these issues quickly and completely. They have committed to have all issues remediated no later than September 19, 2019 and also to submit proof of the remediations to the auditor by that date. Verizon has committed to us that they are pushing to close the issues even sooner and will provide us with reports on their progress weekly, so they are openly sharing information. Verizon has stated that they are working towards a completed and clean point-in-time audit by September 19, 2019 as well. Further, Verizon is no longer issuing new certificates and had stopped issuing such certificates prior to May 1, 2018, which is the start of the audit period in scope. We also have discussed and agreed on a plan for the accelerated decommissioning of the SSL OnDemand platform, including revocation of the related CAs, by the end of January 2020, which will allow Verizon’s customers to have an orderly transition of services off of these CAs without adverse customer and end-user impact. To ensure adequate monitoring of progress, DigiCert will conduct on-site reviews of the Verizon PKI environment leading up to the final shut down date and receive a weekly status report from Verizon, effective immediately.

Regarding impact of immediate revocation, Verizon has informed us that as of September 10, it will have roughly 6,750 certificates active from these two CAs, which include Sub Alt Name including about 13,000 hosts. Roughly 750 certificates are expiring each month. Verizon is no longer issuing new certificates and had stopped issuing such certificates prior to May 1, 2018, which is the start of the audit period in scope. Due to the volume of hosts involved, replacing certificates in all hosts is a complex project, as Verizon and/or its customers would need to coordinate with the host and owner of the host, and the owner would need to generate Certificate Signing Requests, submit orders for replacement certificates, go through the required validation processes, retrieve signed certificates, and install them into each relevant host. Given the extensive efforts and coordination involved, this is not a task that can be completed in a few days, no matter how expeditiously all parties want to get this resolved.

We hope this update provides the community the level of detail that explains the situation with Verizon. Thank you.

Comment 13

[Wayne Thayer]

Brenda: thank you for this information. Please update this bug as you receive new information from Verizon, and no later than 20-September.

Comment 14

[Brenda Bernal]

Thanks Wayne. Per our quick synch yesterday with the Verizon team, the remediation effort is moving along as planned. We will provide a next update on the 20th when the PIT audit should be completed.

Comment 15

[Brenda Bernal]

Attachment: 19 - Verizon - WebTrust CA - PIT - Report of Independent Certified Publi....pdf

Comment 16

[Brenda Bernal]

Attachment: 19 - Verizon - WebTrust EV SSL - PIT - Report of Independent Certified P....pdf

Comment 17

[Brenda Bernal]

Attachment: 19 - Verizon - WebTrust SSL - PIT - Report of Independent Certified Publ....pdf

Comment 18

[Brenda Bernal]

Verizon has successfully completed their Point-in-Time audit to confirm that their last annual audit's findings were fully remediated. I am attaching the signed copies of the PIT audit letters here.

We can also share in this update that as per Comment 12, we have agreed to a mid-November date for DigiCert's on-site review of Verizon’s PKI environment. This will help us gain assurance that the environment is operating as expected leading up to the final shut down date in January 2020. Verizon continues to keep on pace with the estimate of roughly 750 active certificates expiring per month; their count is at 614 certificates that have expired so far from the beginning of September to as of yesterday, Sept 18th.

We can provide the next update by December 2nd on the outcome of our onsite review.

Comment 19

[Brenda Bernal]

As mentioned, we would provide our next update on Dec 2nd at the conclusion of our onsite review of Verizon operations:

In November 2019, DigiCert Compliance representative attended onsite at the Verizon operations Centre in Brussels, Belgium.
The main objective of this review was to confirm the remediations from the audit findings have been completed.
DigiCert also wanted to confirm compliance with the cease issuance agreement, track certificate revocation/expiration is in line with the plan to have all CA’s revoked by 31st Jan 2020 and to confirm operational security for the CA.

Regarding the 5 observations from the Verizon WebTrust audit (performed by EY), we were able to go through the updated documentation and processes to confirm that these issues have been resolved. Interviews with staff members showed that these processes are more than just documented but enforced and in use currently. Using direct database queries DigiCert was able to confirm that no end user certificates have been issued in the period since 30th April 2019 (a OCSP response has been signed to continue OCSP operations). Using the same query through 1st of Jan 2019 returned multiple certificates confirming correctness of the query.

To confirm attrition of certificates, the CRL was downloaded from the external web site, and this was confirmed via the database as the most current. The CRL was parsed via openSSL and entries were confirmed as revoked recently. Samples of these certificates were manually checked via the database to confirm status and that results matched. Current valid certificates were found to be 577 less than we had last been notified 2 weeks prior by Verizon, demonstrating a drop in certificates as per expectations to meet the January 2020 target shut down date.

A physical walk though was conducted confirming access controls, video capture for the data center.
This onsite review was completed with a focus on operational and contractual compliance with Verizon’s obligations to DigiCert as the Root CA and to validate the fixes with the findings of the WebTrust audit.

Items outside the above aforementioned including controls tested during a WebTrust audit were not in scope for this compliance onsite review. We are aware of the CPS discrepancy issue which is being handled separately via https://bugzilla.mozilla.org/show_bug.cgi?id=1596931

DigiCert is confident that Verizon remains on target for their January 2020 shut down. They are being proactive about their operation of the CA and have committed to filing a close out audit after the CAs have been revoked.

Our next update will be 15-January-2020 on progress before the shutdown.

Comment 20

[Wayne Thayer]

Brenda: thank you for the update. It sounds as if things are progressing according to plan.

Comment 21

[Brenda Bernal]

Here is the progress report on the Verizon PKI shut down activities:

1.   The plan is still on target to revoke remaining issuing CAs by end of January (specifically 29-January is when this is scheduled).
   

2.   Coordination of key destruction and database archival process to be executed by end of Feb 2020.
   

3.   Planning for long-lived CRL generation for the existing CAs to be revoked by end of January.
   

4.   Arrangement for internal auditor to witness step 2) above.
   

The next update will be on or shortly after 29-January, after the revocation completion.

Comment 22

[Brenda Bernal]

The CAs for Verizon has been revoked. Here are the crt.sh links with the revocation updated:
https://crt.sh/?id=4498824
https://crt.sh/?id=7309070

At this point, we would like to request for this bug to be closed given the completion of the revocations.

Comment 23

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.