1606031 - SecureTrust: BR Audit 2019 - matters to be resolved

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Kathleen Wilson]

There are some matters noted in SecureTrust's 2019 BR Audit, which do not appear to already have corresponding Bugzilla Bugs.

0. One item modified the auditor's opinion: human review of application and system logs ... were not being performed monthly.

There were 4 items listed that did not modify the auditor's opinion.

1. Bug #1551374

2. Bug #1546776

3. scope of vulnerability scanning process

4. quarterly self-assessment of at least 3 percent of certs

Corey, Please use this bug to provide an Incident Report for the items that do not already have corresponding Bugzilla Bugs.

Comment 1

[Corey Bonnell]

Thank you Kathleen. Due to the holidays in the US, we will provide a comprehensive Incident Report for these audit findings on the week of January 6th.

Comment 2

[Corey Bonnell]

How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

We contracted our auditor for WTCA, WTBR, WTCS, and WTEVSSL annual audits covering the dates October 1, 2018 through September 30, 2019.

They noted the following findings which did not impact their opinion:
A. The scope of SecureTrust’s vulnerability scanning process did not include all public and private IP addresses for the Certificate System for two of the four quarters during the engagement period.
B. One quarterly self-assessment completed by SecureTrust of at least three percent of certificates issued was not performed in a timely manner.

Our auditor noted the following finding, which impacted their opinion:
C. Human reviews of application and system logs to validate the integrity of logging processes and test the monitoring, logging, alerting, and log-integrity-verification functions are operating properly were not being performed monthly during the examination period.

A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

In September 2019, our auditor informed SecureTrust of their conclusion regarding the 2019 WTBR audit findings, two of which (items A +B) did not impact their opinion and one (item C) that impacted their opinion.

Upon learning of these findings, remediation plans were developed and put in place:
A. Remediation is ongoing; please see the detailed description below of why this control was deficient and our current remediation efforts.
B. Starting in the 2020 audit year (which commenced on October 1, 2019), the cadence of the internal audit of the issued certificate corpus was changed from quarterly to monthly.
C. Starting in the 2020 audit year , a monthly scheduled task to perform a human review of the integrity of the logging and alerting facilities was implemented.

Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

N/A

A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

N/A

The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

N/A

Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

A. No quarterly vulnerability scans were performed from within the High Security Zone network where the online production CA machines reside. SecureTrust’s opinion was that such scanning presents a significant risk to the security of the High Security Zone by either requiring ingress/egress permissions for an external vulnerability scanner to gain access to the network or by installing a scanning appliance in the High Security Zone, thereby increasing the attack surface of the machines on the network (by installing a device that does not have a role in the certificate issuance workflow) as well as necessitating additional ingress/egress firewall rules so that scans can be instrumented from the external network. Our opinion was that the strong compensating controls (such as strict firewall configuration, mutual TLS authentication for communication with the CA server, strong physical security measures to prevent tampering of equipment, etc.) in place to prevent unauthorized access to the High Security Zone were effective mitigations to ensure the security of the Zone such that a malicious actor would be unable gain access to the Zone and exploit a vulnerability.
B. The team responsible for performing the internal certificate audit changed between Q1 and Q2 2019. The timing of the audits was not coordinated between teams, and as a result, the Q1 and Q2 2019 audits were performed at roughly the same time.
C. Weekly human reviews of administrative access logs were being performed during the audit period, but there was no documented evidence of human reviews of other log types as required by WTBR control 4-3.5. Additionally, there was no documented evidence of manual monthly checks to ensure proper operation of the logging and alerting facilities.

List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

A. Remediation is ongoing. We have discussed this requirement with our auditors to explore remediation options that fulfill the requirement but also address our concerns regarding the impact such vulnerability scanning will have on the security posture of the High Security Zone. We are researching whether to install a scanning appliance in the High Security Zone or, alternately, temporarily allow a vulnerability scanner on the external network to gain access to the High Security Zone to perform a vulnerability scan to fulfill this requirement. The cross-functional remediation team will meet within two weeks to evaluate the results of this investigation and present the results to SecureTrust leadership and the Trustwave Certification Practice Board for approval of the preferred solution.
B. Starting in audit year 2020, we have changed our internal certificate audit cadence from quarterly to monthly. This ensures that we not have an audit gap regardless of timing or staffing issues. Additionally, the increased cadence allows issues to be investigated and remediated more quickly.
C. Starting in audit year 2020, we have scheduled a monthly task to perform a human review of the integrity of the logging and alerting facilities. Additionally, we are currently in the planning stages of automating this process to leverage the automation options afforded by version 1.3 of the Network Security Requirements with the passage of Ballot SC21.

Comment 3

[Ryan Sleevi]

Setting N-I for Corey for the follow-up on Issue A

The cross-functional remediation team will meet within two weeks to evaluate the results of this investigation and present the results to SecureTrust leadership and the Trustwave Certification Practice Board for approval of the preferred solution.

Comment 4

[Corey Bonnell]

The meeting with SecureTrust leadership took place late last week and it was decided that we will proceed with the installation of the vulnerability scanning appliance in the High Security Zone. Additionally, we have received approval from the Trustwave Certificate Practice Board to implement this solution. We have not yet determined an exact timeline for when the device will be installed, as we are still working out the logistics regarding procurement, configuration, and installation.

The schedule will likely be determined within the next few days, at which point I will update this bug with a more concrete timeline. Please expect an update the week of February 3rd, if not sooner.

Comment 5

[Corey Bonnell]

We have purchased the scanning appliance and are currently awaiting delivery, which is expected to happen in the next several days. We anticipate that we will configure and install the appliance in the High Security Zone by the end of February, barring unforeseen delays in delivery of the appliance. Targeting the end of February for the installation will ensure that there is ample time to perform vulnerability scanning and corresponding analysis of the scanning results within the current quarter. I will update this bug when installation has completed or if there is a deviation from the proposed plan.

Comment 6

[Ryan Sleevi]

That plan sounds good. I've updated the Next-Update to check in at EOM.

Comment 7

[Corey Bonnell]

While the firewall configuration for the High Security Zone was updated to allow for ingress/egress communication for the vulnerability scanning appliance and we were anticipating installing the device in the CA room this week, there has been a delay in the provisioning process. The reason for delay is that the appliance at delivery did not have the correct firmware image installed by the team responsible for initial configuration.

Standard practice calls for the team that handles such configuration to purchase, configure and ship a device to HQ. In this instance the process deviated from standard practice because the appliance was shipped directly to HQ from the vendor as opposed to having the provisioning team (which is located at another site) purchase and configure the appliance before shipment to HQ.

At present, the provisioning team is configuring an appliance to be shipped overnight to HQ. While installation will not be complete by February 28 as targeted, we remain confident that we will timely complete the quarterly vulnerability scanning of the High Security Zone required by the NSRs. I will update this bug within a week to confirm the anticipated installation date.

Comment 8

[Corey Bonnell]

As an update to my prior post, on Friday, February 28, the IT team at HQ worked with the provisioning team to configure the appliance already on-site at HQ, obviating the need to have an off-site appliance configured and shipped. The team entered the CA room on Monday, March 2 and completed installation of the appliance in the High Security Zone.

I will update this bug when the initial vulnerability scan has been scheduled. Please expect an update no later than March 11.

Comment 9

[Corey Bonnell]

There was an issue with the appliance installed in the CA room last week that made it unsuitable for vulnerability scanning of the room. Another appliance was requisitioned, configured, and installed this morning. Additionally, have completed the initial vulnerability scan of the High Security Zone network this afternoon, fulfilling this quarter’s requirement for vulnerability scanning of the High Security Zone.

We will routinely scan this network on at least a quarterly basis in accordance with the NSR requirement and associated WebTrust criteria.

Given this, I believe we have completed all audit remediation actions.

Comment 10

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.