1613505 - DigiCert: WTCA / WTBR Audit 2019 - Matters to be resolved

Below is our management response to the matters of emphasis in our WebTrust audit letters that did not qualify the corresponding report(s). For clarity, these were items identified during the audit, which closed on January 29, 2019, and therefore, were not already part of an open Bugzilla. As part of an audit closing activity, management has an opportunity to respond to observations and non-qualifying items outside of the audit reports published. For transparency, we are addressing the “matters” here by publishing the resolution for each item. We have closed all but one item (#7 below), which we will update once we finalize discussion/next step.

Offline Logs
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123455, Matter 2
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123447, Matter 2
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123452, Matter 2

Emphasis item: The auditors noted that we use an automated process for monitoring and alerting logs instead of a human review.
Resolution: After SC21 passed, we implemented automated log auditing and real-time review. The audit criteria have not been updated to reflect the ballot passing. As a further enhancement for auditability, we will create monthly tickets to document and affirm that Operations has reviewed the logs (both online and offline). We consider this matter resolved.

System Access
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123447 Matter 3
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123452, Matter 3
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123442, Matter 5

Emphasis item: System access reviews were performed at the network and physical access levels. For the first three quarters of the engagement period, access reviews did not include application level access or offline systems in the scope of the review.
Resolution: DigiCert has compensating controls that prevents unauthorized users from accessing the system as their network and physical accounts are disabled at termination. DigiCert has instituted a quarterly access review to include also all application level. Application access is granted and disabled via OKTA which is part of our standard review process. We consider this matter resolved.

Vulnerability Report Archival
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123452, Matter 4
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123455, Matter 4
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123447 Matter 4

Emphasis item: The auditors noted that historical vulnerability scan reports for four months were not retained due to DigiCert switching providers of the service from Qualys to Tenable.
Resolution: We demonstrated during the audit that we perform the required assessment (and any risks are captured in tickets for remediation). We scan every month, but only really are required to retain the findings and remediation. The CPS doesn't specify required retention of vulnerability scan data. This was a matter of emphasis because traceability to the vulnerability remediation to a report from the scan could not be achieved. Although there is direct requirement to hold the scan data, lack of access to the scan results made it difficult to test traceability. We consider this matter resolved.

clientAuth only certs
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123453, Matter 1

Emphasis item: The auditor noted that 4 certs did not conform to a section of the CPS. However, the Subject Common Name field conforms to subscriber's requirements for internal use, as noted.
Resolution: This is out of scope for Mozilla and not under BR coverage since these are clientauth certificates only. There is a reference in the STN CPS which allowed the usage of pseudonyms in non sMIME and clientAuth cert , which was not factored for one cert identified.
In addition, there is a clause in the STN CPS that talked about how these are validated for specific customers (footnote 1 of the CPS). There is inconsistency with the STN CPS and the client CPS that governs the usage of these certs (for the remaining three certs).
The certs are both out of scope for Mozilla and they do comply with the respective CPS.

AKI Extension in an ICA
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123452, Matter 1

Emphasis item: The auditors noted that one ICA did not include a Key ID of AKI extension.
Resolution: Although it included an AKI extension, the id was based on the issuer name and serial number. The auditors noted this as a non conformance to what was stated in RFC 5280, but we interpreted differently based on the following from 4.2.1.1 of RFC 5280:
" The authority key identifier extension provides a means of
identifying the public key corresponding to the private key used to
sign a certificate. This extension is used where an issuer has
multiple signing keys (either due to multiple concurrent key pairs or
due to changeover). The identification MAY be based on either the
key identifier (the subject key identifier in the issuer's
certificate) or the issuer name and serial number….”

The keyID is based on the issuer name and serial number in this cert for this CA. This is a matter of differing interpretation of the RFC. This ICA was issued from a legacy Symantec root that was assumed to no longer be trusted.

Self audits
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123445, Matter 2
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123444, Matter 1
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123442, Matter 4

Emphasis item: The auditors noted that our 3% audit were not conducted in a timely basis for the last quarter of the audit year end period.
Resolution: The internal audit team was focused on nearly 100% audit of issued, active certificates towards the latter part of last year for the Some-State and EV Jurisdiction of Information incidents; the audits were not focused on the full validation but on those specific issues.
A random sample of 100% is more than 3%, which is why we don't think this is an issue.
Therefore, we contend that our approach to an audit of the full population of active certs would have mitigated risks we were aware of in our issuance vs. a random sampling.

We also started using analytics to do the auditing rather than a manual review of each order. Using automated tools to review the 3% is not prohibited by either the guidelines or our CPS.

HSM Key Backup
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123443, Matter 4

Emphasis item: The auditors noted that some HSMs used for backing up offline keys are not operated at FIPS 140-2 Level 3 as per our CPS.
Resolution: This is still in investigation and pending resolution. DigiCert will provide an update.

BasicConstraints Criticality
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123456, Matter 1
Emphasis item: For 45 out of 45 certificates selected, the Basic Constraints criticality was false. However, in the Thawte CPS document, the basic constraints criticality was marked to be set as true.

Resolution: These are related to codesigning and timestamping end entity certs, as TLS has already migrated to DigiCert and its corresponding CPS. Furthermore, we believe there is no violation to our STN CP/CPS but an inconsistency, given this statements below:
STN CP: 7.1.2.4 Basic Constraints – (Thawte CPS chains to this CP), states….”The criticality field of this extension shall be set to TRUE for CA Certificates, but may be set to TRUE or FALSE for end-user Subscriber Certificates.”
Additionally, CABF Codesigning states for this field:
Codesigning and timestamping certificates --> D. basicConstraints (optional)
If present, the cA field MUST be set false

The next step is to amend the CPS for this discrepancy.

Flags: needinfo?(brenda.bernal)

Ryan Sleevi

Reporter

Comment 3

•

5 years ago

(In reply to Brenda Bernal from comment #2)

Offline Logs
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123455, Matter 2
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123447, Matter 2
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123452, Matter 2

Emphasis item: The auditors noted that we use an automated process for monitoring and alerting logs instead of a human review.
Resolution: After SC21 passed, we implemented automated log auditing and real-time review. The audit criteria have not been updated to reflect the ballot passing. As a further enhancement for auditability, we will create monthly tickets to document and affirm that Operations has reviewed the logs (both online and offline). We consider this matter resolved.

I'm not sure how I see the relationship, and I'm hoping you can explain further. The statement "Offline storage locations were not being reviewed for compliance with retention requirements" does not seem to comply with the modified SC21 either, namely:

Monitor the archival and retention of logs to ensure that logs are retained for the appropriate amount of time in accordance with the disclosed business practices and applicable legislation.

While SC21 did remove the phase Monitoring of archival and retention should be performed in the same manner as the system log integrity is monitored., this does not seem to be the same as what DigiCert is asserting here, and so I'm hoping you can connect the missing pieces here to explain further.

System Access
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123447 Matter 3
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123452, Matter 3
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123442, Matter 5

Emphasis item: System access reviews were performed at the network and physical access levels. For the first three quarters of the engagement period, access reviews did not include application level access or offline systems in the scope of the review.
Resolution: DigiCert has compensating controls that prevents unauthorized users from accessing the system as their network and physical accounts are disabled at termination. DigiCert has instituted a quarterly access review to include also all application level. Application access is granted and disabled via OKTA which is part of our standard review process. We consider this matter resolved.

Why wouldn't you review offline systems to ensure they are not accessed? There seems to be a meaningful distinction here between "physical account was disabled" and actually reviewing access to ensure that, well, things worked as expected. I'm not sure how the described control is a compensating control, as it seems relevant?

Vulnerability Report Archival
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123452, Matter 4
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123455, Matter 4
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123447 Matter 4
<SNIP>
Although there is direct requirement to hold the scan data, lack of access to the scan results made it difficult to test traceability. We consider this matter resolved.

Was this supposed to say there is not a direct requirement to hold the scan data? I'm having trouble making sense of this statement.

clientAuth only certs
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123453, Matter 1

Emphasis item: The auditor noted that 4 certs did not conform to a section of the CPS. However, the Subject Common Name field conforms to subscriber's requirements for internal use, as noted.
Resolution: This is out of scope for Mozilla and not under BR coverage since these are clientauth certificates only. There is a reference in the STN CPS which allowed the usage of pseudonyms in non sMIME and clientAuth cert , which was not factored for one cert identified.
In addition, there is a clause in the STN CPS that talked about how these are validated for specific customers (footnote 1 of the CPS). There is inconsistency with the STN CPS and the client CPS that governs the usage of these certs (for the remaining three certs).
The certs are both out of scope for Mozilla and they do comply with the respective CPS.

I don't think we consider such matters out of scope. Even for certificate profiles not trusted by Mozilla, the ability of the CA to follow its CP/CPS is extremely relevant to the trust of the organization, and so any divergences from the CP/CPS are relevant, and understanding how the situation happened and is remediated is important.

Could you please provide more specific references about why DigiCert believes these did, in fact, comply with its CP/CPS? The information provided appears too generic and dismissive of the issue, and I think we do want to treat this as an important issue to be very clear and precise on, as well as understanding what, if any, steps are being taken going forward. The inconsistency between the STN CPS and the client CPS, and how such issues weren't detected and are (or are not) being remediated seems relevant to general issues of delegating trust, and even for operation for CAs that no longer delegate trust and merely maintain multiple CP/CPSes.

AKI Extension in an ICA
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123452, Matter 1

Emphasis item: The auditors noted that one ICA did not include a Key ID of AKI extension.
<SNIP>
The keyID is based on the issuer name and serial number in this cert for this CA. This is a matter of differing interpretation of the RFC. This ICA was issued from a legacy Symantec root that was assumed to no longer be trusted.

I disagree that this is a matter of interpretation, and believe the auditors are correct and DigiCert's response is lacking here. I'm actually quite concerned about this response, because it matches other problematic/selective interpretations DigiCert has offered in the past, such as regarding underscores.

Immediately following the text DigiCert quoted is a clear and unambiguous requirement, which is further reflected in linting tools such as ZLint.

   The keyIdentifier field of the authorityKeyIdentifier extension MUST
   be included in all certificates generated by conforming CAs to
   facilitate certification path construction.

Given that you included such certificate within the scope of an audit that relies on RFC 5280, and that this is an intermediate CA, and thus not subject to the one exception listed, I'm curious to understand why this happened and what's being done to remedy this. In particular, this sounds like a certificate profile issue regarding the creation of the certificate, and given DigiCert's past issues with key ceremonies, that seems extremely relevant to understand.

Self audits
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123445, Matter 2
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123444, Matter 1
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123442, Matter 4

Emphasis item: The auditors noted that our 3% audit were not conducted in a timely basis for the last quarter of the audit year end period.
Resolution: The internal audit team was focused on nearly 100% audit of issued, active certificates towards the latter part of last year for the Some-State and EV Jurisdiction of Information incidents; the audits were not focused on the full validation but on those specific issues.
A random sample of 100% is more than 3%, which is why we don't think this is an issue.

I'm having trouble reconciling this statement, since as noted, the audits were not focused on the full validation, and thus not, in fact, audits according to the expectations of a self-audit.

This opens a broader concern in understanding what DigiCert does consider appropriate for a self-audit and what steps may be out of scope (for example, focusing on the validation process / full validation). I think it's great that DigiCert has been re-examining things, but the self-audit is an important defense in depth that seems not to be met here.

BasicConstraints Criticality
https://bug1458024.bmoattachments.org/attachment.cgi?id=9123456, Matter 1
Emphasis item: For 45 out of 45 certificates selected, the Basic Constraints criticality was false. However, in the Thawte CPS document, the basic constraints criticality was marked to be set as true.

Resolution: These are related to codesigning and timestamping end entity certs, as TLS has already migrated to DigiCert and its corresponding CPS. Furthermore, we believe there is no violation to our STN CP/CPS but an inconsistency, given this statements below:
STN CP: 7.1.2.4 Basic Constraints – (Thawte CPS chains to this CP), states….”The criticality field of this extension shall be set to TRUE for CA Certificates, but may be set to TRUE or FALSE for end-user Subscriber Certificates.”
Additionally, CABF Codesigning states for this field:
Codesigning and timestamping certificates --> D. basicConstraints (optional)
If present, the cA field MUST be set false

The next step is to amend the CPS for this discrepancy.

I don't think this explanation is necessarily reassuring. If the Thawte CPS stated a more restrictive requirement, and this more restrictive requirement was not met, the fact that the STN CP is more permissive should not be relevant here. Here again, despite the timestamping and code-signing uses, the concern and interest is understanding how the CP/CPSes align and are enforced. Given that the community directly relies on the CA's CP and CPS, issuance that does not align with them represents a serious and significant concern. If one CPS enumerated a set of methods [A, B, C, D, E] from those of 3.2.2.4 of the BRs, and the child CA's CP/CPS stated methods [A, B, C] are used, it would be a big deal if methods D or E were used by that child/subordinate CA.

Overall, I'm appreciative to BDO for bringing attention to these matters, as while they may not raise to the degree of a qualification in the auditor's professional opinion, they're extremely relevant to helping understanding systemic issues, challenges, or interpretative differences, so that we can all work to build more reliable systems. While I do challenge some of DigiCert's conclusions, I appreciate the answers, because the goal of this analysis is to make sure that there are no problematic trends, that there's a good trajectory in understanding, and that opportunities to improve or clarify requirements are not missed.

Flags: needinfo?(brenda.bernal)

Brenda Bernal

Assignee

Comment 4

•

5 years ago

Below are DigiCert's responses to Ryan from Comment 3 above:
1.) Offline Logs
The issue was we didn’t keep a record of the log review because we only keep 3 months of logs online and afterwards logs are kept offline, stored offsite. In order to provide proof we’ve reviewed everything for the full year, we needed to have some means of recording the review was done. This was stated as part of our resolution – create monthly tickets to document and affirm review of logs, which our auditors were in agreement with this resolution.

2.) System Access
The offline systems are stand-alone, non networked and accessed only through physical layers within a secured room. This does not pertain to a cold standby of online systems, whatsoever. If there is any turnover of limited personnel who have access to the secured space, those access would get addressed immediately.

3.) Vulnerability Report Archival
Yes, you are correct. This was meant to say “not”.

ClientAuth Certs
These clientauth certs were issued from old CAs from the Verisign Japan era. The four certs in question are from 2 customers (for the sake of simplifying – I will refer to them as Entity 1 and Entity 2)

Entity 1 (with 3 certs) issued off a Class 1 CA, and is covered with section 3.1.3 of the former STN CPS which states, “Class 1 subscribers may use pseudonyms. Unless when required by law or requested by a State or Government authority to protect the identity of certain end user subscribers (e.g., minors, or sensitive government employee information)”. However, the auditors were noting that in that same section, there is a reference to “Persona Not Validated” will need to be included in the CN for Class 1 individual Certificates. This customer is no longer issuing from this hierarchy and has moved to private hierarchy as of September 2019.

Entity 2 (with 1 cert) issued off a Class 2 CA and is covered by section 1 of the STN CPS which states, “1 Additionally, DigiCert issues organizational Client (non-SSL) certificates that are not subject to the CA Browser Forum Baseline Requirements. In addition to practices pertaining exclusively to the CA Browser Forum (ie, for OV SSL certificates), this CPS describes practices that pertain to any Class 2 or Class 3 certificate that is issued to an organization and contains organization information. Such certificates are referred to throughout this CPS as “organizational certificates”. We don’t consider this a divergence but a further explanation.

AKI Extension in an ICA
Still investigating; will update when we have next steps.
Self-Audits
Please let me clarify. I am stating that our self-audits were looking at the full validation. They have always been conducted on full validation. I am pointing out that there was a backlog on self-audits for the last quarter (which is what the auditors noted). The team’s cycles were focused on the investigation of our some-state and EV JOI issues, in addition to the self-audits performed. Because of the high demands to complete the analysis on those two incidents in a timely manner, plus the volumes of data to sift through, the self-audits lagged and were caught up after the high priority investigations completed.
Update on HSM Key Back up (comment 2 above)
The HSM was running in FIPS 140-2 level 3 but a recent firmware upgrade on that particular model was applied that was not submitted for Level 3 certification by the manufacturer. A prior version of the firmware was applied, which is Level 3 certified since this observation item was noted. According to our HSM manufacturer, although the firmware version is not a FIPS validated firmware due to the fact that this specific version wasn’t submitted for validation, it is still comparable to their closely related firmware versions that are NIST certified.
BasicConstraints Criticality
The issue is the alignment of the CPS to its CP which the latter states the proper value for criticality for basicConstraints, and were in alignment with the Baseline Requirements. With that said, we just finished the consolidation of the CPS into one document and have published that consolidated CP/CPS for all of DigiCert’s PKI, replacing all legacy Symantec. The basic constraint was in error in the CPS and that it should have been marked not critical.

Flags: needinfo?(brenda.bernal)

Jeremy Rowley

Comment 5

•

5 years ago

After further discussion, we decided that the interpretation behind issuing #5 was wrong and revoked the ICA. Are there any other questions about this bug?

Brenda Bernal

Assignee

Comment 6

•

5 years ago

As noted, we have revoked the ICA (item #5 above), and here's the information:
https://crt.sh/?id=1743356118. Revocation date: 2020-03-26

Ryan Sleevi

Reporter

Comment 7

•

4 years ago

Kathleen: I wasn't sure if audit issues go to you or Wayne

Flags: needinfo?(kwilson)

Kathleen Wilson

Comment 8

•

4 years ago

(In reply to Brenda Bernal from comment #6)

As noted, we have revoked the ICA (item #5 above), and here's the information:
https://crt.sh/?id=1743356118. Revocation date: 2020-03-26

I confirm that this cert has been revoked and indicated in the CCADB as Ready to Add to OneCRL.

I believe that all of the other items have been sufficiently addressed as well, so I will close this bug as fixed.

Thanks!

Status: ASSIGNED → RESOLVED

Closed: 4 years ago

Resolution: --- → FIXED

Kathleen Wilson

Updated

•

4 years ago

Flags: needinfo?(kwilson)

Nobody; OK to take it and work on it

Updated

•

2 years ago

Product: NSS → CA Program

David Lawrence [:dkl]

Updated

•

2 years ago

Whiteboard: [ca-compliance] → [ca-compliance] [audit-finding]

Bugzilla

Quick Search

DigiCert: WTCA / WTBR Audit 2019 - Matters to be resolved

Categories

(CA Program :: CA Certificate Compliance, task)

Tracking

(Not tracked)

People

(Reporter: ryan.sleevi, Assigned: brenda.bernal)

References

Details

(Whiteboard: [ca-compliance] [audit-finding])

Crash Data

Security

(public)

User Story

Description

Comment 1

Updated

Comment 2

Comment 3

Comment 4

Comment 5

Comment 6

Comment 7

Comment 8

Updated

Updated

Updated