1716670 - TWCA: Intermediate CA Certificate Missing from Audit Reports

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Ben Wilson]

TWCA's WebTrust audit reports do not list the following CA Certificate:

TWCA Global Root CA issued by TWCA Root Certification Authority
SHA256 hash of certificate:
8AD47F6D70A44FA80AF0F931125FFE3A76876FFAD219A4D40A13C038DC85E69E

https://crt.sh/?q=8AD47F6D70A44FA80AF0F931125FFE3A76876FFAD219A4D40A13C038DC85E69E

TWCA has indicated that it will remediate this issue with its next audit report on or before March 31, 2022, which is unacceptable to Mozilla.

Comment 1

[Hao-Chun Li]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP mailing list, a Bugzilla bug, or internal self-audit), and the time and date.

We have received email from Ben Wilson of Mozilla and Bugzilla regarding this problem.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

2021-01-22 The WebTrust audit is completed.
2021-03-09 We received the audit reports, but did not noticed the cross-signed CA certificate in question is missing from the report.
2021-03-20 We completed the CCADB audit update process.
2021-04-21 We received the April 2021 CA Survey and checked the ALV errors, but thought it was caused by our non-EV SSL issuing CA, which is EV-capable as per Mozilla Policy, missing from the EV audit report. We have respond to the survey indicating we will remediate that issue with our next regular audit.
2021-06-16 9:00 We became aware of this bug and started investigation.
2021-06-16 13:00 We have discussed with our auditor and confirmed the cause of this problem. We decided on a preliminary plan to provide a corrected audit report listing the cross-signed CA certificate.

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

We will ensure that this certificate is properly listed in the audit report in the future.

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

https://crt.sh/?q=8AD47F6D70A44FA80AF0F931125FFE3A76876FFAD219A4D40A13C038DC85E69E

5. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

https://crt.sh/?q=8AD47F6D70A44FA80AF0F931125FFE3A76876FFAD219A4D40A13C038DC85E69E

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

We did not check the audit reports sufficiently to detect this problem, and assumed the certificate is listed in the reports as in the past years.
We had another issue on ALV checks and did not realize the error means another certificate is missing from the reports.

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

We have discussed with our auditor and are working to provide the corrected audit report as soon as possible.
We will develop additional procedures to check audit reports properly listing CA certificates in the future.
I will post update after we further determined the remediation plan.

Comment 2

[Hao-Chun Li]

Attachment: WebTrust for CA audit report 2020 - updated

Comment 3

[Hao-Chun Li]

Attachment: WTCA BR audit report 2020 - updated

Comment 4

[Hao-Chun Li]

Attachment: WTCA EVSSL audit report 2020 - updated

Comment 5

[Hao-Chun Li]

Above are the updated audit reports including the missed CA certificate.

The procedure to check audit reports properly listing CA certificates is still being discussed.

Comment 6

[Ryan Sleevi]

1. When was this certificate first created?
2. What audit report was this CA first included in?
3. When was this CA first omitted from the audit reports?

My concern is primarily over this statement:

assumed the certificate is listed in the reports as in the past years

And I'm trying to ensure we have clear details about the sequence of audits this particular certificate was and was not included in.

I don't think this should be closed yet, because this is an incomplete incident report still. Comment #1 gives no direct timeline for when those new procedures will be determined, nor does Comment #6. https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report includes the following language:

For example, it’s not sufficient to say that “human error” of “lack of training” was a root cause for the incident, nor that “training has been improved” as a solution. While a lack of training may have contributed to the issue, it’s also possible that error-prone tools or practices were required, and making those tools less reliant on training is the correct solution. When training or a process is improved, the CA is expected to provide specific details about the original and corrected material, and specifically detail the changes that were made, and how they tie to the issue. Training alone should not be seen as a sufficient mitigation, and focus should be made on removing error-prone manual steps from the system entirely.

If the argument is that the right process was not followed, then we need a discussion of what the existing process was, and how that process has changed. It would seem that there's still no systemic improvements for

We did not check the audit reports sufficiently to detect this problem,

and that's critical to understanding how incidents like this will be prevented.

Comment 7

[Hao-Chun Li]

(In reply to Ryan Sleevi from comment #6)

1. When was this certificate first created?

This certificate is created in 2014 to facilitate the transition from the old RSA 2048 root (TWCA Root Certification Authority) to the new RSA 4096 root (TWCA Global Root CA).

We started the transition to issuing end-entity certificates from issuing CAs under the RSA 4096 root in 2014.
The cross-certificate (the certificate in question) is created to provide compatibility for old devices that do not update their trusted roots.
The certificate subject and public key are identical to the new RSA 4096 root's self-signed certificate, and is signed by the old RSA 2048 root.

2. What audit report was this CA first included in?

The cross-certificate has been in the scope of the audit report since 2014, when it was created.
Mozilla began to require that audit reports disclose all CA certificates in scope since 2017, and this certificate was listed in the 2017~2019 audit reports.

3. When was this CA first omitted from the audit reports?

The CA is in the audit scope, but the certificate details of the cross-certificate was missing from the 2020 audit report.
The audits cover both the issuing CA (the RSA 2048 root) and the CA itself of this cross-certificate, so the scope of the audit should not have been affected.

My concern is primarily over this statement:

assumed the certificate is listed in the reports as in the past years

And I'm trying to ensure we have clear details about the sequence of audits this particular certificate was and was not included in.

I don't think this should be closed yet, because this is an incomplete incident report still. Comment #1 gives no direct timeline for when those new procedures will be determined, nor does Comment #6. https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report includes the following language:

For example, it’s not sufficient to say that “human error” of “lack of training” was a root cause for the incident, nor that “training has been improved” as a solution. While a lack of training may have contributed to the issue, it’s also possible that error-prone tools or practices were required, and making those tools less reliant on training is the correct solution. When training or a process is improved, the CA is expected to provide specific details about the original and corrected material, and specifically detail the changes that were made, and how they tie to the issue. Training alone should not be seen as a sufficient mitigation, and focus should be made on removing error-prone manual steps from the system entirely.

If the argument is that the right process was not followed, then we need a discussion of what the existing process was, and how that process has changed. It would seem that there's still no systemic improvements for

We did not check the audit reports sufficiently to detect this problem,

and that's critical to understanding how incidents like this will be prevented.

We have further examined this incident and the updated response is as follows:

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
   We did not check the audit reports sufficiently to detect this problem, and assumed the certificate is listed in the reports as in the past years.
   We had another issue on ALV checks and did not realize the error means another certificate is missing from the reports.

The root cause is that we did not have a process to ensure that correct CA certificates are listed in the audit reports.

Our WebTrust audits cover both public-trusted root CA and all sub-CA under them (except for EV SSL one which excludes Issuing CAs with non-EV policy oid), so we falsely assumed that it is a trivial task to include all CA certificates in the hierarchy in the reports and relied on the auditor to do it correctly.
While we have communicated the requirements of Mozilla/CCADB with the auditor, we should have provided them with the exact list of certificates to be listed.

In the case of 2020 audit reports, our auditor removed the certificates which they thought are unnecessary. (Besides the cross-certificate, a self-signed root certificate which was never used and not trusted by Mozilla is removed.)
And then we did not check the audit reports sufficiently to detect the problem.

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

We will take following steps to ensure all certificates are listed in the future:

1. Maintain the expected CA certificate list for each WebTrust audit.
2. Provide the CA certificate list to the auditor and ensure the list is consistent with the audit scope.
3. Before the report is finalized, test the preliminary report with ALV and resolve unexpected errors.

Comment 8

[Ben Wilson]

I have updated the CCADB record for this CA - https://ccadb.force.com/001o000000nToL5AAK - by unchecking "Audit Same as Parent" and entering the URL for the three attachments to this bug. Next year, assuming this CA is included in the audit, the record should be updated to read "Audit Same as Parent." I believe this bug can be closed and will schedule it for closure on or about Friday, 9-July-2021.

Comment 9

[Hao-Chun Li]

Attachment: TWCA_WTCA_Audit Report_2021(Browser).pdf

Comment 10

[Hao-Chun Li]

Attachment: TWCA BR_Audit Report_2021(Browser).pdf

Comment 11

[Hao-Chun Li]

Attachment: TWCA_EVSSL_Audit Report_2021(Browser).pdf

Comment 12

[Ben Wilson]

See my email about the ALV process having trouble reading images (conversion using OCR) of SHA256 hashes in PDFs. That is why we have this requirement:
"MUST: be encoded in the document (PDF) as text searchable, not an image" in https://www.ccadb.org/policy#51-audit-statement-content

Comment 13

[Hao-Chun Li]

The PDF files uploaded are now text searchable.
I have changed the links in the CCADB audit update case to links of these files.

Comment 14

[Ben Wilson]

Greetings Hao-Chun,
Has your auditor indicated whether the files on CPA Canada will be updated? If so, when?
If not, then can we expect that this issue will not reoccur in next year's audit reports?
Thanks,
Ben

Comment 15

[Hao-Chun Li]

Hi Ben,

We have checked with our auditor and been informed that updating the files on CPA Canada would be equal to applying for new seals and costly, so if it is acceptable to Mozilla, we do not plan to update the files.

We definitely will make sure this issue will not reoccur in the next year.

Thank you,
Hao-Chun Li

Comment 16

[Ben Wilson]

Thanks for checking.

Comment 17

[Hao-Chun Li]

Attachment: TWCA_WTCA_Audit Report_2022(Browser).pdf

preliminary audit report for ALV test

Comment 18

[Hao-Chun Li]

Attachment: TWCA BR_Audit Report_2022(Browser).pdf

preliminary audit report for ALV test

Comment 19

[Hao-Chun Li]

Attachment: TWCA_EVSSL_Audit Report_2022(Browser).pdf

preliminary audit report for ALV test

Comment 20

[Hao-Chun Li]

Attachment: TWCA_WTCA_Audit Report_2022(Browser).pdf

Comment 21

[Hao-Chun Li]

Attachment: TWCA BR_Audit Report_2022(Browser).pdf

Comment 22

[Hao-Chun Li]

Attachment: TWCA_EVSSL_Audit Report_2022(Browser).pdf