1599571 - TrustCor: Non-revocation of CA certificates within 7 days

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Neil Dunbar]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36

Steps to reproduce:

We’ve learned that two (2) of TrustCor’s Subordinate CA certificates
needed to be disclosed in the SSL-BR Audit report. These certificates
were always disclosed in the standard WebTrust for CAs report, but
indeed were not also listed in our SSL Baseline report. The discrepancy
was due to a misunderstanding of the BRs and the WebTrust for CAs - SSL
Baseline with Network Security standards. We initially considered these
CA certificates to be out of scope, since they were intended for S/MIME
issuance only.

However, since these certificates were issued prior to the BR
technically constrained requirement, then technically speaking there is
a possibility that non-constrained certificates could be used to verify
SSL certificates, even if they were improperly signed using the S/MIME
Subordinate CA private keys. TrustCor recognises that this failure to
disclose in both audits represents a violation of BR sections 8.1 (the
remediation of which is discussed in Bugzilla ticket
1599503). Ordinarily this would call for a revocation of the Subordinate
CAs within 24 hours; failure to do so within 7 days represents a further
violation of BR section 4.9.1.2. This ticket is an explanation of why
those revocations did not take place within the initial 7 days of
discovery.

1. How your CA first became aware of the problem (e.g. via a problem
   report submitted to your Problem Reporting Mechanism, a discussion
   in mozilla.dev.security.policy, a Bugzilla bug, or internal
   self-audit), and the time and date.

A post to the Mozilla Dev Security Policy group from Kathleen Wilson on
2019-10-08 alerted TrustCor CA to discrepancies between the basic audit
report and SSL-BR audit report.

2. A timeline of the actions your CA took in response. A timeline is
   a date-and-time-stamped sequence of all relevant events. This may
   include events before the incident was reported, such as when a
   particular requirement became applicable, or a document changed, or
   a bug was introduced, or an audit was done.

(Times are in UTC)

2019-10-09 09:00:00: Initial investigation to establish the scope of the original issue (ALV discrepancies)
2019-10-10 14:30:00: Formal suspension of the Enhanced Secure Email CA program until a resolution could be determined.
2019-10-12 16:00:00: TCPA requests CA Administrator to reconcile all HSM activities using Email CA private keys since November 2015 with issuance logs for S/MIME CAs and report any discrepancies. TCPA takes the decision that immediate revocation of both CA certificates would harm TrustCor CA customers unduly.
2019-11-15 15:30:00: Completion of self audit establishing that the Email CA private keys have not signed any SSL certificates. Forwarded results to independent auditor.
2019-11-20 14:22:00: TCPA takes the notion that revocation of the Subordinate CAs would harm TrustCor CA customers without a balancing decrease of risk to browser users of a theoretical SSL certificate existing. Resumption of the Enhanced Secure Email CA program begins.

3. Whether your CA has stopped, or has not yet stopped, issuing
   certificates with the problem. A statement that you have will be
   considered a pledge to the community; a statement that you have not
   requires an explanation.

These are CA certificates rather than end-entity ones. No non-compliant
certificates under the hierarchy of the Email Subordinate CAs have been
noted. Stopping the entire S/MIME program for TrustCor CA would affect
many TrustCor CA customers unduly.

4. A summary of the problematic certificates. For each problem:
   number of certs, and the date the first and last certs with that
   problem were issued.

The 2 CA certificates are:

Basic Secure Email CA:

Subject DN: CN=TrustCor Basic Secure Email (CA1); OU=TrustCor Network; O=TrustCor Systems S. de R.L.; C=PA
Issuer DN: CN=TrustCor RootCert CA-1; OU=TrustCor Certificate Authority; O=TrustCor Systems S. de R.L.; C=PA
Serial Number: 00A60D883219A3FD59

Enhanced Secure Email CA:

Subject DN: CN=TrustCor Enhanced Secure Email (CA2); OU=TrustCor Network; O=TrustCor Systems S. de R.L.; C=PA
Issuer DN: CN=TrustCor RootCert CA-2; OU=TrustCor Certificate Authority; O=TrustCor Systems S. de R.L.; C=PA
Serial Number: 0AF3E61240471752

5. The complete certificate data for the problematic
   certificates. The recommended way to provide this is to ensure each
   certificate is logged to CT and then list the fingerprints or crt.sh
   IDs, either in the report or as an attached spreadsheet, with one list
   per distinct problem.

Fingerprints and data:

(Basic Secure Email CA): 02BEF922B32D46DFE7520B0EE7E3EAF588EE2B9CAB81B84837E6B955E0759A90
crt.sh: https://crt.sh/?id=170664054

(Enhanced Secure Email CA): A6D365161B58539CB44B29D77C648126F33DB3C493116C3040E18DE3E01A4242
crt.sh: https://crt.sh/?id=170664056

6. Explanation about how and why the mistakes were made or bugs
   introduced, and how they avoided detection until now.

There was no failure to detect: the determination to revoke was put on
hold until a complete risk assessment could be made as to the scope of
harm to TrustCor CA users as well as browser users. As for the ALV
discrepancy, that is discussed in another ticket.

7. List of steps your CA is taking to resolve the situation and ensure
   such issuance will not be repeated in the future, accompanied with a
   timeline of when your CA expects to accomplish these things.

At the instruction of the TrustCor Policy Authority, once the ALV
discrepancy was noted, the CA Administrator was required to perform an
audit of the HSM logs for the S/MIME CA private keys to establish that
no object was signed using those keys for any purpose other than either
issuing S/MIME certificates, or issuing certificate status information
regarding said S/MIME certificates.

This audit was completed recently and showed that no non-S/MIME related
activity has ever taken place since these private keys were
generated. The results of this self-audit was provided to TrustCor’s
independent auditor.

The TCPA has decided that revoking the Subordinate CAs immediately would
cause high levels of business disruption to TrustCor’s customers who are
not in a position to wholesale replace their Email protection
certificates quickly. And, since we can confirm that no SSL certificate
has ever been issued under the S/MIME CA certificates, the ongoing risk
to Relying Parties is minor.

The two Email Subordinate CAs are now included in the scope WebTrust for
CAs (Baseline Requirements and SSL Network Security) audit.

Note: in ticket 1599503 we have already requested that the Secure Email
CA Certificates are added to OneCRL in partial mitigation of risk to
Mozilla customers - though, of course, we acknowledge that this does not
affect any SSL validating software which does not consult OneCRL.

We will disclose to our auditors the existence of these tickets and the
resolutions thereof; these will be mentioned in the 2020 audit reports.

In the future, armed with more correct understanding of the disclosure
requirements, we hope to avoid any repetition of this compliance issue.

We are hopeful that this strikes the correct balance of care towards
TrustCor CA customers, as well as minimising the risk towards any users
of browsers which trust TrustCor CA’s root certificates.

Comment 1

[Ryan Sleevi]

The TCPA has decided that revoking the Subordinate CAs immediately would
cause high levels of business disruption to TrustCor’s customers who are
not in a position to wholesale replace their Email protection
certificates quickly. And, since we can confirm that no SSL certificate
has ever been issued under the S/MIME CA certificates, the ongoing risk
to Relying Parties is minor.

Is the understanding correct that TrustCor has no plans to abide by its CP/CPS or the Baseline Requirements, because compliance is inconvenient and TrustCor has deemed compliance undesirable to its business interests?

While I appreciate the description of the (internal) audit procedures, such a description is not reliable for relying parties that expect and have trusted TrustCor to abide by its stated policies. This does seem rather concerning, especially given that other CAs, recognizing this, have moved to enact revocation of such certificates, either through replacement with constrained intermediates or through replacement of the leaf certificates.

Comment 2

[Neil Dunbar]

Ryan,

No, it is not TrustCor's position to remain in violation of the BRs and our CP/CPS conditions. The purpose of this ticket was to take the first step to rectify the situation and to show that we are acting in good faith.

We wanted to provide an explanation of why there was a delay in revoking the Subordinate CAs within the expected 7 day window, as we needed the additional time to assess the full impact revocation would have on our customer base.

Due to our assessment, that revoking both Subordinate CAs immediately would cause an immense amount of disruption to TrustCor's customers, we have put together a tentative outline of the course of action we intend to take. This plan is tentative because we might discover, prior to these dates, that it is feasible to speed up this timeline. In which case, any variations will be posted to this ticket.

We intend to formally revoke the Basic Secure Email CA and Enhanced Secure Email CA certificates according to the following schedule:

Dec 5th, 2019: Formal revocation of Enhanced Secure Email CA Certificate
March 1 - April 1, 2020: Bulk replacement of existing Basic Email CA certificates under new hierarchy, plus restart of the Enhanced Secure Email program
April 1, 2020: Formal revocation of Basic Secure Email CA Certificate

At this point, any residual risk to the ecosystem from those CA certificates, not fully disclosed, should be obviated and we will have a new PKI hierarchy to take up its previous position.

Regards,
Neil

Comment 3

[Neil Dunbar]

The Enhanced Secure Email CA Certificate (Serial 0xaf3e61240471752, SHA-256 A6D365161B58539CB44B29D77C648126F33DB3C493116C3040E18DE3E01A4242) has now been formally revoked as of December 5, 2019 17:10:18 UTC.

We will update this ticket per the timeline above, unless we can bring forward the certificate replacement and revocation timescale.

Comment 4

[Ryan Sleevi]

Thanks for clarifying, Neil, and for the update on revocation.

I think one thing missing from this incident response so far (and apologies if it was missed), but is captured at https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation

That you will perform an analysis to determine the factors that prevented timely revocation of the certificates, and include a set of remediation actions in the final incident report that aim to prevent future revocation delays.

I'm hoping this can provide good opportunity to find solutions on how best to minimize delays in the future, as well as minimize disruption to TrustCor consumers/users.

Comment 5

[Neil Dunbar]

Ryan,

With regard to ensuring that future revocations of Subordinate CAs can be done within the normal BR time limits while minimising the
disruption to TrustCor customers, we identified the following areas where our process can be improved:

We intend to implement a new certificate hierarchy which will shift from three levels (Root - Subordinate - End-Entity), into four levels (Root - Business CA [ e.g. Email, SSL, etc] - Issuing CA [ e.g. Algorithm + Validation Standard] - End-Entity). By adding an extra level into the hierarchy, we expect that this will not only create an added layer of isolation from the issuing root, but also allow for revocation of an issuing CA via an online Subordinate CA. This will diminish the physical and manual process of revoking and reissuing a Subordinate CA that exist today, which will help to streamline this process going forward.

We are still evaluating the best arrangement for the four level hierarchy, so this initial plan might be subject to change in detail. The new hierarchy will roll out in incremental steps over the course of next year and any new subordinates issued will be disclosed to the CCADB within seven (7) days of generation, as well as in our next round of audit reports.

In addition to this new arrangement, we are also looking into producing name constrained and technically constrained Subordinate CAs, wherever possible, for our our larger customers.

We hope that these improvements help to address the report requirements you highlighted, as we anticipate these improvements will minimise delays in the future, all while minimising disruption to TrustCor consumers/users.

Regards,

Neil

Comment 6

[Ryan Sleevi]

Thanks for the update.

Wayne, passing to you, per discussions on how best to handle this.

Comment #2 identifies 2020-4-1 for the revocation of the other CA, and Comment #5 identifies a new approach is being implemented (and will roll out over the next year). I think the policy side of Comment #5 sounds like it's being implemented sooner, so Comment #2 is the main actionable task.

Comment 7

[Wayne Thayer]

Now that Ryan's questions have been answered via comment #5, I agree with the disposition of Next Update when the Basic Secure Email CA is revoked, or when new or updated information is available.

Comment 8

[Neil Dunbar]

As of 2020-04-01 17:05:00 UTC, the TrustCor Basic Secure Email CA certificate was revoked, per the described plan in this ticket. New CRLs and OCSP responses have been published to the standard repositories.

Comment 9

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.