1599503 - TrustCor: Non-mention of Email CAs in WTBR audit reports

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Neil Dunbar]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36

Steps to reproduce:

TrustCor was informed by Mozilla that two (2) of our Subordinate CAs, that were listed in our WebTrust for CAs report, also needed to be included in our SSL Baseline with Network Security report.

We recognise that failing to disclose the Subordinate CAs in both reports was a violation of BR Section 8.1. The omission was due to a misreading of the requirements surrounding CA certificates which are not intended for SSL certificate issuance (i.e. S/MIME in this case).

The overall timeline for this incident is as follows:

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

A post on the Mozilla Dev Security Policy from Kathleen Wilson on 2019-10-08 alerted TrustCor CA to discrepancies between the basic audit report and SSL-BR audit report.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

(Times are in UTC)

2019-10-09 09:00:00: Initial investigation to establish the scope of the issue. Initial findings were that 4 of the reports were mistaken and that 2 were real
2019-10-10 10:15:28: Communication to TrustCor's independent auditor that the Email Subordinate CA certificates needed to be mentioned in the next audit report, due in November
2019-10-10 14:30:00: Formal suspension of the Enhanced Secure Email CA program until a resolution could be determined. The Basic Secure Email program is in active use and suspension would cause major disruption to TrustCor CA customers
2019-10-12 16:00:00: TCPA requests CA Administrator to reconcile all HSM activities using Email CA private keys since November 2015 with issuance logs for S/MIME CAs and report any discrepancies
2019-10-31 14:34:00: Followup to Kathleen Wilson of Mozilla requesting clarification on the report and reorganisation of the mistaken certificates in the ALV report
2019-11-07 15:59:00: Further communications with Mozilla regarding the removal of the mistaken certificates (due to structure of CCADB)
2019-11-07 19:07:00: CCADB data reorganised, resulting in removal of 4 certificates from ALV report, leaving only 2 reports for Email certificates
2019-11-15 15:30:00: Completion of self audit establishing that the Email CA private keys have not signed any SSL certificates. Forwarded results to independent auditor
2019-11-15 17:13:00: Formal submission of new audit reports to WebTrust, mentioning the Email CAs in the SSL-BR report
2019-11-20 14:30:00: Formal decision of TCPA to request addition of Email CA certificates to OneCRL

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

These were CA certificates rather than end-entity ones. No end-entity certificates which are non-compliant have been noted. S/MIME certificates are still being issued, but no SSL certificates have ever been issued under those subordinate CAs.
Stopping the entire S/MIME program for TrustCor CA would affect many TrustCor CA customers. Disclosure of the CA certificates to WebTrust has been done, thus we assess that there is minimal ongoing risk to browser customers. (Note that our CT log monitors would flag an alert if any SSL certificate under a CA other than those which TrustCor normally uses was ever found).

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

The 2 CA certificates are:
Basic Secure Email CA:
Subject DN: CN=TrustCor Basic Secure Email (CA1); OU=TrustCor Network; O=TrustCor Systems S. de R.L.; C=PA
Issuer DN: CN=TrustCor RootCert CA-1; OU=TrustCor Certificate Authority; O=TrustCor Systems S. de R.L.; C=PA
Serial Number: 00A60D883219A3FD59
Enhanced Secure Email CA:
Subject DN: CN=TrustCor Enhanced Secure Email (CA2); OU=TrustCor Network; O=TrustCor Systems S. de R.L.; C=PA
Issuer DN: CN=TrustCor RootCert CA-2; OU=TrustCor Certificate Authority; O=TrustCor Systems S. de R.L.; C=PA
Serial Number: 0AF3E61240471752

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Fingerprints are:
(Basic Secure Email CA): 02BEF922B32D46DFE7520B0EE7E3EAF588EE2B9CAB81B84837E6B955E0759A90
PEM:
-----BEGIN CERTIFICATE-----
MIIGNjCCBR6gAwIBAgIJAKYNiDIZo/1ZMA0GCSqGSIb3DQEBDQUAMIGkMQswCQYD
VQQGEwJQQTEPMA0GA1UECAwGUGFuYW1hMRQwEgYDVQQHDAtQYW5hbWEgQ2l0eTEk
MCIGA1UECgwbVHJ1c3RDb3IgU3lzdGVtcyBTLiBkZSBSLkwuMScwJQYDVQQLDB5U
cnVzdENvciBDZXJ0aWZpY2F0ZSBBdXRob3JpdHkxHzAdBgNVBAMMFlRydXN0Q29y
IFJvb3RDZXJ0IENBLTEwHhcNMTYwMjA1MTEzNjUwWhcNMjkxMjMwMTYyMDEwWjCB
izELMAkGA1UEBhMCUEExDzANBgNVBAgMBlBhbmFtYTEkMCIGA1UECgwbVHJ1c3RD
b3IgU3lzdGVtcyBTLiBkZSBSLkwuMRkwFwYDVQQLDBBUcnVzdENvciBOZXR3b3Jr
MSowKAYDVQQDDCFUcnVzdENvciBCYXNpYyBTZWN1cmUgRW1haWwgKENBMSkwggIi
MA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDZ4TZUDMNEyZ08U9AKX3gvpCsL
t3GKhWYFL6Lsz4po1IP7/NBywvXFuhKAzkf7zEUesJsSxmNDSq/WaTRBU9HbKg5U
6S/WjxVElvPKQVyhOjyRSF2dt8tRODZJMI4QXc2ZnWm6z8BOSE/qW5qyYzQQwWUI
pFFNOy4xqHwsuK8wzN7MRJFpY81xwCubVeZgvXA/2fxfl/EPH1mS8nGtP8J1fnkG
kv2ffdR3Ebx1ffdRdORpveINfTXz+QQ6sI8Ujxl+o03vEQiTnwFqETRO2XgDZUI2
9hxnk5ROvXbUKbDXVbhlhgpKa9FF5F+Yw0fdwHhgsm0EbMSAJl+zQPJTP6b09EIC
SRSsQVza7K55qsQkVE6ib1ElL+f/lISeavcJPFCs7j74liY3vQvCtn+qjJiaIoQH
g7ay5WYTTC/iLHNn6X86LLJk0fSXWIkcDGSHrsaoqpe/6nuz/Z9r3veZ4ylgjaui
E4ajLxVsgPX7WrxkQACNrNP7vP2CmYZ+EnRLiiz0pORYRUpOJKCYBYKDwkZseAbf
8wdu+ishb8vPOwm6Z8voc182VT+F5CKje5HbtgRkRD/zOGQP78Ftg8By7Ltb/kkV
tS0ofI1rS0ajlMFFo4DnYrRWqlMds46JBbglUNLHHX/6fA6/oR09Sb5bSYRL1JQP
0VfpGPKME6lM+wvqjQIDAQABo4IBgDCCAXwwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAYYwHQYDVR0OBBYEFOQLHVuaqn7JxmtkOHZxwHbKlyk6MB8GA1Ud
IwQYMBaAFO5rSTx6Pw3jsQm3isirGZ9zM1DnMFgGA1UdIARRME8wTQYLKwYBBAGC
138BAQEwPjA8BggrBgEFBQcCARYwaHR0cDovL3d3dy50cnVzdGNvcnN5c3RlbXMu
Y29tL3Jlc291cmNlcy9jcHMucGRmMDQGA1UdHwQtMCswKaAnoCWGI2h0dHA6Ly9j
cmwudHJ1c3Rjb3IuY2Evcm9vdC9jYTEuY3JsMIGIBggrBgEFBQcBAQR8MHowLAYI
KwYBBQUHMAGGIGh0dHA6Ly9vY3NwLnRydXN0Y29yLmNhL3Jvb3QvY2ExMEoGCCsG
AQUFBzAChj5odHRwOi8vd3d3LnRydXN0Y29yc3lzdGVtcy5jb20vY2VydHMvVHJ1
c3RDb3JfUm9vdENlcnRfQ0ExLnBlbTANBgkqhkiG9w0BAQ0FAAOCAQEAPWhBFCGP
VIhF6lbwdC3EAT6LiAjXvC+WDNV7WTA0FSrQ4r7WRft+d4nbLOJqXm8+0gqLL6pu
eIOFiassui0OeqTfYN4BRZtEaVQkUB6ftPL+9ZB9KlPch6Y0stG+TlwgoLYDJKNb
YnyqiifSOWvWRxtjM0M+s/pm88L2y1cwCQ1iuDxS4aGGUUqTN9WXvXMfAMYGbcKS
SOFzD1NSI8ZDKjQEdBOVHvi/9kI0IcP8OjL5VlXrqAaQyIGl1sfM0XB/rwn1m6Sk
JVhOpR3hsscdt4iip2jy2DVtVpWC246UaRnHTpEuqFDzSJw4GnOCaBeIAajDaCPi
Ioi84Kd1WvwkXQ==
-----END CERTIFICATE-----

(Enhanced Secure Email CA): A6D365161B58539CB44B29D77C648126F33DB3C493116C3040E18DE3E01A4242
PEM:
-----BEGIN CERTIFICATE-----
MIIHODCCBSCgAwIBAgIICvPmEkBHF1IwDQYJKoZIhvcNAQENBQAwgaQxCzAJBgNV
BAYTAlBBMQ8wDQYDVQQIDAZQYW5hbWExFDASBgNVBAcMC1BhbmFtYSBDaXR5MSQw
IgYDVQQKDBtUcnVzdENvciBTeXN0ZW1zIFMuIGRlIFIuTC4xJzAlBgNVBAsMHlRy
dXN0Q29yIENlcnRpZmljYXRlIEF1dGhvcml0eTEfMB0GA1UEAwwWVHJ1c3RDb3Ig
Um9vdENlcnQgQ0EtMjAeFw0xNjAyMDUxMTUwMjhaFw0yOTEyMzAxNjIwMTBaMIGO
MQswCQYDVQQGEwJQQTEPMA0GA1UECAwGUGFuYW1hMSQwIgYDVQQKDBtUcnVzdENv
ciBTeXN0ZW1zIFMuIGRlIFIuTC4xGTAXBgNVBAsMEFRydXN0Q29yIE5ldHdvcmsx
LTArBgNVBAMMJFRydXN0Q29yIEVuaGFuY2VkIFNlY3VyZSBFbWFpbCAoQ0EyKTCC
AiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMswzswrWTDAAKGbEs/poiCL
ZlHCWTpORqfCvMrbG61WY6PJYtXqkypVFyJUVHxZODdPVgk/VPyiJQvhw/XC3DzT
Gc2hgOV2vkBfQNVpvZyf0/WibmTLeCvPPhwP4v1e6nee8Xyt8zlrZM7lew650rGU
36EbrB7Mw8SQ64HNUwT3joKBD0yGNoI2RK5pq04EOZjnf3HpvAW5bo3xOG8lvQaY
YLoeNGeYnCnfE0JoSPKLYPMsL2y59qbNqilqJ/f77wczMLtWwUd0V+5b0Ftrcepi
EJ6+iNgZXOwLeyBBHE7BoBjJN8Lp+nWayIfRQmzJwlZI3CyJbYdQdSJGnB6IEzdS
HZugWUPpW3uA3v3vV3v1bLPFBO6cUiBAY3l/9DfKK/9CKyevSkoHXoe3RxbxvCHx
uNaZhawSs+y0zsc6Dwk7MebUZz+jEoQiQgnPszpkWkY2v+XTgv3EVwp26d2dpKRM
RYIatG/DpKp+cr2dB+DepqrX0BYzx/vtZtaJsoBcrnW7GlPr3qBPZmgMoSfFy8x0
zHzt77MJt0JAoXs5aChglqfLV6MHNUYF6J2YBYIfo5C6eUoK2uziun/o7/OjpRgS
JkkdCP2XMVepvQl+Iucklp8HNX96aZ0Tefvvc5gkKCJEuA7jXValPSxm2+LzGGns
lOkj5nXqGajl5EAYoQXBAgMBAAGjggGAMIIBfDAPBgNVHRMBAf8EBTADAQH/MA4G
A1UdDwEB/wQEAwIBhjAdBgNVHQ4EFgQUqCXG3oT0KHpseqYDZZRRChasyf4wHwYD
VR0jBBgwFoAU2f4hQG6UnrybPZx9mCAZ5YwwYrIwWAYDVR0gBFEwTzBNBgsrBgEE
AYLXfwEBATA+MDwGCCsGAQUFBwIBFjBodHRwOi8vd3d3LnRydXN0Y29yc3lzdGVt
cy5jb20vcmVzb3VyY2VzL2Nwcy5wZGYwNAYDVR0fBC0wKzApoCegJYYjaHR0cDov
L2NybC50cnVzdGNvci5jYS9yb290L2NhMi5jcmwwgYgGCCsGAQUFBwEBBHwwejAs
BggrBgEFBQcwAYYgaHR0cDovL29jc3AudHJ1c3Rjb3IuY2Evcm9vdC9jYTIwSgYI
KwYBBQUHMAKGPmh0dHA6Ly93d3cudHJ1c3Rjb3JzeXN0ZW1zLmNvbS9jZXJ0cy9U
cnVzdENvcl9Sb290Q2VydF9DQTIucGVtMA0GCSqGSIb3DQEBDQUAA4ICAQAF/Lcr
mQhEFYo7HyRYrnVIhWYuVpDg5uVqVA8nClA41P39UX+VH+M6SkInPvwPUf3j0Lgp
TB70t/h16Z1m3Bw1rwHoHi8bgFtTWJ10MKxL7oj77vO/68PWp4DRIiCplecPIrRt
enk13rj5XvLcaIWJUK8lsjScgHchQAcbyMBAo0h8y1rzxITsWiCiop1sZ/NjMcph
QbcG43qoHu50231MkGgJjVWeI5sMk9MCI/mRxPz421GXMFOa2WTf+o5Ba+WeEnpH
inHcGMvUktLARqhZmNRbYEhQGMspUooQIF47KoJt+orhicAb4fPE5vr13U7OGsqi
Uju80WZTS95Ra6q/TXAX5yERZuE7klwmdG0bDcgwljqcpmxdMhEwzYcHPemGPPIf
8uyVvUW7G9icfgMFD5TSSjNnRLZTcoLn37653Wukro6Ze3DgYuB+bUjIFGM+PXWm
adaV03vR6WxhOWJBzbN0LwKJ6dPPIvy1rGvX2TNjF+rzkJnTGKJmtX/ODdwGRaoZ
gseSmiBn8UpZaqvZQrvZzkxUFvoavzEW6b2QWSTtDcgPPo4qKVaZdePFACAkWVEN
3sgyWVtQuXbSfD6MRyvp2tUPu2qg80tqzg03nt16yO3t7fxxlYP416lCtl5vTKqG
hw6kt9pNkn4JEsjE+9xLEiXO74hzo+hq1yej9g==
-----END CERTIFICATE-----

Note: these are not in crt.sh, because no SSL certificate has been logged under them.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The mistake was a misreading of the requirements for disclosure: it is clear that CA certificates which even have the technical potential to issue an SSL certificate (even if never intended to do so, and even where business and technical control exist to prevent such issuance) should be disclosed in the SSL-BR audit report.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Since being alerted to this discrepancy, the TrustCor Policy Authority (TCPA) gave instructions to the CA Administrator to begin a reconciliation of the HSM logs for the Secure Email CA private keys with the EJBCA issuance logs, and establish that the only items signed by the HSMs keys were:

1. S/MIME certificates, duly logged in the audit log.
2. CRLs for the Secure Email CA program, duly published
3. OCSP Responder certificates, which in turn validate pre-generated OCSP responses for S/MIME certificates, duly published to the OCSP origins.

Specifically, the request was to state, with a high level of confidence that no private key intended for S/MIME has been used for any SSL related purpose. This self-audit has now completed and TrustCor states with very high confidence that no SSL certificate has been issued via the Email CA private keys. Since the audit went back some four years, it took some time to complete. The results have been passed on to TrustCor's independent auditor to be noted in the next audit report.

Having completed that audit, the TCPA has determined that four courses of remediation shall be followed forthwith. The balancing considerations have been:

A) minimising risk to users of software which trust TrustCor CA's root certificates (CA-1, CA-2, and ECA-1).
B) minimising harm to the many users who use TrustCor CA S/MIME certificates and are not in a position to replace them quickly.

1. The existing Secure Email CAs mentioned above shall be disclosed in both standard and SSL-BR audits. This has been completed and sent to WebTrust for seal issuance.

2. We ask for Mozilla's assistance in adding both Basic and Enhanced Secure Email CAs to OneCRL as soon as is possible. This shall at least mitigate any residual risk to users of software which consults OneCRL for certificate status information: we have ascertained that this should not affect Thunderbird users who use TrustCor CA S/MIME certificates.

3. We will follow up with the other Root Programs to which TrustCor is also enrolled to determine their feedback to this proposed remediation.

4. Following any Mozilla (or any other Root Program) policy change, the TCPA shall convene and assign two different persons to conduct a review of the necessary changes in behaviour which the policy change will entail. These reviews shall be entered into the CA's audit log and submitted to the WebTrust auditor.

We are fully aware that the normal timeline for revoking a Subordinate CA per the BRs is 7 days after a BR violation is noted. However, considering the harm which would obtain to TrustCor CA S/MIME users, and the self-audit establishing that there are no (and never have been any) SSL certificates which would risk Relying Parties; it is TrustCor CA's belief that not revoking the Email Subordinate CAs rapidly is justified. A separate ticket will be generated discussing the revocation issue, separate from this one.

Comment 1

[Neil Dunbar]

By way of correction: the crt.sh IDs are here (I meant that no SSL certificates have been logged under them via CT logs)

Basic Secure Email CA: https://crt.sh/?id=170664054
Enhanced Secure Email CA: https://crt.sh/?id=170664056

Comment 2

[Kathleen Wilson]

(In reply to Neil Dunbar from comment #1)

By way of correction: the crt.sh IDs are here (I meant that no SSL certificates have been logged under them via CT logs)

Basic Secure Email CA: https://crt.sh/?id=170664054
Enhanced Secure Email CA: https://crt.sh/?id=170664056

I updated the corresponding records in the CCADB as "Ready to Add" to OneCRL.
https://ccadb-public.secure.force.com/mozilla/PublicInterCertsReadyToAddToOneCRL

Comment 3

[Neil Dunbar]

As of 2020-04-01 17:05:00 UTC, the TrustCor Basic Secure Email CA certificate was revoked, per the described plan in this ticket. New CRLs and OCSP responses have been published to the standard repositories.

The Enhanced Secure Email CA Certificate was now been formally revoked as of 2019-12-05 17:10:18 UTC.

This means that both the CA certificates mentioned in this bug have been formally revoked.

Comment 4

[Ryan Sleevi]

Bug 1599571 tracks the delayed revocation, so I think this is over to Wayne for closure if he agrees.

Comment 5

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.