1496088 - Certinomis: test certificate for test.com, O=Entreprise TEST

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Andrew Ayer]

Certinomis has issued the following pre-certificate:

https://crt.sh/?sha256=516e750a2fff294f5468e5b29a4fa0d9648fc9d7d8e7936d2f9847a30d3e0267

It contains a DNS SAN for test.com, and the subject contains O=Entreprise TEST.  I strongly suspect this is misissued.

Comment 1

[Wayne Thayer]

As precertificates are considered a commitment to issue by the CA, this does appear to be a misissuance. Please provide an incident report for this problem, as described here:
https://wiki.mozilla.org/CA/Responding_To_A_Misissuance#Incident_Report
The incident report should be posted to the mozilla.dev.security.policy forum and added to this bug. The report should include an explanation of why this incident was not promptly reported to Mozilla.

Comment 2

[Franck Leroy]

Hello

Here is an incident report aligned to the mozilla template :

1/ How your CA first became aware of the problem.
I've been aware by this bug openned by Andrew and affected to me by Wayne.

2/ A timeline of the actions your CA took in response.
2018-10-03  12:22:59 UTC    : human error in validation of a test certificate
2018-10-03  14:22    UTC    : bug openned by Andrew
2018-10-03  15:41    UTC    : buzilla notification done by Wayne, received my email
2018-10-03  16:03:55 UTC    : revocation of the invalid certificate

3/ Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem.
It is a human error, not a systemic one.

4/ A summary of the problematic certificates. 
One certificate : test.com

5/ The complete certificate data for the problematic certificates.

https://crt.sh/?sha256=516e750a2fff294f5468e5b29a4fa0d9648fc9d7d8e7936d2f9847a30d3e0267

6/ Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
We had an issue with the SMTP server so the certificates were not sent to the customers.
When the SMTP issue has been resolved, the sales departement has tested the full application process by requesting a 'test certificate'.
The 'test certificate' process is to use "Entreprise TEST" account and apply for the domain "test.certinomis.com"

The operator made a mistake in the TLS certificate application, is has entered "test.com" instead of "test.certinomis.com"
The operator did not send the certificate to the customer, as there is no customer, it is for test purpose only.
When I've been notified of such error (and Andrew/Wayne were pretty fast!) I immediatly revoked the invalid certificate, and sent an incident email to Certinomis' CEO (I was out of office at an ETSI meeting).

7/ List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future.
We will change in a few days the testing process to avoid such human errors.
The "Entreprise TEST" account will be limited to one domain name : "test.certinomis.com".
So this error could not be done again in this process.

The incident was not reported "promptly" because the operator did not understand that he made a mistake.
The timeline of the event is very small, a window of less than 4 hours.
It is quite difficult to imagine a process that detects, analyses, and reports an incident in less than 4 hours.

All operators have been briefed of this incident, and a reminder of the validation/report rules has been done by Certinomis' CEO.

Best regards
Franck Leroy

Comment 3

[Wayne Thayer]

Franck: please update this bug when the technical control limiting the "Enterprise TEST" account to a Certinomis domain has been implemented.

The incident report implies that Certinomis operators can override technical domain validation controls on any certificate request, so a similar error could be made on a fraudulent request from a customer. What prevents an operator from accidentally approving a request for a high value domain such as facebook.com or gmail.com?

Comment 4

[Ryan Sleevi]

(In reply to Franck Leroy from comment #2)
> Hello
> 
> Here is an incident report aligned to the mozilla template :
> 
> 1/ How your CA first became aware of the problem.
> I've been aware by this bug openned by Andrew and affected to me by Wayne.
> 
> 2/ A timeline of the actions your CA took in response.
> 2018-10-03  12:22:59 UTC    : human error in validation of a test certificate
> 2018-10-03  14:22    UTC    : bug openned by Andrew
> 2018-10-03  15:41    UTC    : buzilla notification done by Wayne, received
> my email
> 2018-10-03  16:03:55 UTC    : revocation of the invalid certificate
> 
> 3/ Whether your CA has stopped, or has not yet stopped, issuing certificates
> with the problem.
> It is a human error, not a systemic one.

At this point in the CA ecosystem, the potential to have human errors - particularly human errors that have affected other CAs in other incident reports - and that the CA has not taken steps to redress *does* highlight systemic flaws in the CA processes.

What is particularly telling about this is when it occurred - within days of Bug 1495524 - which seems to share a similar root cause. This suggests that the response to Bug 1495524 was problematic, in that one of the expectations of any CA, particularly one affected by a human error issue, would be to review its process and procedures with its staff to retrain them on the appropriate validation processes.

Would you like to provide any more detail beyond what is provided in 5.3.3, 5.3.4, and 5.7.1 to describe how Certinomis handles and responds to incidents, particularly those related to human errors, and what steps it takes to mitigate the ongoing risk of human factors?

> The incident was not reported "promptly" because the operator did not
> understand that he made a mistake.
> The timeline of the event is very small, a window of less than 4 hours.
> It is quite difficult to imagine a process that detects, analyses, and
> reports an incident in less than 4 hours.

It doesn't seem difficult, considering that this was noticed within 2 hours of the incident. Instead, it's quite difficult to imagine why a CA in 2018 would be able to have such an issue, days after having a similar incident, and has not implemented automated controls to the entry, validation, and processing of domains and certificates.

> 
> All operators have been briefed of this incident, and a reminder of the
> validation/report rules has been done by Certinomis' CEO.

This is troubling - again, in light of Bug 1495524 - and seriously calls into question the domain validation procedures practiced at Certinomis. One possible interpretation is that Certinomis' response to Bug 1495524 with its employees did not adequately stress the importance of proper domain validation when entering in information - which would be a lapse by Certinomis' PKI managers. Another possible interpretation is that the employees did not respond to the previous message, in which case, the remediation here is quite inadequate.

Comment 5

[Franck Leroy]

Attachment: screenshot.pdf

Comment 6

[Franck Leroy]

Hello,

>The incident report implies that Certinomis operators can override technical domain validation controls 
Operators cannot override domain validation when the feature is activated.

This option for domain validation is neccessary as we can have customers that apply for publicly trusted or non publicly trusted certificates (named corporate) .

The issue was that the "Entreprise TEST" demo account was not correctly restricted for SSL/TLS certificates.

I uploaded some screenshots to explain the actions that has been done to prevent from similar errror.

The first screenshot shows that for the demo account, the area for CN (and SANs) is free text, so "test.com" is possible.

The second screenshot shows the "SSL restriction" activated with only one domain : certinomis.com

The third screenshot shows that with this option activated, only "certinomis.com" domain is possible for an application, and the free text area is only for the sub-domain.

The second action was to change the certificate type. Indeed to test if the system works correctly it is not necessary to issue a publicly trusted certificate.

The last screenshot shows that the certificate profile has been changed to issue a TLS certificate under "Corporate G4" CA, that is not publicly trusted.

Best regards
Franck Leroy

Comment 7

[Ryan Sleevi]

(In reply to Franck Leroy from comment #6)
> >The incident report implies that Certinomis operators can override technical domain validation controls 
> Operators cannot override domain validation when the feature is activated.
> 
> This option for domain validation is neccessary as we can have customers
> that apply for publicly trusted or non publicly trusted certificates (named
> corporate) .
> 
> The issue was that the "Entreprise TEST" demo account was not correctly
> restricted for SSL/TLS certificates.

These three statements do not provide sufficient assurance that the root issue has been mitigated. It sounds like the answer to Wayne's question is, in fact, that operators can override domain validation if the customers are not correctly restricted for SSL/TLS certificates.

This issue demonstrates at least one customer (Certinomis itself, apparently) was not properly restricted. The root cause analysis to date has not highlighted this as a systemic remediation - rather, a single account was restricted. Why is it not appropriate to re-examine every one of your customer accounts to ensure appropriate restrictions are configured? Or, perhaps more explicitly, to ensure that any unrestricted accounts cannot obtain certificates from publicly trusted CAs? Why does the account restriction even exist, since it relates to which CA is being used to issue (and thus should not matter how the 'account' is configured)

> I uploaded some screenshots to explain the actions that has been done to
> prevent from similar errror.
> 
> The first screenshot shows that for the demo account, the area for CN (and
> SANs) is free text, so "test.com" is possible.
> 
> The second screenshot shows the "SSL restriction" activated with only one
> domain : certinomis.com

This screenshot does not support the statement in comment #2 that:
> > The "Entreprise TEST" account will be limited to one domain name : "test.certinomis.com".
> > So this error could not be done again in this process.

It seems like the operator could request www.certinomis.com using this interface and potentially misissue certificates. Is that not correct?

Comment 8

[Franck Leroy]

Hello

>This issue demonstrates at least one customer (Certinomis itself, apparently) was not properly restricted. 
Yes, it is a demo account used internally to test or demonstrates the certificate application.

>Why is it not appropriate to re-examine every one of your customer accounts to ensure appropriate restrictions are configured?
You are right, I will do it as soon as possible.

>Why does the account restriction even exist, since it relates to which CA is being used to issue
Our system application exists since 2008 and managed many workflows on many different CAs.
Some CAs are not publicly trusted and mainly used internally in 'La Poste Group'.
So in that case, the GUI is not restricted in order to allow any subject in the certificate.

The rule is that no unrestricted account shall be allow to issue PTC. 
I miss this one, may be my fault, so I have to review them all.

>one domain name : "test.certinomis.com".
Yes the system is not designed to limit to some FQDNs, only domain names.
I can enter "test.certinomis.com" is the configuration, and in that case, any 'www' will produce: 'www.test.certinomis.com"

>It seems like the operator could request www.certinomis.com using this interface and potentially misissue certificates.
Yes, but I also changed the PKI certificate profile to use the corporate CA G4.
It could be usefull to make a demo that reflects a real certificate application, but no more under a publicly trusted CA.

Best regards
Franck Leroy

Comment 9

[Andrew Ayer]

Certinomis has also issued a pre-certificate for www.pourtest.com with O=Entreprise TEST in the subject:

https://crt.sh/?sha256=B77B246619EB66F97F72A753EC5A97F07D48DC44047EB717639019E0C0A8B628

I strongly suspect that the applicant's control over www.pourtest.com was not validated, as the domain is not registered to Certinomis, nor does it appear to be a customer of Certinomis, as all other certificates for this domain are issued by Let's Encrypt (https://crt.sh/?q=%25.pourtest.com).

This certificate was revoked on 2018-10-03 at 16:05:27 UTC - less than two minutes after the certificate for test.com was revoked (2018-10-03 16:03:55 UTC).  However, Certinomis did not mention this certificate in their incident report.

Here are five more unexpired (pre-)certificates with O=Entreprise TEST in the subject:

https://crt.sh/?sha256=08BCB09E6AE3B4E21F298F51C44F7ED8632FDC92E88E188D685B0900F87BF61D
https://crt.sh/?sha256=B1940AD03E642275D785C5C87290CD4B999B56C9F584423CB11F5A0565F8E7DD
https://crt.sh/?sha256=8F5E62DB7D52CA96FB6E60924F9D11AAD67958EF92A7BD46F0554B2EADEAD80E
https://crt.sh/?sha256=5365B1C22A20E3FB4A7F80D111D0EA9DACABC1663366B6AED5A0FB052C94C064
https://crt.sh/?sha256=4D9A21EFBF0D73E6C247BF5CE035B21AB5EB445E2D236E6751F078F7BEB14A18

Only the first two are revoked at this time, suggesting that Certinomis' investigation and remediation are still incomplete 25 days later.

I would also like to note that Certinomis' incident report was not posted to mozilla.dev.security.policy as required.

Comment 10

[Andrew Ayer]

It has been 15 days since I reported the incomplete remediation and the likely-misissued certificate for www.pourtest.com, and Certinomis has yet to post an update.  Per https://wiki.mozilla.org/CA/Responding_To_An_Incident, "Once the report is posted, you should provide updates at least every week giving your progress, and confirm when the remediation steps have been completed - unless Mozilla representatives agree a different update schedule with you."  I would therefore expect to have seen an update from Certinomis in the last 15 days addressing the new information.

Comment 11

[Franck Leroy]

Hello

I have reviewed all certificates issued under the "Entreprise TEST" demo account.

3 TLS certificates were issued under "Certinomis Web CA"
	names are : www.pourtest.com, www.seres-as2.fr, testpro.darva.com
4 TLS certificates were issued under "Certinomis Easy CA"
	names are : test.com, aboxnote.com, expe.visio.douane.gouv.fr, TRANSGOURMET - CHORUS TEST
	
The 7 certificates has been revoked.

The "Entreprise TEST" demo account is configured to "Certinomis Corporate CA" that is not publicly trusted. So demo usage will no more issue PTC certificates.

As said previously, one workflow in our RA software (used internally for test and demo) can bypass domain validation.
This workflow has been disabled.
A new development is running to implement domain validation in this workflow, then when available it will be possible to reactivate it.
The new version of the RA software is expected in production by the end of the year.

Bets regards
Franck Leroy

Comment 12

[Ryan Sleevi]

> The new version of the RA software is expected in production by the end of the year.

Franck, can you provide an update now that we're in the new year?

Comment 13

[Wayne Thayer]

Franck is no longer a Certinomis representative. Reassigning to Marc.

Comment 14

[Ryan Sleevi]

Marc: It's been nearly two weeks. Do you have an update?

Comment 15

[Ryan Sleevi]

Marc: Checking to see for updates?

Comment 16

[Ryan Sleevi]

Marc: It's been one month. Do you have updates?

Comment 17

[Wayne Thayer]

Francois: I kindly request an immediate update on this incident.

Comment 18

[François CHASSERY]

Hello,

The development that implements domain validation in the workflow has been delivered.
It is now running on pre-production platform for testing.

If nothing is wrong it should be installed on production plaform within 3 weeks.

Kind Regards,

François

Comment 19

[Wayne Thayer]

Has "domain validation in the workflow" been installed in production yet?

Comment 20

[Andrew Ayer]

Here's another certificate with O=Entreprise TEST in the subject:

https://crt.sh/?sha256=97C78F92745645FE7ABC5A531C27F4C29D54F193563FB2035C01A0BE74CA3BBA

It was issued in January - after Certinomis indicated in comment 11 that the Entreprise TEST account had been configured to use a non-public CA - and it is still unrevoked today.

Comment 21

[François CHASSERY]

Dear Andrew,

Franck did not write false, and he really moved "Entreprise test" in a "demo zone" where only non PTC are available.
But he did half the job.

Indeed, there is two different access in our RA services for a certificate request: "normal" procedure is that a customer enter a request in "Front-Office" access that is afterward controlled and validated by a RA operator. And there is also the possibility for a RA operator to enter directly a request in a back-office access. This second possibility is for customer service purposes, and is also used for issuing test certificates.

What happened in last November is that Franck moved "Entreprise test" in a "demo zone" where only non PTC are available. This affects Front-Office access. But settings in back-office and front office are not bijective which means that a restriction applied in front office is not automatically transferred in back-office. So our full range of products remained available for "Entreprise test" in back-office access.
And it has been used in January for creating a new "test certificate".

Presently, entering certificate request in back-office access has been disabled for SSL certificates (on 25th of March).
It means that it is now impossible to create a certificate request in back-office access.
(Later a restricted back-office access will be re-opened only for external RA where domain name and Entreprise name are constrained)

In addition, everybody in Certinomis is now aware that test certificates are forbidden in SSL activity, contrary to what we practice for other certificates.

So you can be certain that you will not see another certificate with O= Entreprise test.

The certificate https://crt.sh/?id=1101522524 has been revoked on April, 18th, together with the certificate "expe.visio.douane.gouv.fr" mentioned in comment#11, and we have made a complete review of certificate issued for "Entreprise test" and have checked that they are now all revoked or expired.

Kind Regards,
François

Comment 22

[Wayne Thayer]

Francois: did the complete review of certificate issued for "Entreprise test" find any certificates that hand not already been revoked?

Comment 23

[Andrew Ayer]

I reported the following certificate in Comment 9, but at the time it seemed like its only problem was O=Entreprise TEST in the subject. It turns out it's also for an unregistered domain (seres-as2.fr):

https://crt.sh/?sha256=8f5e62db7d52ca96fb6e60924f9d11aad67958ef92a7bd46f0554b2eadead80e

I wanted to note this for completeness.

Comment 24

[François CHASSERY]

Dear Andrew,

Yes the core problem was the issuance of test certificate from PTC hierarchy.
And in test certificate not everything can be true.

This problem is now covered by isolating test account in a sepcial demio zone.

Kind Regards,

François

Comment 25

[Wayne Thayer]

The Certinomis Root CA is being removed from the Mozilla root store in bug 1552374, so I am resolving this bug. Additional comments that may be useful when considering any future application by Certinomis are welcome.