1390998 - Kamu SM: Non-BR-Compliant Certificate Issuance

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Kathleen Wilson]

The following problems have been found in certificates issued by your CA, and reported in the mozilla.dev.security.policy forum. Direct links to those discussions are provided for your convenience.

To continue inclusion of your CA’s root certificates in Mozilla’s Root Store, you must respond in this bug to provide the following information:
1) How your CA first became aware of the problems listed below (e.g. via a Problem Report, via the discussion in mozilla.dev.security.policy, or via this Bugzilla Bug), and the date.
2) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates with the problems listed below.
3) Complete list of certificates that your CA finds with each of the listed issues during the remediation process. The recommended way to handle this is to ensure each certificate is logged to CT and then attach a CSV file/spreadsheet of the fingerprints or crt.sh IDs, with one list per distinct problem.
4) Summary of the problematic certificates. For each problem listed below: number of certs, date first and last certs with that problem were issued.
5) Explanation about how and why the mistakes were made, and not caught and fixed earlier.
6) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
7) Regular updates to confirm when those steps have been completed.

Note Section 4.9.1.1 of the CA/Browser Forum’s Baseline Requirements, which states:
“The CA SHALL revoke a Certificate within 24 hours if one or more of the following occurs: …
9. The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement; 
10. The CA determines that any of the information appearing in the Certificate is inaccurate or misleading; …
14. Revocation is required by the CA’s Certificate Policy and/or Certification Practice Statement; or 
15. The technical content or format of the Certificate presents an unacceptable risk to Application Software Suppliers or Relying Parties (e.g. the CA/Browser Forum might determine that a deprecated cryptographic/signature algorithm or key size presents an unacceptable risk and that such Certificates should be revoked and replaced by CAs within a given period of time).

However, it is not our intent to introduce additional problems by forcing the immediate revocation of certificates that are not BR compliant when they do not pose an urgent security concern. Therefore, we request that your CA perform careful analysis of the situation. If there is justification to not revoke the problematic certificates, then explain those reasons and provide a timeline for when the bulks of the certificates will expire or be revoked/replaced. 

We expect that your forthcoming audit statements will indicate the findings of these problems. If your CA will not be revoking the certificates within 24 hours in accordance with the BRs, then that will also need to be listed as a finding in your CA’s BR audit statement.

We expect that your CA will work with your auditor (and supervisory body, as appropriate) and the Root Store(s) that your CA participates in to ensure your analysis of the risk and plan of remediation is acceptable. If your CA will not be revoking the problematic certificates as required by the BRs, then we recommend that you also contact the other root programs that your CA participates in to acknowledge this non-compliance and discuss what expectations their Root Programs have with respect to these certificates.


The problems reported for your CA in the mozilla.dev.security.policy forum are as follows:

** Failure to respond within 24 hours after Problem Report submitted
https://groups.google.com/d/msg/mozilla.dev.security.policy/PrsDfS8AMEk/w2AMK81jAQAJ
The problems were reported via your CA’s Problem Reporting Mechanism as listed here:
https://ccadb-public.secure.force.com/mozilla/CAInformationReport
Therefore, if this is the first time you have received notice of the problem(s) listed below, please review and fix your CA’s Problem Reporting Mechanism to ensure that it will work the next time someone reports a problem like this.

** Serial Numbers less than 64-bit entropy 
https://groups.google.com/d/msg/mozilla.dev.security.policy/YequotPYLdc/J0K3lUyzBwAJ

Comment 1

[Tuğba ÖZCAN]

Bence cok temiz oldu herkesin eline sağlık 
----- Orijinal Mesaj -----
Kimden: Tuğba Özcan <[e-mail]>
Kime: Tamer ERGUN <[e-mail]>
Hi Kathleen,

We would like to point out that we are sorry for this situation.

In fact, we follow the rule that we should respond within 24 hours after
Problem Report submitted.

Please let us explain why we used our old root(Our old root is: TÜBİTAK
UEKAE Kök Sertifika Hizmet Sağlayıcısı - Sürüm 3.) for certificate
issuance although our new root(TUBITAK Kamu SM SSL Kok Sertifikasi -
Surum 1) was added in NSS. Because some of our customers (all of them are
government entities) required using mobile devices with Android and IOS
operating systems. As you know, Android default browser Google Chrome is
using Android's trusted CA list and IOS uses Safari Browser as default.
We advised using Mozilla Firefox. You can check it on our official web
site(http://kamusm.gov.tr/duyurular/). But they insisted on using
default browsers because of their usage habits. We have been trying to submit
our new root to Android and IOS. Android and IOS do not have any Root
Submission Program like Mozilla.
Then we tried to contact to related operating systems with email. But,
really we could not get any answer from these Operating systems via all
of our effort along last two years!

So, we solved the problem by using our old root which was added in
Android, IOS, and Mozilla NSS. We know that this was a temporary
solution since our old root will expire on 21/08/2017. We set expire date
of all certificates to 21/08/2017 which is the expiry date of old root.
Please note that this old root period will expire in 4 days. Now, we will
wait  for the certificates to expire.
After expiry date old root will not used, all the certificates issued from this root will also expired.
We will have to use new root which uses 64-bit entropy. All certificates issued by this root are/will obey the rule of 64-bit entropy.
But they will live problem with Android and IOS, it is a big problem and we could not find any solution despite all our efforts.

Also, we want to inform you that now we are developing a service which
will check all CAB BR requirements. By this service we plan to be informed about any unsuitability immediately.
After, service implementation is finished, we will inform you about that.

Comment 2

[Tuğba ÖZCAN]

Hi Kathleen,

Sorry for the last comment. Please do not consider it.

We would like to point out that we are sorry for this situation.

In fact, we follow the rule that we should respond within 24 hours after
Problem Report submitted.

Please let us explain why we used our old root(Our old root is: TÜBİTAK
UEKAE Kök Sertifika Hizmet Sağlayıcısı - Sürüm 3.) for certificate
issuance although our new root(TUBITAK Kamu SM SSL Kok Sertifikasi -
Surum 1) was added in NSS. Because some of our customers (all of them are
government entities) required using mobile devices with Android and IOS
operating systems. As you know, Android default browser Google Chrome is
using Android's trusted CA list and IOS uses Safari Browser as default.
We advised using Mozilla Firefox. You can check it on our official web
site(http://kamusm.gov.tr/duyurular/). But they insisted on using
default browsers because of their usage habits. We have been trying to submit
our new root to Android and IOS. Android and IOS do not have any Root
Submission Program like Mozilla.
Then we tried to contact to related operating systems with email. But,
really we could not get any answer from these Operating systems via all
of our effort along last two years!

So, we solved the problem by using our old root which was added in
Android, IOS, and Mozilla NSS. We know that this was a temporary
solution since our old root will expire on 21/08/2017. We set expire date
of all certificates to 21/08/2017 which is the expiry date of old root.
Please note that this old root period will expire in 4 days. Now, we will
wait  for the certificates to expire.
After expiry date old root will not used, all the certificates issued from this root will also expired.
We will have to use new root which uses 64-bit entropy. All certificates issued by this root are/will obey the rule of 64-bit entropy.
But they will live problem with Android and IOS, it is a big problem and we could not find any solution despite all our efforts.

Also, we want to inform you that now we are developing a service which
will check all CAB BR requirements. By this service we plan to be informed about any unsuitability immediately.
After, service implementation is finished, we will inform you about that.

Comment 3

[Ryan Sleevi]

Thanks for responding. I think it's still necessary to provide additional detail.

That is, we can look at the problem from two dimensions: The problem itself, and the systemic issues that allowed the problem to manifest. Your description focuses on the resolution of the problem, but doesn't indicate any systemic changes have been made. As a consequence, it does not help the community feel that your CA has taken steps to reduce the risk of future violations (of any requirement) in the future. That is, one dimension is "Did you fix the bug", but another dimension is "How was the bug introduced, why was it not detected, and what steps are you taking to prevent future bugs"

To understand how you might approach this problem, consider https://groups.google.com/d/msg/mozilla.dev.security.policy/vl5eq0PoJxY/W1D4oZ__BwAJ and how it provided a timeline of events, the steps that were already in place (and with substantial detail), where there were controls missing or mistakes made, details about the steps being taken (e.g. "We fixed the bug" is not sufficient detail to understand), and a holistic, systemic awareness of how the CA is managed and the opportunities for errors.

For this specific case:
- Note that it's not clear that you've responded to items 1-4, or if so, it may be incomplete.
- It sounds like your old root (and infrastructure) was not BR compliant, while your new infrastructure (and root) are supposed to be. Is that correct? If so, why not migrate your old infrastructure to your new infrastructure, which would have prevented this?
- Your description of needing to use the old infrastructure for leaf issuance raises a question - why did you not simply cross-certify your new root with your old root, as virtually all publicly trusted CAs have done when transitioning? That is, a version of your 'new' root (key, subject name, related attributes) signed by your old root? This would have ensured that anyone presenting a certificate from your 'new' PKI would also have a path to your 'old' PKI. Further, given many platforms ignore expiration on root certificates (as per RFC 5280), this would have allowed a full and timely transition to your new infrastructure, with minimal disruption? For more details, participants on mozilla.dev.security.policy would be happy to explain and consult on this matter to ensure an appropriate and timely transition.

Comment 4

[Tuğba ÖZCAN]

Hi Kathleen and Ryan,

We will try to answer your questions in detail:
 
@Kathleen

1) We have become aware of the problem via this Bugzilla Bug on August 17th. 
The problem reporting mechanism part for us in the ccadb includes procedures for revocation of government agencies certificates but not an email address. 
Therefore Jonathan (I guess) could not inform us about this issue. But still we should follow mozilla security group more frequently in order not to miss such issues.

2) We have stopped to issue TLS/SSL certificate from our old root which is expiring in 3 days and will begin to issue certificates from our newly added root in which the serial numbers have 64-bit entropy. 

 
3) You can find the complete list of certificates in attachment. 38 certificates was detected by Mozilla, when we examine the issue,we see 61 certificates are in problematic case.

 
4) Number of Certificates: 61

Date First: October 4th 2016

Date Last: May 30th 2017


5) Ballot 164 about this issue was considered in our new root inclusion submission period and we requested from the responsibles to resolve this issue but unfortunately they applied this remediaton only to new root but not the old one. We could not catch this issue because the problem is not reported until yesterday. 


6) Since our old root is expired in 3 days, our steps about to resolve this situation is for our newly included root:

      a) In our new root, the setting for serial number entrophy is set to 64-bit and can only be changed with the approval of at least 2 authorized personels.  

      b) We will log our certificates to CT infrastructure  

      c) We will update ccadb problem reporting mechanism part and add e-mail for reporting problem.

      d) We will follow mozilla security group daily and with replacement. 

 

Timeline of remediation steps:

      a) Done

      b) The work for logging to Test CT environment was successfully done but for the production environment we can give the timeline next week

      c) Done. Please see ccadb Report - Problem Reporting Part. 

      d) From now on

 

7) You will be aware of the regular updates in time.
 

Note Section 4.9.1.1 of the CA/Browser Forum’s Baseline Requirements, which states:

“The CA SHALL revoke a Certificate within 24 hours if one or more of the following occurs: …


The statement is clear and we are expected to revoke the problematic certificates today (we are informed yesterday) but since our root will expire in 3 days we are waiting for them to expire. But this short expiration period is really specific/unique and if there occurs an other problematic case as stated in 4.9.1.1, we will revoke them all in 24 hours.  

Since we are not revoking these certificates we will also contact with Microsoft, the only other root program, to acknowledge them with this non-compliance.
Also, during the forthcoming audit, we suppose our auditor will detect this problem because they also strictly track all the requirements and changes in CAB/BR. Lastly, we confirm that we will share our remediation plan with our auditor and our findings will be listed in audit statement. 

@Ryan

Firstly, thanks for your prompt and quick answer.

- You can find our action plan to solve the problem in our answers to Katleen's questions above.Also, we've responded to items 1-7.

- As it is written in our previous comment, we did not plan to use old root after the new root is added to trusted lists. But we was obliged
 to use old root for mobile devices. On the other hand, we did not suppose that our root inclusion process will last longer that two years.
 Please, could you share your opinion with us about this issue. Because it was the root cause of our problem!
 Also,as a part of Google family, could you direct us how can we proceed, what should be done for adding Android trusted lists?

 
- We thought using an expired certificate is not a suitable way for solution. 
Using an expired certificate was not safe for us, as you know it is so open to private key compromise.

Comment 5

[Tuğba ÖZCAN]

Attachment: Misissued SSL List.xlsx

Problematic certificate list.

Comment 6

[Tuğba ÖZCAN]

Attachment: Misissued SSL cert.rar

All problematic certificates

Comment 7

[Jonathan Rudenberg]

(In reply to Tuğba ÖZCAN from comment #6)
> Created attachment 8898948 [details]
> Misissued SSL cert.rar
> 
> All problematic certificates

I have submitted the 61 certificates in this archive to several CT logs.

Comment 8

[Jonathan Rudenberg]

I emailed a problem report about this certificate, can you please investigate and explain why it was not received?

Here are the relevant message headers:

  From: Jonathan Rudenberg <[e-mail]>
  Subject: Misissuance - serial numbers with <64 bits of entropy 
  Message-Id: <[e-mail]>
  Date: Thu, 10 Aug 2017 13:46:18 -0400
  To: [e-mail]

Comment 9

[Jonathan Rudenberg]

(In reply to Jonathan Rudenberg from comment #7)
> I have submitted the 61 certificates in this archive to several CT logs.

These certificates are now listed at: https://misissued.com/batch/16/

Comment 10

[Tuğba ÖZCAN]

(In reply to Jonathan Rudenberg from comment #8)
> I emailed a problem report about this certificate, can you please
> investigate and explain why it was not received?
> 
> Here are the relevant message headers:
> 
>   From: Jonathan Rudenberg <[e-mail]>
>   Subject: Misissuance - serial numbers with <64 bits of entropy 
>   Message-Id: <[e-mail]>
>   Date: Thu, 10 Aug 2017 13:46:18 -0400
>   To: [e-mail]

Hi Jonathan,

Unfortunately your mail has fallen in the spam box. We are very sorry for this situation. You can be sure that the necessary action is taken to ensure that this situation does not happen again. We will use a new group for this purpose only. The group address is [e-mail]. Also, I will update ccadb with this address. 

BR.

Comment 11

[Tuğba ÖZCAN]

(In reply to Kathleen Wilson from comment #0)

> 
> We expect that your CA will work with your auditor (and supervisory body, as
> appropriate) and the Root Store(s) that your CA participates in to ensure
> your analysis of the risk and plan of remediation is acceptable. If your CA
> will not be revoking the problematic certificates as required by the BRs,
> then we recommend that you also contact the other root programs that your CA
> participates in to acknowledge this non-compliance and discuss what
> expectations their Root Programs have with respect to these certificates.
> 

Hi Kathleen,

We would like to mention that we sent email to Microsoft Trusted Root Program today to acknowledge about this non-compliance.

Comment 12

[Ryan Sleevi]

Having completed the proposed immediate remediation steps, with the exception of proactively disclosing new certificates via CT, I will be marking this issue as Resolved/Fixed.

There are still opportunities for further mitigations to minimize the risk of future misissuance, such as the integration of automated tools such as certlint to examine certificates (the tbsCertificate) prior to issuance. An example of one CA doing so is at https://groups.google.com/d/msg/mozilla.dev.security.policy/oTQ9OYgS8D4/KRFUWERDAgAJ , which involves taking a tbsCertificate prior to signing, encapsulating it with the same Algorithm ID in the signatureAlgorithm, a zero length signature field, and passing to such tools.

I'm going to flag this as NeedsInfo for Kathleen to ensure that the problem reporting mechanism mentioned in Comment #10, below, is updated in CCADB.

> We will use a new group for this purpose only. The group address is [e-mail]. Also, I will update ccadb with this address.

Comment 13

[Kathleen Wilson]

(In reply to Ryan Sleevi from comment #12)
> I'm going to flag this as NeedsInfo for Kathleen to ensure that the problem
> reporting mechanism mentioned in Comment #10, below, is updated in CCADB.
> 
> > We will use a new group for this purpose only. The group address is [e-mail]. Also, I will update ccadb with this address.

Updated.