1467414 - GDCA: Misissuance of certificates with small RSA keys

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Rob Stradling]

GDCA have issued 4 precertificates during the past week that contain RSA-1024 keys:

https://crt.sh/?id=496289019&opt=cablint,zlint,x509lint
https://crt.sh/?id=506519022&opt=cablint,zlint,x509lint
https://crt.sh/?id=506945512&opt=cablint,zlint,x509lint
https://crt.sh/?id=506962000&opt=cablint,zlint,x509lint

<RSA-2048 keys have been forbidden by the BRs for several years.

Comment 1

[capoc]

Hi Rob,

Many thanks for your e-mail and the report.
 
We noticed this issue and we are handling this now. We will post the findings here and the mozilla.dev.security.policy soon.

Thanks.

Comment 2

[capoc]

Hi Rob,

Thank your again for bringing this to our attention. 


We conducted an investigation, and the following is our initial report. I will post the report to the mozilla.dev.security.policy in a minute. 


1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, via a discussion in mozilla.dev.security.policy, or via a Bugzilla bug), and the date.

We became aware of the problem via an e-mail sent to GDCA’s Problem Reporting Mechanism ([e-mail]) by Mr. Rob Stradling, the e-mail was sent at June 7, 2018 18:10 (UTC+8). Mr. Rob Stradling also filed a report at June 7, 2018 18:07 (UTC+8) via Bugzilla (https://bugzilla.mozilla.org/show_bug.cgi?id=1467414). 


2. A timeline of the actions your CA took in response.

   A. June 7, 2018 18:10 (UTC+8)- Mr. Rob Stradling sent the e-mail to [e-mail]
   B. June 7, 2018 18:40 (UTC+8) – GDCA became aware of the reported problem;
   C. June 7, 2018 18:58 (UTC+8) – GDCA suspended the issuance of the GDCA DV SSL certificates;
   D. June 7, 2018 19:10 (UTC+8) – GDCA replied Mr. Rob Stradling’s e-mail, indicated that we were looking into the issue;  
   E. June 7, 2018 20:30 (UTC+8) – GDCA confirmed the mis-issuance of the reported certificates;
   F. June 7, 2018 21:00 (UTC+8) – GDCA revoked the 4 mis-issued certificates;
   G. June 7, 2018 21:27 (UTC+8) – GDCA notified the subscribers that the mis-issued certificates were revoked;
   H. June 8, 2018 09:30 (UTC+8) – GDCA identified the reason of the mis-issuance;
   I. June 8, 2018 11:43 (UTC+8) – GDCA found three additional DV SSL certificates that were mis-issued through scanning all the SSL certificates issued by the GDCA TrustAUTH R5 ROOT and its Subordinate CAs;
   J. June 8, 2018 12:04 (UTC+8) – GDCA revoked the additional three mis-issued certificates, notified the subscribers through the e-mail addresses of the domain owners.



3.Confirmation that your CA has stopped issuing TLS/SSL certificates with the problem.

  GDCA suspended the issuance of DV SSL certificates as of June 7, 2018 18:58 (UTC+8). 


4.A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

A total of 7 certificates were mis-issued and for the same reason, these certificates were issued between December 06,   2017 and June 05, 2018. 

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Certificate 1: https://crt.sh/?id=496289019
Certificate 2: https://crt.sh/?id=506519022
Certificate 3: https://crt.sh/?id=506945512
Certificate 4: https://crt.sh/?id=506962000
Certificate 5: serial=5C1C8443711D52FCA1051201E7B188B3 (attached)
Certificate 6: serial=71A4A5DEE3921C27D0BF3B37052EF5C0 (attached)
Certificate 7: serial=161DE0BAE09F3071B8A7B5ABB91FBBDC (attached)

6.Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

After conducting an investigation, we found that a bug was introduced during an upgrade in our certificate issuance system which was misconfigured later, causing the failure of detection on the minimum RSA key size. 



7.List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

  A. Suspended the issuance of DV SSL certificates;
  B. Scanned all the SSL certificates issued by the GDCA TrustAUTH R5 ROOT and its Subordinate CAs to find out if other certificates with small RSA keys were mis-issued;
  C. We are currently fixing the bug in the issuance system and working to have it correctly configured by June 10, 2018;
  D. We are adding a function in the key parts of the issuance system to regularly detect the minimum RSA key size, and such function is expected to be enabled by June 20, 2018. 


Your further comments and suggestions will be much appreciated.

Thanks!


Xiu Lei 
GDCA

Comment 3

[capoc]

Attachment: serial=5C1C8443711D52FCA1051201E7B188B3.crt

Comment 4

[capoc]

Attachment: serial=71A4A5DEE3921C27D0BF3B37052EF5C0.crt

Comment 5

[capoc]

Attachment: serial=161DE0BAE09F3071B8A7B5ABB91FBBDC.crt

Comment 6

[Rob Stradling]

Thanks Xiu.  I've submitted certificates 5, 6 and 7 to some CT logs:
 
https://crt.sh/?id=519216976
https://crt.sh/?id=519216977
https://crt.sh/?id=519216982

Comment 7

[Ryan Sleevi]

(In reply to capoc from comment #2)
> 6.Explanation about how and why the mistakes were made or bugs introduced,
> and how they avoided detection until now.
> 
> After conducting an investigation, we found that a bug was introduced during
> an upgrade in our certificate issuance system which was misconfigured later,
> causing the failure of detection on the minimum RSA key size. 

Thanks for your quick reply!

That sounds like a very interesting bug to have, and one that could apply to any checks - including domain validation steps. Could you speak a bit more about what your change management process is? That is, I'm assuming you have tests for all new versions to make sure no major non-conformities, and understanding if this was an error that was tested for as part of the upgrade testing would be useful, and may reveal further opportunities to improve.

Similarly, it sounds like this was detected by a third-party running post-issuance lint steps. Do you have any plans to run pre-issuance linting, to help detect issues? Similarly, to run post-issuance checks for non-conformities?

Comment 8

[capoc]

Hi Ryan,

Many thanks for your comment. 

1.With regard to our system change management, we follow the following internal process:

Change Request – Change Request Approval – Change Solution Design and Review – Development Implementation – Test Validation (under the TEST system) – Technical Review – Review and Approval of System Release – System Release    

Since December 1, 2017, the issuance of the GDCA DV SSL certificates migrated from manual validation mode to automatic validation mode. We executed the Test Validation (including prohibition test on the issuance of the RSA 1024 bits certificates) according to our change management process before the release of the system, and no abnormalities were identified for the TEST system.  

On June 7, 2018 when Mr. Rob Stradling notified us the mis-issuance of certificates with RSA 1024 bits keys, we immediately inspected our PRODUCTION system, and we found that the certificate templates under the PRODUCTION mode and those under the TEST mode are different, such a situation prohibited the issuance of RSA 1024 bits certificates under the TEST mode by virtue of the certificate templates even when a Bug existed in the RA, while the PRODUCTION mode was affected by the Bug. We failed to detect this problem in a timely manner because we did not conduct a negative test on the PRODUCTION system just to avoid any waste data. 

2.As of now, we already fixed in the Bug in the RA, making sure to check RSA key size of the CSR submitted by subscribers in real time. Meanwhile, we have already adjusted the certificate templates under the PRODUCTION mode, and prohibited any further possible issuance of certificates with RSA 1024 bits keys.

In addition, we are working actively on relevant integrated linting development. We will, as recommended by Mozilla, integrate the three tools (cablint, x509lint, and zlint) into our issuance system. We expect to begin implementing pre-issuance and post-issuance checks for non-conformities by June 20, 2018. 

Thanks.

Xiu Lei 
GDCA

Comment 9

[Ryan Sleevi]

(In reply to capoc from comment #8)
> Hi Ryan,
> 
> Many thanks for your comment. 
> 
> 1.With regard to our system change management, we follow the following
> internal process:
> 
> Change Request – Change Request Approval – Change Solution Design and Review
> – Development Implementation – Test Validation (under the TEST system) –
> Technical Review – Review and Approval of System Release – System Release    
> 
> Since December 1, 2017, the issuance of the GDCA DV SSL certificates
> migrated from manual validation mode to automatic validation mode. We
> executed the Test Validation (including prohibition test on the issuance of
> the RSA 1024 bits certificates) according to our change management process
> before the release of the system, and no abnormalities were identified for
> the TEST system.  
> 
> On June 7, 2018 when Mr. Rob Stradling notified us the mis-issuance of
> certificates with RSA 1024 bits keys, we immediately inspected our
> PRODUCTION system, and we found that the certificate templates under the
> PRODUCTION mode and those under the TEST mode are different, such a
> situation prohibited the issuance of RSA 1024 bits certificates under the
> TEST mode by virtue of the certificate templates even when a Bug existed in
> the RA, while the PRODUCTION mode was affected by the Bug. We failed to
> detect this problem in a timely manner because we did not conduct a negative
> test on the PRODUCTION system just to avoid any waste data. 

Thanks for these additional details! They help understand where systems and flows can go wrong.

We've seen situations like this before - where TEST and PRODUCTION configurations do not align - most notably when a CA failed to properly configure the basicConstraints extension, granting unconstrained CAs to servers!

It's unclear if your change control process broke down, or whether you were missing a step entirely, to review that the PRODUCTION and TEST configurations are identical - such as a multi-party system of review (for example, one person implements the changes, two other people review and sign-off). Understanding a bit more about how these configuration changes are made, and why these template differences weren't noticed, is important to helping the ecosystem improve.

Similarly, understanding how that configuration is examined is also helpful as part of the post-mortem process. For example, if two people have to review through the UI, then this can create situations where visually similar things are overlooked. Configuration management systems that use configuration files can be more readily and easily reviewed (for example, 'diff'ing them), reducing the risk of human error. Can you describe a bit more about your current system of management and whether there are opportunities to improve such as above?

Comment 10

[capoc]

Hi Ryan, Many thanks for your comment. 

1.The procedures for our CA system configuration changes include:

Configuration Change Request – Configuration Change Request Approval – Configuration Change Solution Review –  Configuration Test Validation (under the TEST system) – PRODUCTION System Configuration Change Approval – PRODUCTION System Configuration Change Activation (under dual control)  

This mis-issuance incident did not involve the missing of the above procedures, it was caused by the fact that the template differences between the TEST and PRODUCTION system were overlooked due to visual similarity in the UI. Lessons we learned: 

a.Negative test must be carried out (by issuing test certificates) after system upgrade or change;
b.We shall examine the effectiveness of overall security controls, and shall not place a particular focus on only one of them.

As of June 20, 2018, we have:

a.Fixed the Bug in the RA;
b.Correctly configured the certificate template in the PRODUCTION system and had it tested;
c.Integrated the three tools (cablint, x509lint, and zlint) recommended by Mozilla into our PRODUCTION system to check the pre-certificates for non-conformities;
d.Deployed backend services to periodically check the final certificates for non-conformities.


2.We currently perform configurations (including CRL release configuration, certificate templates configuration etc.) through system UI. We believe it is a great idea to implement configuration changes through modifications of the configuration files, thank you for the suggestion. 

We are considering rapid and effective configuration review as one of the demands in our next system upgrade.  

Through an analysis of this incident, we believe the best practices for the procedures of system changes should at least cover the following steps:

a.Stipulate and strictly execute change procedures;
b.Negative test must be carried out upon upgrades or changes of the PRODUCTION system,
c.The effectiveness of overall security controls must be examined, without placing a particular focus on only one of these controls;
d.Conformity tools should be used to periodically check the issued certificates for non-conformities.  

Thanks.

Xiu Lei
GDCA

Comment 11

[Wayne Thayer]

Xiu: please update this bug when all pre-issuance linting described in comment 8 has been deployed and is actively checking all certificates issued in your production environment.

Comment 12

[capoc]

Hi Wayne,
 
As mentioned in comment 10, we have completed the linting integration and deployment, we are now performing a final confirmation and will resume the DV SSL certificates issuance service on 27 June 2018 (next Wednesday). I will update this bug again by then. 
 
Thanks.
 
Xiu Lei
GDCA

Comment 13

[capoc]

Hi Wayne,
 
We just resumed our DV SSL certificates issuance service, pre-issuance linting is now enabled to check all certificates to be issued in our PRODUCTION environment for non-conformities, in addition, backend services are in place to periodically check the final certificates for non-conformities. 
 
We will follow closely any future changes to the linting tools and will update our system when necessary, we will also strengthen our management to avoid similar mis-issuance incident.
 
Thanks!

Xiu Lei

Comment 14

[Wayne Thayer]

Remediation completed.