Closed Bug 1311832 Opened 8 years ago Closed 7 years ago

StartCom: Action Items

Categories

(CA Program :: CA Certificate Root Program, task)

Product:
CA Program
Component:
CA Certificate Root Program
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: kathleen.a.wilson, Assigned: kathleen.a.wilson)

Details

(Whiteboard: [ca-incident-response])

Attachments

(1 file)

SC-02-summary-report.pdf
68.25 KB, application/pdf
Details 
As per Bug #1309707, new certificates issued after October 21, 2016 that chain up to certificates with the following Subject Distinguished Names will no longer be trusted in Mozilla products, beginning with Firefox 51.

1) CN=StartCom Certification Authority, OU=Secure Digital Certificate Signing, O=StartCom Ltd., C=IL 
2) CN=StartCom Certification Authority G2, OU=null, O=StartCom Ltd., C=IL 

StartCom may apply for inclusion of new (replacement) root certificates[1] following Mozilla's normal root inclusion/change process[2] (minus waiting in the queue for the discussion), after they have completed all of the following action items, and shown that WoSign has no control (people or code) over StartCom.

1. Provide a list of changes that the CA plans to implement to ensure that there are no future violations of Mozilla's CA Certificate Policy and the CA/Browser Forum's Baseline Requirements.

2. Implement the changes, and update their CP/CPS to fully document their improved processes. The CP/CPS must explicitly state that it is prohibited to backdate the notBefore of certificates by more than one day.

3. Provide a public-facing attestation from a Licensed WebTrust Practitioner[3] acceptable to Mozilla that the changes have been made. This audit may be part of an annual WebTrust CA audit.

4. Provide auditor[3] attestation that a full performance audit has been performed confirming compliance with the CA/Browser Forum's Baseline Requirements. This audit may be part of an annual WebTrust BR audit. 

5. Provide auditor[3] attestation that a full security audit of the CA’s issuing infrastructure has been successfully completed.

6. 100% embedded CT for all issued certificates, with embedded SCTs from at least one Google and one non-Google log. The CA should not fulfill the non-Google log requirement by using logs that they run themselves. For as long as they do so, they will need to demonstrate ongoing evidence of efforts to get other logs to take their volume, and why those efforts have not been successful.

Notes:
[1] The new (replacement) root certificates may be cross-signed by the Affected Roots. However, the Affected Roots may *not* be cross-signed by the new (replacement) root certificates, because that would bring the concerns about the Affected Roots into the scope of the new roots. Due to the way we are implementing the distrust, the new root certificates must have a Subject Distinguished Name that does not overlap with the Subject Distinguished Names listed above.
[2] Mozilla's root inclusion/change process includes checking that certificates in the CA hierarchy comply with the CA/Browser Forum's Baseline Requirements.
[3] The auditor must be an external company, and approved by Mozilla.
Whiteboard: Incident Action Items
Would this be why my email digital signatures suddenly stopped working, even though the certificates were issued last January.  I get no error other than the certificate can not be found.
Matt: no.

Gerv
Matt - if you're having issues with email signatures (I'm assuming in Thunderbird?), please file a new bug here: https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=Security%3A%20PSM with as much detail as you can include. Thanks!
Flags: needinfo?(unicorn.consulting)
Bug 1325901 filed
Flags: needinfo?(unicorn.consulting)
Thankfully all but one of the Startcom certificates that I manage was minted before the cutoff date for this action ... but because that certificate won't work in browsers any more, I had to replace it with a certificate from another provider.

If these had been free certificates, then I wouldn't mind so much ... but we have paid Startcom for their services.  The amount we paid was significantly less than most providers, but it's still very frustrating.

Is there much chance of Startcom's efforts resulting in their certificates being useful again?
trncfrmcn: this is not the right place for your comments. Please don't add more.

Gerv
Component: CA Certificates → CA Certificate Mis-Issuance
Whiteboard: Incident Action Items → [ca-incident-response]
The requirements in comment 0 have been communicated to StartCom, and so this bug is currently not actionable. If and when StartCom reapply for inclusion, we can make sure they have met all these conditions.

Gerv
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → INCOMPLETE
Summary: StartCom Action Items → StartCom: Action Items
Product: mozilla.org → NSS
Hi all,

I´d like to update this bug because StartCom just applied for inclussion in Mozilla again.

1.- List of changes

As stated in the remediation plan, StartCom has followed and finished all these steps. 

a.- StartCom is now a 100% Qihoo360 owned subordinate Company. Management has also changed. 
b.- There´re no StartCom employees working at any Wosign premises. StartCom has subcontracted Qihoo 360 for all PKI and development management.
c.- StartCom acquired EJBCA PKI software from Primekey (CA, VA and TSA). There´s no in-house development for PKI
d.- All StartCom servers are under Qihoo 360 premises in different locations, in China and US.
e.- StartCom has developed a new CMS system and website, using a new language, PHP, from scratch.  

2.- External pentesting

StartCom hired Cure53 as suggested by Mozilla. They were analyzing the new web and CMS system, firstly separately, and finally, integrated with the overall system. Only when the overall system was Ok, StartCom started to issue certificates from it.

3.- Webtrust audit

StartCom hired PwC for doing a full webtrust audit. There were findings, and they are reflected in the reports, but all were fixed. StartCom will perform an additional Webtrust audit after summer. The audit reports can be found at StartCom website (www.startcomca.com/policy)

4.- CT

StartCom logs all the SSL issued certificates in several CT logs. Currently in Google and StarCom CT log servers. StartCom used Venafi as well but was disqualified. It was also requested to use a not own-managed CT log and contacted several CT log server providers but with different answers or no answers. StartCom has contacted CNNIC, Symantec and Digicert but with no results. Also contacted Comodo, and they are willing to include the new StartCom certs when included in Chrome, once this is done, will start also publishing at new Comodo CT logs.


If you need further clarification, don´t hesitate in asking.

Regards

    
    

    
  
(In reply to Iñigo from comment #12)
What does the following statement mean...

> 4.- CT
> 
. StartCom used Venafi as well but was disqualified.

    
    

    
  
Hi Ken,

the Venafi CT log was disqualified by Google and hence not able to use anymore. StartCom used that CT log and when was disqualified we have to change our "CT configuration" for not logging/using that CT log.
The disqualification was effective on March 28, 2017. Check this: https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CACvaWvY6hvnL1QM0q01aw%3D6oug9Hh6v8AWrvVz-DEzCs0WKYHA%40mail.gmail.com.

Regards

    
    

    
      
  

    
    

    
      
  

    
    

    
  
>  the previous post is intended not only to mock this, but

Bugzilla is our professional working environment as much as it is our issue tracker, and this is not an acceptable use of it. If you'd like to continue participating in this or any bug, please take a moment to read the Bugzilla Etiquette And Contributor Guidelines, and take them to heart before doing so. 

https://bugzilla.mozilla.org/page.cgi?id=etiquette.html

Should you elect not to meet those standards in your participation here, my next action will be to suspend your account.

If you'd like to discuss that, feel free to email me directly and have a good weekend.
QA Contact: gerv

    
    

    
  
Hi all,

I´ve realized that there has not been a good communication path to announce all the tasks and actions performed by StartCom during this time and this email will try to remediate it. I´d also like to ask you for some feedback, comments and/or suggestions on how to improve. I think we´ve done a great effort and developed a robust system but it´s also true that we have had some errors even though all were fixed timely (despite the discussions that are still alive regarding some of them) and put all the countermeasures needed for not happening again. 

Thanks for all your suggestions and comments. I will try to summarize everything.

Areas

1. Remediation plan

a. Legal structure and management separation
b. operations separation
c. system separation
d. CT logging

2. Mozilla requirements (bugzilla #1311832)

a. provide a list of changes
b. implement changes and update CP/CPS, stating explicitly backdating is prohibited
c. providing full webtrust audit
d. security audit
e. All certs CT logged, if possible not using own CT log

3. Issues

a. test certificates
b. pre-certificates for test certificates
c. use of specific curves not allowed by Mozilla but BRs
d. Country code
e. RSA parameters
f. Unicode vs punnycode

4. Improvements

a. apply newest version of EJBCA v6.9.0
b. integrate cablint/x509lint in CAProxy
c. check of CSRs
d. integrate crt.sh in database


1. Remediation plan
This is what StartCom planned to do and was indicated last year in October. It was presented to Mozilla and also in an email sent to the m.d.s.p
These are the main tasks performed, which are also included in bugzilla #1311832

a. Legal structure and management separation
StartCom HK, which is the main head of the group, is owned 100% by Qifei International development Co. Ltd in HK, which is also controlled 100% by 360 technology Inc. In november 2016, there were some news indicating the completed transer of 100% shares from WoSign to Qihoo 360, so StartCom officially became a wholly owned subsidiary of Qihoo 360.
So, there´s no control from Wosign related to StartCom.

StartCom has offices at China, UK, Israel and Spain.
Besides there was a change in the management in which Xiaosheng Tan of 360 was named chairman of the board and Iñigo Barreira, CEO.
Richard Wang has no relation, control nor influence in StartCom. It was removed from the StartCom directors at the UK office last year. (Note: recently had to be named again due to some issues related to the bank that have been already fixed and hence removed again)

b. Operations separation
All operations are performed by Startcom employees, from own premises. There´s no relation at all with any other Wosign operations or departments.
All validations, support and customer service is provided by StartCom. 
Besides StartCom signed a contract with 360 to provide hosting services of the PKI infrastructure (CA, VA, TSA, CT, CMS, HSM, web,…), maintenance and support, as well as developing, testing, etc.

c. System separation
All systems used for the PKI has been updated or directly changed and all are hosted in 360 premises as stated above. 

Web and CMS have been totally updated, recoded under another programming language, PHP, for which 360 is more familiar with. This coding has been performed by the 360 R&D team and checked later by its security team. Cure53 was the firm hired for the security audit.

OTOH, StartCom finally decided to acquire the Primekey´s PKI solution, EJBCA. We considered a significant step forward use this commercial PKI software, well known in the industry.
Furthermore, the StartCom OCSP also uses the Akamai CDN as well as some other services.

d. CT logging
StartCom logs all SSL issued certificates in CT logs. Currently we´re using Google CT logs and also StartCom CT log.
We´re also planning to use Comodo CTs Mammoth and Sabre as well as the new Venafi one.

2. Mozilla requirements (bugzilla #1311832)

a. provide a list of changes
The remediation plan was the whole list of changes that were presented to Mozilla and they were agreed with that plan.

b. implement changes and update CP/CPS, stating explicitly backdating is prohibited
All those changes have been implemented as per the remediation plan stated above. The CPS was updated accordingly

c. providing full webtrust audit
The full webtrust audits have been performed by PwC. StartCom did a Webtrust for CA, webtrust BR, webtrust EV and webtrust for CS. There were some findings, which are reflected in the reports, but most of them were fixed. Right now only the configuration of the TSA and the update of the Business Continuity Plan are delayed, which in any case, will be finished soon. 

d. Security audit
For the security audit Mozilla recommended 2 firms they worked with them in the past and finally hired Cure 53. 

e. All certs CT logged, if possible not using own CT log
StartCom logs all the SSL certs issued. For not using our own CT log as requested, we contacted other CT log providers to know if they were willing to accept our certs. We contacted CCNIC, Symantec and Digicert and in some were refused and some didn´t answer. Recently contacted Comodo and accepted us. Also checked the Venafi 2 log.

Note: the logging of all issued certs were done in different steps, firstly we started with EV (because we issue very few), and gradually when realized that everything worked fine, move to the other certificate types, OV, IV and finally DV, and hence, there are some delays from the first one logged, and EV, and first DV logged. We´re working in a solution to log all those issued not logged.
 
3. Issues

a. Test certificates
It´s been detailed in bugzilla #1369359. There´s an attachment with a detailed explanation what happened, when, who, what was done to remediate and to avoid it happening in the future. Those certs were all the time under our control and lived for some minutes while testing the CT behaviour.
Actions: removed issuance rights for the CA administrator

b. Pre-certificates
It´s explained in bugzilla #1386894. Perhaps I was wrong with the word “intent” and then no need to revoke as they were not real certificates, but once we had to do it, had to work with Primekey to revoke those certs, because they didn´t exist for the EJBCA system as they only have certificates, not pre-certificates. 
Actions: all pre-certs revoked 
 
Recently there are some emails threads in the m.d.s.p as if these pre-certificates should be considered certificates, what to do with them, etc. If the same rules of 24h revoking time applies, … so I think it would be good to have same/similar/specific requirements for these certs.
 
c. use of specific curves not allowed by Mozilla but BRs and errors related to this like unallowed key usages for this.
This is also covered in bugzilla #1386894. 
According to the BRs, the use of other curves, like P-521, are allowed but not in Mozilla, so dediced to revoke them and put countermeasures to avoid them again. We´re not allowing the use of those curves.
Additionally to this, the use of those curves implied a change in the profile because of the different key usages, the key encipherment instead of the key agreement, and double checked the profiles defined in the EJBCA system. 
Action: denied the use of these curves and double checked of the profiles in the system

d. Country code
There was an issue with an old country code for Zaire, which changed the official name to the Democratic Republic of Congo. Once realized of the error, revoked the cert, and replaced with a new one correctly. 
Actions: reviewed the whole list of ISO country codes and updated our Database accordingly. 

e. RSA parameters
One of the certs had an error, as stated in https://crt.sh/?opt=cablint&id=160150786
We revoked the certificate when were informed as the first step in this case and started to investigate that error and why happened. We contacted Primekey and they didn´t control/check that at the issuance, then decided to implement a solution in our CMS system to check the CSR files
Actions: application developed to check CSR before sending to EJBCA

f. Unicode vs punnycode or bad characters in domain names
This issue has been replied in the m.d.s.p email list and are also covered by bugzilla #1386894 and also recently requested in https://groups.google.com/forum/#!msg/mozilla.dev.security.policy/D0poUHqiYMw/Pf5p0kB7CAAJ
Well, for this particular case in which has been some discussions in CABF (one ballot failed) and in the m.d.s.p mailing list, we revoked those certs timely and changed our system to convert them to punnycode.
Actions: all “non-standard” domain names changed to punnycode 
 
 
4. Improvements

a. apply newest version of EJBCA v6.9.0
Primekey has just released the new version of EJBCA, v6.9.0 (end of august).
This new version comes with some important improvements, such as:
-	CAA automatic checking
-	Perform public key validation (RSA and ECC) 

b. integrate cablint/x509lint in CAProxy
These tools have been integrated at the issuance process. We´ll check all certificates issued and will send an email internally to act accordingly, i.e, revoking the cert and to start an inmediate investigation of the error. 

c. Check of CSRs to avoid the error “RSA keys must have a null parameter” and integrate&configure EJBCA new release 6.9.0 wich also comes with a Key validator for RSA and ECC.

d. integrate crt.sh 
Developed a script to check automatically the crt.sh database daily to inform of any possible error that has not been controlled.


Finally, I´d like to also request feedback on the use of the cross-signed certs by Certinomis. As per the disclosing according to the mozilla policy within one week, I think this requirement was set in policy version 2.5, which was not published nor required at the time when those certs were created so Certinomis acted correctly.
As Franck has explained they retained those certs until we got the Webtrust certification and you can check that we haven´t issued any certificate with that chain, firstly because we didn´t have them and later, when disclosed, because there were some doubts. 

Please, feel free to make any question, suggestion and/or request more information regarding any of the points. If you need additional details don´t hesitate in asking. 

Best regards
Product: NSS → CA Program

    
  
Component: CA Certificate Compliance → CA Certificate Root Program

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: