In regards to recent issues with the IPS SERVIDORES root certificate (O=IPS Seguridad CA, CN=IPS SERVIDORES), Mozilla is requesting that the following action items be performed by the end of October, 2009: 1) Provide an OCSP server which is alive and responding properly to requests again. 2) Provide information about the steps IPS has taken to ensure that revocation information continues to be available. 3) Provide information about the steps IPS has taken to ensure that no new embedded-nul certificates are issued. 4) Provide an explanation of why IPS is issuing certs from this root that have a longer lifespan than the root. (expires December 29, 2009) 5) Provide a timeline for a new audit against these revised practices. Do to the serious nature of recent events, Mozilla will need to turn off the trust bits for this root (essentially disabling it) if these actions are not performed by the end of October, 2009. However, if these actions are performed, Mozilla does not plan to take further action. IPS CA Information: Main page: http://www.ipsca.com Renewed Webtrust Seal for CAs: https://cert.webtrust.org/ViewSeal?id=933 Current CPS: http://www.ipsca.com/es/Certificates/CPSIPSCAv31.pdf IPS will also be creating a root-inclusion request for their updated roots. Kathleen
Per my comments in mozilla.dev.security.policy, the IPS Internet Publishing Services operations and root certificates should be examined to determine whether they too have any of the same problems as IPS Servidores. After all, it appears that the two CAs are actually the same organization.
In order to provide the requested info, let me make a list with your request in order to be answered each one. 1 - Provide an OCSP server which is alive and responding properly to requests again. 2 - Provide information about the steps IPS has taken to ensure that revocation information continues to be available. 3 - Provide information about the steps IPS has taken to ensure that no new embedded-null certificates are issued. 4 - Provide an explanation of why IPS is issuing certs from this root that have a longer lifespan than the root. 5 - Provide a timeline for a new audit against these revised practices. #1 - We are working in providing an OCSP service. We estimate that it could be working at the end of next week. Anyway, we will keep you updated. #2 - Revocation info is available through CRL. Two sites are active in order anyone can download revocation info. URI: http://www.ipsca.com/ipsca2002/ipsca2002CLASEA1.crl URI: http://wwwback.ipsca.com/ipsca2002/ipsca2002CLASEA1.crl #3 - Null prefix attacks in certificates is a hacking technique consisting in introducing null characters in the CN of the server when a certificate signed request (CSR)is generated. These null characters are not interpreted by browsers and this is because hackers have discovered a way to fool browsers when accessing https secure sites. Last 2nd of August, 2009 we were warned by Daniel Veditz, Mozilla Security Team about some certificates which were issued by us. These certificates were issued with null characters inside. We stopped issuing certificates at once and immediately we revised all issued certificates in order to find out the scope of the problem. There were found several certificates which were revoked immediately and a new CRL was published. Certificate issuance was not retaken until our procedures were completely revised and an additional check for null characters inside requests was added to the procedure. After these measures have been taken, our activity was started again, and now we are watching closely all requests received in order to not have such a problem again. All null-prefix issued certificates, identified and revoked are in the CRL. You can download it at: URI: http://www.ipsca.com/ipsca2002/ipsca2002CLASEA1.crl URI: http://wwwback.ipsca.com/ipsca2002/ipsca2002CLASEA1.crl #4 - The reason for issuing certificates with a longer lifespan than the root is the consequence of an ingenious system that will enable us to renew the signature of the root CA of IPS SERVIDORES, therefore granting the continuity of those certificates issued with the intermediate CA CLASE A1. Consequently, the impact of the close expire date of the root CA will be controlled thanks to the resigning of keys. Unfortunately there have been several circumstances that have advice us against implementing this plan, as the algorithms used for the CA IPS SERVIDORES and for its intermediate CLASE A1, in some cases include keys which length makes them prone to a very close obsolescence o directly contain HASH algorithms which collision probability advices against its use for a long period of time. This, together with the apparition of certificates with a null character and the possibility of someone using this problem in order to create a long time period insecurity situation, made us take the decision to plan the discontinuity of certificates issued with the CA IPS SERVIDORES and prepare ourselves for its expiration date at the end of the present year. In its place, we have made a great effort and to generate two other trust hierarchies that will help us provide our clients with new certificates without any additional cost and with an extra validity period of time as regards to the already issued certificates. With this mechanism we will be able to transfer all certificates issued to the new trust hierarchies, therefore providing safer certificates and avoiding certificate petitions with null characters. The result will be of a total cleaning. We only have two requirements: preserve the trust of browsers in IPS SERVIDORES until the end of its lifecycle and speed up the recognition of the new CAs in the corresponding browsers. In this great effort to improve our CA, we have spent a great part of the present year preparing these new hierarchies in order to be as safe as browsers demand and we trust in the acceptance of these new CAs. Regarding the root CAs that have been recognized by Firefox, none of them is used to issue server certificates and these aspects cannot be applied. Moreover, the use of 1024 keys and MD5 algorithm does not make them the best candidates in order to replace IPS SERVIDORES with the best guarantees nowadays, although it does not rule them out either for their corresponding mission. #5 - Our next audit on procedures is scheduled for next March 2010. We are sure that the measures which have been taken are good enough in order to detect and discard CSRs with null characters, so we will keep that date for the audit which will confirm our practice is good. Do not hesitate to ask whatever you consider necessary in order to close this issue.
Here are some clarifications from IPS. The other IPS roots that are currently included in Mozilla are CN = IPS CA Timestamping Certification Authority CN = IPS CA Chained CAs Certification Authority CN = IPS CA CLASE1 Certification Authority CN = IPS CA CLASE3 Certification Authority CN = IPS CA CLASEA1 Certification Authority CN = IPS CA CLASEA3 Certification Authority None of these roots have SSL certificates in their hierarchy, so the Websites trust bit should be turned off for them. These roots use the same OCSP responder as the IPS SERVIDORES root. IPS is working on providing an OCSP service which will be used by all of these roots. In regards to the expiration of the IPS SERVIDORES root, IPS plans to re-issue all of those certs (currently chaining up to the IPS SERVIDORES root) under a new root. The new roots will be included in their upcoming audit.
Status: NEW → ASSIGNED
IPS had their OCSP service running on Friday and have been running tests over the weekend. They want to add another server or two to their configuration before claiming that the problems have all been resolved. They have requested one more day to complete this, and then they will post their official response in this bug tomorrow.
After having the service running till last Friday, we have done many tests during the weekend, Monday and Tuesday. All these operational tests have been successful but we have detected a bug of our OCSP software provider during the stress tests. We have tried to avoid this issue balancing the OCSP requests into as many servers as we could set up in this days, but we know that this is only a patch before the bug is being solved. Our provider is doing a huge effort to solve it, and we expect to have the patch by the end of this week. Regarding to disable the SSL trust bit from our root CA we strongly think that this would not be worth for the internet more secure. We are doing the things right and we have updated the required auditories. The decision of deactivate that bit, in the short term what it makes it will be left thousands of sites properly protected with our certificates, see in the Firefox users as untrusted. We consider that the only thing that this will make is to damage Mozilla and our customers interest, what, I am sure that Mozilla will agree, should protect. In the interest of the security, we expect that Mozilla will decide to do the best for all parts and be conscious of the possible business objectives that could be moving some opinions appeared. Although our CRL is being working properly and can be checked at anytime, we apologize for this inconvenient. We are collaborating with other companies and players in the market, and doing all the human and economical possibly effort for have the OCSP working to give the best service as possible to our thousands of customers.
We have received a fix from our software provider and we have installed in a couple of servers in order to monitorize wether it solves the problem. As we have currently six servers balanced, we will increase the number of OCSP servers in order to increase availability of the service meanwhile the fix is being monitored. This will last until tuesday when we will install the fix in the remaining servers provided that monitoring is successful.
The fix provided has been proved until today with good results. Anyway, we keep experiencing some troubles on certain hours when we reach a peak of users. We are not treating this issue as minor, so we are working in order to make it dissapear completely. We are working back to back with our ocsp software provider. We think this issue will be over in two or three days.
FYI/FWIW and just as a data point, I attempted some manual OCSP queries to various of the IPS SERVIDORES ocsp DNS names (ocsp.ips.es -> ips.certiver.com -> ocsp.ipsca.com) today 18-Nov-2009 between the hours of approx 1500h PST and 1630h PST, and consistently received a response of either.. "error:27070072:OCSP routines:OCSP_sendreq_bio:server response error:ocsp_ht.c:147:Code=403,Reason=Forbidden ( The server denied the specified Uniform Resource Locator (URL). Contact the server administrator. ) " ..or the query hung forever (and eventually timed out) and I never received a response. I tried from two different machines on the internet, and i tried using both "http" and "httpS" URLs. I don't have time right now to examine my test results in detail, but it seems the connection timeouts were (only) when trying to use an "httpS" OCSP URL (yes, I know that the URLs given in the AIA field of the certs is an "http" one, but the IPS ocsp service used to answer on https for whatever reason back in Oct-2009). The above error:27070072 appears to be returned consistently for ocsp queries using "http" URLs. Hope this helps.
First of all, thanks a lot JeffH for your remarks. They are helpful for clarifying some aspects of this issue. We are currently focusing on IPS SERVIDORES, as the other roots have no certificates to be managed, so please note that all OCSP which points to ocsp.ips.es are not active. We have assumed that the following roots will be removed the SSL trust bit, so please, in order not to have interferences in this job, please you can remove the SSL trust bit from the other IPS roots that are currently included in Mozilla which are: CN = IPS CA Timestamping Certification Authority CN = IPS CA Chained CAs Certification Authority CN = IPS CA CLASE1 Certification Authority CN = IPS CA CLASE3 Certification Authority CN = IPS CA CLASEA1 Certification Authority CN = IPS CA CLASEA3 Certification Authority We are focused on providing a good response for ocsp.ipsca.com which corresponds to the hierarchy under IPS SERVIDORES. I confirm hereto that all our efforts are being performed in order to grant OCSP responses from ocsp.ipsca.com until the end of the livecycle of the root IPS SERVIDORES. At this moment we are getting a good response from that ocsp, but we estimate that it may be better. We have identified a problem in a networking element which makes the load balancer dropping more packets that it would desirable. So we are working in moving services behind another load balancer.
Last weekend there were moved all the OCSP responders which were behind the problematic networking element it was needed to syncronize against an accurate NTP server. All DNS records were changed and then some tests were performed. At the time I am updating this entry some tests are being performed from several countries. The general feeling is that OCSP responses are quite good in time and accuracy. I will post some results in a couple of hours, but it seems the OCSP problem is almost over.
Please find results from tests performed in the latest hours from Madrid, Paris and Mexico. Tests were performed against the same hosts from three different separated places in the world and performed every 10 minutes. Results are as follows: spain Request date/time Response time(Sec) Result 23/11/2009 20:21 0,14 SUCCESS 23/11/2009 20:31 0,11 SUCCESS 23/11/2009 20:41 0,13 SUCCESS 23/11/2009 20:51 0,11 SUCCESS 23/11/2009 21:01 0,13 SUCCESS 23/11/2009 21:11 0,13 SUCCESS 23/11/2009 21:21 0,54 SUCCESS 23/11/2009 21:31 0,54 SUCCESS 23/11/2009 21:41 0,39 SUCCESS 23/11/2009 21:51 0,52 SUCCESS 23/11/2009 22:01 0,26 SUCCESS 23/11/2009 22:11 3,31 SUCCESS 23/11/2009 22:21 2,15 SUCCESS 23/11/2009 22:31 0,42 SUCCESS 23/11/2009 22:41 0,52 SUCCESS 23/11/2009 22:51 0,76 SUCCESS 23/11/2009 23:01 2,81 SUCCESS 23/11/2009 23:11 1,03 SUCCESS 23/11/2009 23:21 0,34 SUCCESS 23/11/2009 23:31 0,13 SUCCESS France Request date/time Response time(Sec) Result 23/11/2009 20:33 0,51 SUCCESS 23/11/2009 20:43 0,43 SUCCESS 23/11/2009 20:53 0,66 SUCCESS 23/11/2009 21:03 0,32 SUCCESS 23/11/2009 21:13 0,25 SUCCESS 23/11/2009 21:23 0,22 SUCCESS 23/11/2009 21:33 0,67 SUCCESS 23/11/2009 21:43 0,56 SUCCESS 23/11/2009 21:53 0,72 SUCCESS 23/11/2009 22:03 0,98 SUCCESS 23/11/2009 22:13 2,34 SUCCESS 23/11/2009 22:23 3,12 SUCCESS 23/11/2009 22:33 1,77 SUCCESS 23/11/2009 22:43 0,42 SUCCESS 23/11/2009 22:53 0,44 SUCCESS 23/11/2009 23:03 0,75 SUCCESS 23/11/2009 23:13 0,12 SUCCESS 23/11/2009 23:23 0,13 SUCCESS 23/11/2009 23:33 0,44 SUCCESS 23/11/2009 23:43 0,56 SUCCESS Mexico Request date/time Response time(Sec) Result 23/11/2009 13:42 0,11 SUCCESS 23/11/2009 13:52 0,12 SUCCESS 23/11/2009 14:02 0,11 SUCCESS 23/11/2009 14:12 0,23 SUCCESS 23/11/2009 14:22 0,29 SUCCESS 23/11/2009 14:32 0,44 SUCCESS 23/11/2009 14:42 0,15 SUCCESS 23/11/2009 14:52 0,32 SUCCESS 23/11/2009 15:02 0,55 SUCCESS 23/11/2009 15:12 0,86 SUCCESS 23/11/2009 15:22 0,97 SUCCESS 23/11/2009 15:32 2,11 SUCCESS 23/11/2009 15:42 1,13 SUCCESS 23/11/2009 15:52 2,23 SUCCESS 23/11/2009 16:02 1,14 SUCCESS 23/11/2009 16:12 0,49 SUCCESS 23/11/2009 16:22 0,36 SUCCESS 23/11/2009 16:32 0,27 SUCCESS 23/11/2009 16:42 0,88 SUCCESS 23/11/2009 16:52 0,23 SUCCESS
It has been confirmed that the OCSP responder for the IPS SERVIDORES hierarchy is now working. This completes the action items that were listed in this bug. There are two new bugs in regards to IPS root certs: 1) Bug #529286 -- Root inclusion request for the new IPS root 2) Bug #530853 – Request to turn off all three trust bits for the six “IPS Internet publishing Services” root certs. This set does NOT include the IPS SERVIDORES root, which was the topic of this bug. For the other roots, IPS has confirmed that all three trust bits may be turned off.
Status: ASSIGNED → RESOLVED
Last Resolved: 10 years ago
Resolution: --- → FIXED
FYI, various people are noting on the firstname.lastname@example.org list that they are having at least some problems with queries sent to ocsp.ipsca.com today.
I've just subscribed to email@example.com in order to have those feedback. Thanks a lot JeffH.
You need to log in before you can comment on or make changes to this bug.