1350615 - Camerfirma: Startcom are issuing by proxy using Camerfirma

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Samuel Pinder]

Attachment: camerfirma post.png

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.98 Safari/537.36

Steps to reproduce:

As this concerns all of Mozilla products that use NSS, I felt it appropriate to file against NSS as a whole. I apologise very much if this is not an appropriate method of giving out information in this bug. I am a customer of Startcom, and they are currently under a minimum one year ban (along with WoSign) for violating several Basline Requirements and Mozilla policies, including issuance of SHA-1 certificates by backdating to avoid browser blocks.
I have discovered that they are using what they call an 'interim' solution to get around the fact they are currently untrusted. I discovered this via their public forum, formerly available at https://forum.startcomca.com , where another customer had posted the information. Instead of issuing from their own roots and intermediates, they are somehow in some form of arrangement with another CA- Camerfirma, a CA that is included in NSS and is not currently subject to any sanctions. I don't know the precise nature of this arrangement but from various conversations I have had with Startcom via their web chat support, I've been told Startcom do the validation (for organisation details and domains) and pass along the details to Camerfirma, who then issue a certificate with the requested details. Apparently this 'solution' is only possible for Class 3 Organisational Validation and Class 4 EV Validated customers. I find this rather odd. The first concern is this: have they acquired or obtained an influence over Camerfirma? Indeed Mozilla policy states that this should be disclosed, and in the case of Startcom and WoSign, they have already failed to do this at least once. I have saved the support conversation logs and am willing to provide these if need be. I have tried to give Startcom the 'benefit of the doubt' since I cannot afford this kind of product elsewhere, and really do hope that I am mistaken in believing that something is not quite right. Indeed this morning, I discovered that Startcom's public forum has mysteriously vanished, or at least it, has been taken out of the DNS so forum.startcomca.com no longer resolves. However several pages of it still exist in Google's Cache at the time of writing, and it hasn't actually been deleted *yet*, one can still reach it by using the IP address 104.192.110.147 by manually pointing forum.startcomca.com to it via your operating system's HOSTS file. At the time of writing, the posts regarding Camerfirma are still present. As I know this *could* be deleted at any time, I have also backed up most of the forum using WinHTTrack and am willing to share this on request. Why would Startcom take this forum down if they did not have something to hide? There is no announcement about teaming up with Camerfirma, indeed the only way for a customer to find out is to actually ask Startcom's support team for a solution, R.e. obtaining a trusted certificate. Because this information 'getting out' could cause Startcom to attempt further cover-up, I would *really* appreciate if this bug was kept private, admittedly however I am also a private person and don't really want the publicity. I have attempted contact via IRC channels, and was told that submitting a private bug here was the best way to disclose private information. If you require any further information please do not hesitate to reply here or contact me via email.


Actual results:

I have succeeded, as a customer, to request a fully valid Camerfirma certificate issued against several domain names in one. My second concern (although in my case, rather convenient too I admit) is that I can attest that although I do not *own* all the domain names in the certificate, I do have full *operational control* over them, as I solely host and manage the domains on others'  behalf. This includes two voluntary organisations I am involved with in person. After looking through the policies at various other CA's, it seems even stranger that Camerfirma are allowing any domain onto a certificate that Startcom state that *they* have validated. The domain names I have included on my certificate were validated using WHOIS admin email address validation- there appear to have been no checks to see if the Registrant on every single domain matches the organisation I have Class 3 Validation for- namely Samspin LTD. All the domains in question are either registered in my real name directly- Samuel Pinder, or in the name of Blackpool Tiggers and 'Volunteer Centre' (except samspin.ltd.uk, since those domains can *only* be in the name of a matching company by Nominet UK registry policy). Normally CA's will only allow domains onto a certificate that is Class 3 OV validated if the registrant name matches with the requester, although they don't require this for Class 2 and under. For the avoidance of doubt: Samspin LTD is a non-trading dormant company that I am the sole director of that I registered to protect my online moniker from being used by anyone else as a UK company. I went through the Class 3 Validation with Startcom with this name solely to apply for the certificate. The certificate is now in use on most of my websites/systems, you may find it accessible via https://www.samspin.com , it is logged to CT here https://crt.sh/?id=107605496 and here https://crt.sh/?id=108348756 . I am aware of only one other customer whom has done this, namely Neil Gunton, whom originally disclosed this information on the Startcom, he has procured a certificate from Camerfirma (via Startcom) that can be seen by visiting https://www.neilgunton.com .



Expected results:

Startcom should at the very least, disclose their relationship with Camerfirma to reassure that they are complying with their obligations. I am concerned that they do not, and that appear to be attempting to hide this relationship. If it turns out this relationship is actually permitted under current policies then fair enough, however if it is not (and I suspect it is not, however much I hate to admit it) then I would rather disclose this information now so that things do not get any worse and so I can take appropriate steps to find another certificate solution. I will attach a screenshot of the relevant initial posting on the Startcom forum that first mentions Camerfirma.

Comment 1

[Gervase Markham [:gerv]]

Samuel: thank you for this report. I have started investigating this, but no bug had been filed, and the person who had "reported" the problem previously was... uncooperative. So I'm glad to hear from you.

I have had a message from Inigo at StartCom relating to this, and I will post it here:


Hi Gerv,

I do hope you´re having a productive meeting at Raleigh.

I´d like to let you know how we are doing regarding our remediation plan and what we´re doing for/with Camerfirma

So, firstly, I´d like to inform you that yesterday we ran succesfully our root key&certificate generation ceremony, witnessed by our PwC auditor. Next week we´re running another pentesting with Cure53 and also will start with the cross-signing of the new roots, the generation of the subCAs and with the migration plan from the current system to the new one, which is going to be a brand new as stated, with an initial planning of starting issuing certificates from our new system on April 10th aprox.

Secondly, I´ve been informed about some questions regarding our deal with Camerfirma. As you may imagine we are losing many customers due to the distrust and as you suggested during our informal meeting at Redmon we were looking for a temporary solution, being as a subCA or as an RA. You know we´re changing all of our systems, acquired a new PKI system (Primekey´s EJBCA), change all the code, servers, DB, etc. to create a new fresh system from scratch with no relation to Wosign and being a subCA of any other CA meant additional time and resources that could not be afforded at this stage. Then, decided to become an RA and then I contacted Camerfirma and we followed all the steps stated in the CABF BRs (wasn´t able to find anything in Mozilla policy) as stated in section 1.3.2. So:

-          We do have a contract with Camerfirma

-          As a CA we keep all the information for 7 years (or more) but of course, we´re providing that info to Camerfirma

-          As Camerfirma RA application is only in Spanish, I´m the only one entitled to perform the RA “actions”. I had to pass an exam with Camerfirma to demonstrate my knowledge and abilities on the PKI J

-          I was identified and then provided with a certificate to operate the RA, etc.

The way we´re working is that we perform all the validations as Startcom and when everything is Ok, we´re able to provide the customer a certificate. We also explain all customers our current situation, where we are “trusted”, reasons, etc. In the case they request an SSL cert, we try to “persuade” the customers to use ours but in case they refuse, we offer the alternative of Camerfirma. If so, then, we´ll have to do the Camerfirma “steps” and provide them all the information they usually request. So, let´s say we do it twice. When all the information is provided, Camerfirma validates again the requests and approve or not the issuance of the certificates. 

Regarding the validation of the domain, well, you know how these things have been going thru the CABF, there was 10 validation methods approved, then the patents, IPRs, etc. came into the room, 3 CAs claimed some, then, new ballot approved the new methods, later, those 3 CAs withdrawn them, and now, I think we´re in a position to re-submit another ballot to re-add the methods to the guidelines, so, it´s been a long discussion to reach where we were time ago. We started developing a new program to adapt to the changes but then those 3 CAs were changing their minds and now, I don´t know if it´s worth to keep going with the development or not.

Startcom has a method to validate the domains in which the customer can choose, but all fall in the “other method” category (3.2.2.4.11) in which we validate all domains and keep the evidence. This information is send to Camerfirma and they can confirm what we provide. Camerfirma does not contact the customers, they contact me in case there´s anything not clear enough or need additional information.
 
Finally, I´m not (and was not) aware, because I couldn´t find anywhere, that this type of deal had to be disclosed or any other action should be done (despite having read what´s going on with some Symantec Ras which is not the same issue) and the last thing I´d do is put Camerfirma in a situation in which his name or reputation can be damage because they´ve helped me in a difficult situation for Startcom and are my friends, so, even this is a temporary solution and for a limited number of certificates according to our contract, let me know if this could be a potential problem and I´ll cancel the deal with Camerfirma. I wouldn´t like them to have any issue because of anything related to Startcom.

Best regards

Iñigo Barreira

CEO
StartCom CA Limited

Comment 2

[Gervase Markham [:gerv]]

I've written to Inigo seeking more information about what he's doing.

Gerv

Comment 3

[Kathleen Wilson]

I don't know why Bugzilla changed the Group. I was just moving this bug into the "CA Certificate Mis-Issuance" component and adding the "[ca-investigation]" whiteboard tag.
I had not intended to change the group or visibility of this bug.

Comment 4

[Gervase Markham [:gerv]]

Bugzilla changed the group because the crypto-core-security group is not available in the "mozilla.org" product.

Gerv

Comment 5

[Samuel Pinder]

I thought I'd give an update here from my perspective. "forum.startcomca.com" now resolves again, and www.startssl.com has ceased to operate, with a message to visit www.startcomca.com instead. This site is visually nearly identical but now only includes their new root hierarchies at https://www.startcomca.com/root (including a new "Startcom Certification Authority G3"). On attempting to logon to the website I discovered that my client certificate no longer worked, I had to request a new one via email (and it appeared to issue from the new hierarchy "Startcom Certification Authority CS"). When I got into my account I discovered that my name, company name, address and recent invoices were still there (with the same expiration date for the validation I have paid for). However, all of my old certificates are no longer available for re-download. All of my prior domain validations are no longer present, all totally blank. Effectively it looks like a brand new account has been made with the same names, address, and email address from the old site. Therefore it is apparent that Startcom have indeed moved infrastructure with the new roots and simply imported old account details as templates for new ones. The site uses a Camerfirma certificate too. All in all it seems a little sloppy, as the imagery still showcases "www.startssl.com" on the homepage but I think they've got more to worry about than that. I have noticed that the proposed intermediates available for download do have CRL addresses embedded but these all return 404 (for example https://www.crt.sh/?id=120339950 includes the URL http://crl.startssl.com/sfsca.crl and this comes back 404). Presumably this is because nothing has actually been revoked (yet), and indeed I cannot actually find any end-entity logged certificates yet. A cross signature from their old root to their new root appears to exist according to CT logs: https://www.crt.sh/?id=120339949 . Oddly a second cross signature exists but appears to have been revoked https://www.crt.sh/?id=117764314 although they look identical from the fields available, there doesn't seem to be anywhere I can download the full cross certificate (yet). I have requested a test certificate via the new www.startcomca.com control panel to see if I can get a hold of this, but a message appeared stating "this request has been marked for approval by our personnel" and I have had no update since. There is a little clue as to how this site came into being: I think it was cloned from a backup template made last year. The URL https://www.startcomca.com/news comes up with an error saying the system is busy, however typing 'index' in the URL structure to go to https://www.startcomca.com/index/news pulls up an archive of news, which goes up to June 3, 2006, leaving quite a few newer entries from the old site no longer visible. I tried this on the off-chance after realising that several links go to https://www.startcomca.com/index/root rather than https://www.startcomca.com/root and appeared identical, so I wondered if typing 'news' on the end of 'index/' would work. I can only guess that Startcom appear to have rushed forward with the changeover but not everything is working yet. I know this is now far beyond the scope of the original report but I hope it helps to have a customer perspective. I have a sneaking suspicion that Startcom *may* request a cross signature from Camerfirma eventually, as their new roots now have the country code ES for Spain rather than IL, and Camerfirma is a Spanish CA. Indeed Startcom does have a company with their name in Spain (see http://portal.kyckr.com/AvailableProducts.aspx?code=B95856506&regAuthor=ES48001) , so I will be keeping a look out for this. Hopefully when issuing does recommence they'll have all the issues worked out and the security industry will be in a far safer place. In the meantime, I hope this information helps.

Comment 6

[Gervase Markham [:gerv]]

Inigo: we have had private conversations about this situation, but please can you summarise your position for the record (including the BR non-compliance issues and your remedial action)?

Thanks,

Gerv

Comment 7

[Iñigo]

Hi all,

I´ll try to explain all the things that have been written here and the current situation. Will try to separate Camerfirma and StartCom.

New StartCom

We´ve been changing our infrastructure as indicating in the remediation plan and I think everyone´s aware of it. Our aim is to have a new StartCom and then get rid of or give up all the "old" Startcom looked like, including the website. So:

- we have a new domain (well, not so new) startcomca.com which is going to be the main webpage of StartCom and all related stuff
- we have new roots, new subCAs and end user cert profiles. 
- the infrastructure has changed and for example OCSP, AIA, CRL links to startcomca
- we have migrated all the info from the old DB to our new DB, but it´s true that not all info was moved
- we´ve cross-signed our new roots with the old ones as suggested for not losing compatibility
- we haven´t cross-signed anything with Camerfirma and have no plans to do so
- we do have offices in Bilbao, Spain and this is one of the reasons to have the C=ES in the hierarchy. 
- the current website is not fully deployed and there could be some mistakes.
- the look&feel is currently the same for a smooth transition for our customers but there are plans to update it
- there´s a new CMS with additional features but not all have been put into production, we´re a little bit slow with this.

Just to update what Gerv has copied is that we´ve run the second external pentesting before going "live" with this new infrastructure. And we´re still making changes.

Camerfirma

- we have a contract with Camerfirma and never tried to hide it. We´ve followed all the steps indicated in the BRs
- this contract is only for OV and EV certificates and for a limited number with the idea to use "ours" asap.
- for this particular case we´re not "acting" as a CA (even we provide all the info as if it were for us) but as an external RA, or a reseller. There´s no "other" deal with Camerfirma than offering their certificates to our customers if they want trusted ones.
- regarding validations, currently Camerfirma is performing the domain and others validations for EV certs and will do so for OV certs starting next week monday at latest. We were suggested to do so and there´s no problem from our side nor from Camerfirma
- We wanted to start earlier this task but found an issue with OV certs because even all the information is validated and included in the application form and in the RA, when issued there was an error that didn´t include the locality name in the certificate. This has been finally fixed by Camerfirma yesterday. Once we were aware of the issue we haven´t issued any other OV until this had been fixed. 
- Now, today, we have started re-issuing correctly those wrong issued certificates and providing them to the customers to replace the "wrong" ones with these new ones. Once this is done, we´ll ask for revocation of those wrong issued. We plan to finish this week and hopefully revoke them all next week when customers had replaced them.


Hope this explanation clarifies our current situation as StarCom and the deal we have with Camerfirma. 

Regards

Comment 8

[Gervase Markham [:gerv]]

To be clear: until recently, Camerfirma were not re-doing the domain validations of certificates initially validated by StartCom, but Mozilla requested a change and so from next week they will be.

Gerv

Comment 9

[Gervase Markham [:gerv]]

Removing security bit and resolving.

Gerv

Comment 10

[Iñigo]

Yes, somehow. Camerfirma was doing domain validations and other validations stuff for EV certs but not for OV certs and as per your suggestion they are starting doing it next week, if not earlier. They are going to do mostly everything, contacting customers, etc. We´re providing just the info for they begin contacting.