Closed Bug 1672409 Opened 4 years ago Closed 3 years ago

Camerfirma: suspicious certificate for com.com

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: agwa-bugs, Assigned: ana.lopes)

Details

(Whiteboard: [ca-compliance] [ov-misissuance])

Attachments

(2 files)

dig.png
164.06 KB, image/png
Details
modulus.png
21.49 KB, image/png
Details

On February 14, 2020, Camerfirma issued the following certificate:

https://crt.sh/?sha256=39D95613D64DFDCA58E39CA12081759771F63C03C1BE10F49A0D725629D0103B

containing the following DNS SANs:

expired.ovcf2019.ca.intesasanpaolo.com.com
expired.ovcf2019.ca.intesasanpaolo.com

It was revoked on February 19.

Considering the suspicious nature of these SANs, coupled with recurring compliance issues with Camerfirma, I would ask Camerfirma to provide information about how this certificate was validated.

Assignee: bwilson → ana.lopes
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance]

Hello Andrew,

thank you very much.

We will give you more information as soon as posible.

Kind regards

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

We were aware of the problem on October 22nd because of the bug 1672423 opened by Andrew Ayer on October 21st.
A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

  1. After reviewing the information and verifying that that certificate was issued with an error in its domain name, we started to investigate the case. (October 22nd)

We contacted Intesa San Paolo to know why it was possible to issue that certificate (October 22nd)
We confirm the measures that have been applied by Intesa San Paolo (October 23rd)

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem

The domain expired.ovcf2019.ca.intesasanpaolo.com is the test domain for the expired subscriber certificate to satisfy the requirement "2.2 Publication of information" of the CabForum BRs:

The CA SHALL host test Web pages that allow Application Software Suppliers to test their software with Subscriber Certificates that chain up to each publicly trusted Root Certificate. At a minimum, the CA SHALL host separate Web pages using Subscriber Certificates that are (i) valid, (ii) revoked, and (iii) expired.

The misissued certificate was issued only for the purpose of satisfying the requirement and was never used.

The domain “expired.ovcf2019.ca.intesasanpaolo.com.com” was incorrectly and forcibly validated due to human error. The error was soon detected when the certificate was then audited.

The CA has stopped issuing certificates with the problem. It was a single case due to human error in issuing an internal certificate for service purposes and the wrong certificate was revoked within the next 5 days.

  1. A summary of the problematic certificates.

https://crt.sh/?q=2458735168

CA: Intesa Sanpaolo Organization Validation 2019 CA

Certificate serial number: 3b:b9:5c:bd:76:b0:42:93:00:e8:ad:37:43:8e:45:ff:51:ef:f0:0c

  1. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

https://crt.sh/?q=2458735168

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

It was a single case due to human error in issuing an internal certificate for service purpose and the error was detected shortly after issuance.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

The following actions have already been put in place:

The wrong certificate has been already revoked.
Greater awareness of Validation Specialists in the verification of operations in order to reduce human errors and to immediately revoke any incorrect certificates.

Also, Camerfirma is planning with the client the deadline to apply the automatic verification of the domain in the CSR through DNS lookup with Intesa San Paolo and will give more details about the progress.

It's worth noting that Comment #2 fails to substantively address the concerns, or demonstrate an awareness that would provide assurance that this issue won't repeat.

In particular, no explanation is provided, for example, about why this incident was not promptly reported to Mozilla. Either it stands to reason that Camerfirma was unaware of it, until reported, in which case, there's clearly lax oversight, or Camerfirma intentionally chose not to report it.

As incidents like Bug 1672423 or Bug 1575530 show, there's clearly a failure here to provide timely notification of reports.

I'd like to encourage Camerfirma to carefully reconsider https://wiki.mozilla.org/CA/Responding_To_An_Incident and consider adding an updated reply, which provides a meaningful timely, analyzes the factors (e.g. that prevented detection and/or reporting, in addition to the underlying issue), and does not attempt to dismiss the issue with "greater awareness", as communicated in the last paragraph of https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report

Flags: needinfo?(ana.lopes)

(In reply to Eusebio Herrera from comment #2)

The domain expired.ovcf2019.ca.intesasanpaolo.com is the test domain for the expired subscriber certificate to satisfy the requirement "2.2 Publication of information" of the CabForum BRs:

The CA SHALL host test Web pages that allow Application Software Suppliers to test their software with Subscriber Certificates that chain up to each publicly trusted Root Certificate. At a minimum, the CA SHALL host separate Web pages using Subscriber Certificates that are (i) valid, (ii) revoked, and (iii) expired.

The misissued certificate was issued only for the purpose of satisfying the requirement and was never used.

The domain “expired.ovcf2019.ca.intesasanpaolo.com.com” was incorrectly and forcibly validated due to human error. The error was soon detected when the certificate was then audited.

How would a certificate valid until February 2022 ever help as a test certificate for an expired certificate?

When did you reach out to the owner of the com.com domain? Which potential security threats did your analysis identify for them? What did you do to help them mitigate these threats?

The ICA in question (https://crt.sh/?caid=153236) currently has only (100 %) issued test, expired, and misissued certificates. You could revoke the whole ICA without any disadvantages for the Web PKI ecosystem. This revocation would surely reach more clients than only revoking a leaf certificate and, thus, be immensely beneficial to the Web PKI ecosystem. Which disadvantages do you see for the Web PKI ecosystem in not revoking the ICA?

When the technical certificates of "Intesa Sanpaolo Organization Validation 2019 CA" were issued there was not a dedicated enrolment procedure for the issuance of the three technical certificates expired, valid, revoked (really only the expired one needed such a dedicate procedure in order to force its expiration afte some hours from the issuance).

Therefore the expired technical certificate was issued by using a not completely automated procedure causing the human errors involving a double .com at the end of the FQDN and the wrong two years duration.

Afterwards a procedure dedicated to the issuance of the three technical certificates has been implemented and is now available. This procedure implements all the controls already implemented in the normal enrolment procedure.

On the other hand, we want to emphasize that this certificate wasn’t one of the included in the percentage of audited certificates and there is no any lint tool that could detect the problem “.com.com” as an issue nowadays.

Appart from that, we also want to highlight that this problem will appear in the next audit report as needed for all the bugs

Update: The automatic verification of the domains in the CSR is already established

Flags: needinfo?(ana.lopes)

(In reply to paul.leo.steinberg from comment #4)

When did you reach out to the owner of the com.com domain? Which potential security threats did your analysis identify for them? What did you do to help them mitigate these threats?

Would Camerfirma care to answer these question? They would show how Camerfirma is worried not only about their own customers but about the WebPKI as a whole.

(In reply to paul.leo.steinberg from comment #7)

When did you reach out to the owner of the com.com domain?

The domain verification was performed manually, and nobody contacted the owner of the wrong domain, they checked the data with the correct domain owner and the problem with the data was that the operator introduced a typo.

Which potential security threats did your analysis identify for them?

The potential security threat in this case is the possibility to use the certificate for a different domain, but in this case, there was not possibility to do that because the certificate was not given to the incorrect person and there were not any possibilities for the certificate to be used for that incorrect subdomain, because it does not exist.

Besides, we want to add that this certificate was never passed to production.

What did you do to help them mitigate these threats?

As we explained in the response for the previous question, we did not consider any security threats to mitigate because it was not possible for the threats to materialize.

(In reply to Ana Lopes from comment #8)

[...] not any possibilities for the certificate to be used for that incorrect subdomain, because it does not exist.

As can be seen, e.g., on https://dns.google/query?name=expired.ovcf2019.ca.intesasanpaolo.com.com , com.com's authoritative name server has a very different opinion on this. How did you come to the conclusion that this subdomain "does not exist"? How can you ever come to such a conclusion, given a potential split-horizon DNS and given the fact that com.com does not protected by DNSSEC (https://dnssec-analyzer.verisignlabs.com/expired.ovcf2019.ca.intesasanpaolo.com.com), which could prove the non-existence of a subdomain?

Besides, we want to add that this certificate was never passed to production.

How can you guarantee this? Was its key in a HSM, which has since been destroyed under audit? Were was the certificate itself saved? Are there any backups?

(I also want to point out, as it so remarkably, that an active MITM attack that could also change a victim's resolving name server's response to provide responses for non-existing subdomains is not part of Camerfirma's threat model. This is so remarkably because active MITM attacks are the only reason to have (DV-)certificates in the first place (with DH, passive-only MITM attacks pose no threat).)

The .com.com domain has characteristics that make a type A register exists with any string placed before of ”.com.com”.
In https://com.com/legal there is a statement about it:

"User-Generated Third-Level Domain Names

Com.com allows users to conduct searches by visiting sub-domains of Com.com. For example, if you visited "example.com.com" you might reach a page with information about "example." Some third-party affiliates also forward to Com.com. If you don't know how you ended up here, it may have been because (1) you or your internet browser appended an extra “.com” in the the address bar (2) a link that you followed contained an extra “.com” at the end or (3) a third-party affiliate forwarded you here. Reference to any organization, specific service or trade mark does not constitute or imply its association, endorsement or recommendation. If you have any questions or concerns, please e-mail legal@com.com."

For example, if you place the string “this_is_a_random_string_f8b18b56bd521ce36df3e6.intesasanpaolo.com.com”
you will obtain a response:
https://dnssec-analyzer.verisignlabs.com/this_is_a_random_string_f8b18b56bd521ce36df3e6.intesasanpaolo.com.com
or:
https://dns.google/query?name=this_is_a_random_string_f8b18b56bd521ce36df3e6.intesasanpaolo.com.com&rr_type=A&ecs=

Attached image dig.png —

Dig command result

(In reply to Eusebio Herrera from comment #11)

The .com.com domain has characteristics that make a type A register exists with any string placed before of ”.com.com”.
In https://com.com/legal there is a statement about it:

Thank you for showing that all of these subdomains do not only exist but are even intentionally used by ".com.com"'s owner (though, "register" is a somewhat strange term for "resource record" and it is worrisome for CAs to lack knowledge of correct DNS terminology).

I am sorry, but whatever this comment ought to prove, it is probably the most ridiculous comment by Camerfirma yet. Similar to https://bugzilla.mozilla.org/show_bug.cgi?id=1672423#c5, I sadly come to the conclusion that there is no point in further discussion in this bug.

As a final point, if you were to suggest that the subdomain in the certificate is not in active use (whatever that term means and contradicting the fact that you have just shown that it, in fact, is in active use), please see https://groups.google.com/g/mozilla.dev.security.policy/c/dSeD3dgnpzk/m/BOVUs0inAAAJ (by Mozilla Root Store Policy, 2.1, CAs MUST follow discussion on mozilla.dev.security.policy) for possible attacks with this certificate: it can be used to, e.g., in some cases, read cookies from and, in all cases, set cookies on the com.com domain. This can enable further attacks like, even in the case of only being able to set cookies, e.g., session fixation attacks (https://en.wikipedia.org/wiki/Session_fixation).

It though takes some knowledge of the web platform to analyze such possible attacks, which makes it even more worrisome when a CA claims trustfulness as TLS-certificate issuing CA by mentioning that they issue "more than 200.000 SMIME [sic] certificates per year" (https://groups.google.com/g/mozilla.dev.security.policy/c/dSeD3dgnpzk/m/NutM8TB1DQAJ). On the contrary, specific knowledge about the WebPKI and its challenges is needed and TLS certificates should be issued form a single-purpose root certificate (preventing issues like https://bugzilla.mozilla.org/show_bug.cgi?id=1685557 to occur).

(In reply to comment #13)

Please bear in mind that the certificate was delivered to the right person, who belongs to Intesa Sanpaolo IT Department. Furthermore, this certificate Keys weren’t generated in an HSM: this means that – as far as we know – an audit report for this process is not mandatory.
Please, let us give you more details:

  1. InfoCert team responsible for CA operations and management asked – in order to comply with the requirements about disclosing a subCA – to the relevant department to generate keys and a CSR, in order to issue:
  • Valid certificate
  • Revoked certificate
  • Expired certificate

  1. This department staff made a typo in generating the CSR file for the expired certificate.
    The CSR was generated to “expired.ovcf2019.ca.intesasanpaolo.com.com” instead of “expired.ovcf2019.ca.intesasanpaolo.com”. These “expired” certificates were issued following a manual procedure: at date, this procedure has been automatized to avoid similar errors.

  2. The first certificate ( https://crt.sh/?id=2458735168 ) was issued:

  • Not Before: Feb 14 09:27:13 2020 GMT

  • Not After: Feb 14 00:00:00 2022 GMT

    This certificate had two issues:

  • Wrong URL

  • Issued with a lifespan that was not what was needed. In fact, it had 2 years lifespan: in February 2020 this approach was still allowed, as issuing a certificate with a lifespan larger than 398 days was compliant with Mozilla Root Store Policy. So, this feature was correct, even if not useful for our purpose. Nevertheless, this certificate was sent to the right person, using an internal messaging application.

  1. Soon after, it was discovered that this first certificate was issued to the wrong domain. In order to solve the issue, new keys and a new CSR were generated.

  2. A second certificate was issued ( https://crt.sh/?id=2459080243 ):

  • Not Before: Feb 14 11:15:16 2020 GMT
  • Not After: Feb 14 00:00:00 2022 GMT
  1. Once it was discovered that the certificate had the wrong lifespan, a third certificate ( https://crt.sh/?id=2459453871 ) was issued:
  • Not Before: Feb 14 13:12:09 2020 GMT
  • Not After: Feb 15 00:00:00 2020 GMT

Comments:

  1. The first and the second certificates were revoked on 2020-02-19 09:22:47 UTC and on 2020-02-19 09:33:17 UTC
  2. Through openssl modulus function, we can prove that different keys were used from the wrong first certificate and the second and third ones (we use the sha256 function just in order to make it more readable):
  • openssl x509 -noout -modulus -in 1_2458735168.crt | openssl sha256
  • openssl x509 -noout -modulus -in 2_2459080243.crt | openssl sha256
  • openssl x509 -noout -modulus -in 3_2459453871.crt | openssl sha256
    (See the attached image: modulus.png )
  1. In order to avoid these issues in the future, the procedure of issuing these sets of certificates (valid, revoked and expired) has been automatized.
Attached image modulus.png —

Modulus

We do not have more updates for this bug.

We do not have more updates for this bug.

We do not have more updates for this bug.

We do not have more updates for this bug.

We do not have more updates for this bug.

We do not have more updates for this bug.

Flags: needinfo?(bwilson)

We do not have more updates for this bug.

Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [ov-misissuance]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: