Closed Bug 1390978 Opened 7 years ago Closed 6 years ago

Certinomis: Non-BR-Compliant Certificate Issuance

Categories

(CA Program :: CA Certificate Compliance, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: kwilson, Assigned: franck.leroy, NeedInfo)

References

Details

(Whiteboard: [ca-compliance] [ov-misissuance])

The following problems have been found in certificates issued by your CA, and reported in the mozilla.dev.security.policy forum. Direct links to those discussions are provided for your convenience.

To continue inclusion of your CA’s root certificates in Mozilla’s Root Store, you must respond in this bug to provide the following information:
1) How your CA first became aware of the problems listed below (e.g. via a Problem Report, via the discussion in mozilla.dev.security.policy, or via this Bugzilla Bug), and the date.
2) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates with the problems listed below.
3) Complete list of certificates that your CA finds with each of the listed issues during the remediation process. The recommended way to handle this is to ensure each certificate is logged to CT and then attach a CSV file/spreadsheet of the fingerprints or crt.sh IDs, with one list per distinct problem.
4) Summary of the problematic certificates. For each problem listed below: number of certs, date first and last certs with that problem were issued.
5) Explanation about how and why the mistakes were made, and not caught and fixed earlier.
6) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
7) Regular updates to confirm when those steps have been completed.

Note Section 4.9.1.1 of the CA/Browser Forum’s Baseline Requirements, which states:
“The CA SHALL revoke a Certificate within 24 hours if one or more of the following occurs: …
9. The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement; 
10. The CA determines that any of the information appearing in the Certificate is inaccurate or misleading; …
14. Revocation is required by the CA’s Certificate Policy and/or Certification Practice Statement; or 
15. The technical content or format of the Certificate presents an unacceptable risk to Application Software Suppliers or Relying Parties (e.g. the CA/Browser Forum might determine that a deprecated cryptographic/signature algorithm or key size presents an unacceptable risk and that such Certificates should be revoked and replaced by CAs within a given period of time).

However, it is not our intent to introduce additional problems by forcing the immediate revocation of certificates that are not BR compliant when they do not pose an urgent security concern. Therefore, we request that your CA perform careful analysis of the situation. If there is justification to not revoke the problematic certificates, then explain those reasons and provide a timeline for when the bulks of the certificates will expire or be revoked/replaced. 

We expect that your forthcoming audit statements will indicate the findings of these problems. If your CA will not be revoking the certificates within 24 hours in accordance with the BRs, then that will also need to be listed as a finding in your CA’s BR audit statement.

We expect that your CA will work with your auditor (and supervisory body, as appropriate) and the Root Store(s) that your CA participates in to ensure your analysis of the risk and plan of remediation is acceptable. If your CA will not be revoking the problematic certificates as required by the BRs, then we recommend that you also contact the other root programs that your CA participates in to acknowledge this non-compliance and discuss what expectations their Root Programs have with respect to these certificates.


The problems reported for your CA in the mozilla.dev.security.policy forum are as follows:

** Failure to respond within 24 hours after Problem Report submitted
https://groups.google.com/d/msg/mozilla.dev.security.policy/PrsDfS8AMEk/w2AMK81jAQAJ
The problems were reported via your CA’s Problem Reporting Mechanism as listed here:
https://ccadb-public.secure.force.com/mozilla/CAInformationReport
Therefore, if this is the first time you have received notice of the problem(s) listed below, please review and fix your CA’s Problem Reporting Mechanism to ensure that it will work the next time someone reports a problem like this.

** Invalid dnsNames (e.g. invalid characters, internal names, and wildcards in the wrong position)
https://groups.google.com/d/msg/mozilla.dev.security.policy/CfyeeybBz9c/lmmUT4x2CAAJ
https://groups.google.com/d/msg/mozilla.dev.security.policy/D0poUHqiYMw/Pf5p0kB7CAAJ

** Invalidly long serial numbers (Serial Number > 20 Octets)
https://groups.google.com/d/msg/mozilla.dev.security.policy/b33_4CyJbWI/74sItqcvBgAJ
Hello

This bug mix 3 problems:
	1/ some invalid certificates issued by Certinomis,
	2/ some invalid certificates issued by StartCom,
	3/ some discussion about the serialNumber DER encoding.

Issue 1:
	
Certinomis has been aware of 4 invalid certificates containing non conformant DNSnames by Jonathan Rudenberg via email on Sunday 2017-08-13.

This bug is known and has been corrected in 2014, but a few set of certificates has not been revoked.

Certificates with extra RFC822name in SANs (3):
https://crt.sh/?id=4925505&opt=cablint
https://crt.sh/?id=42495728&opt=cablint
https://crt.sh/?id=19404500&opt=cablint
Certificate with an extra space in a DNSname (1):
https://crt.sh/?id=7252922&opt=cablint 

A miss configuration between RA and CA (september 2014) made that some TLS certificates were created with the contact email address added as a RFC822name in the SAN list. 
An non-repeatable issuance bug made a certificate with an extra space in the DNSname (March 2015). The certificate has been reissued for the customer, but the invalid certificate has not been revoked.

3 certificates are not used and have been revoked immediately.
1 is used for a TLS web server (expires within 16 days), it has been revoked and we have notified the customer to renew its certificate.

Issue 2:

Another bug is open on the subject: https://bugzilla.mozilla.org/show_bug.cgi?id=1386894 

Issue 3:

Discussion within mozilla.dev.security.policy  shows that the RFC5280 requirement on serialNumber is not crystal clear.
It is not clear that 20 bytes limit shall be understand as:
	+ Conforming CAs MUST NOT use serialNumber values longer than 20 octets when encoded into DER including the extra 00 octet for positive padding when needed.
or
	+ Conforming CAs MUST NOT use serialNumber values longer than 20 octets. Users MUST be aware that an extra 00 octet for positive padding may be encoded.
		ie: TAG, LENGTH, [00], significant integer value
		eg: 13, 21, 00, DE 3C 5F FE D6 E9 59 62 C9 E2 F3 07 41 5B 8B 43 33 61 C6 E5

As there is no clear position on the subject, no security impact and no interoperability issue. These certificates will not be revoked immediately.

But limiting the encoded value to 20 octets including the padding implies to technically limit the significant integer value to 19 octets, because the piece of software that manages serialNumber is independent of DER encoding rules.
SerialNumber are unique then could be predictable. So it is necessary to hide the real serialNumber value. Commons implementations uses encryption or hashing for that purpose. But limiting to 19 octets forbids the use of SHA1 in order to masquerade the real serialNumber value. 


Best regards
Franck Leroy
Thanks for responding. I think it's still necessary to provide additional detail.

That is, we can look at the problem from two dimensions: The problem itself, and the systemic issues that allowed the problem to manifest. Your description focuses on the resolution of the problem, but doesn't indicate any systemic changes have been made. As a consequence, it does not help the community feel that your CA has taken steps to reduce the risk of future violations (of any requirement) in the future. That is, one dimension is "Did you fix the bug", but another dimension is "How was the bug introduced, why was it not detected, and what steps are you taking to prevent future bugs"

To understand how you might approach this problem, consider https://groups.google.com/d/msg/mozilla.dev.security.policy/vl5eq0PoJxY/W1D4oZ__BwAJ and how it provided a timeline of events, the steps that were already in place (and with substantial detail), where there were controls missing or mistakes made, details about the steps being taken (e.g. "We fixed the bug" is not sufficient detail to understand), and a holistic, systemic awareness of how the CA is managed and the opportunities for errors.


For this specific case:
---
- What makes the bug in https://crt.sh/?id=7252922&opt=cablint non-repeatable? What controls existed to prevent this, and what controls are now in place to prevent this?
  - In particular, the presence of the space before "ch-stdenis.fr" implies a manually-entered domain name. Is that correct? If so, what steps have you taken to ensure no domain names can be manually entered, or that any domain names manually entered have been appropriately verified?
- Regarding Issue 3, multiple implementations have properly understood that, by placing a limit of 20 _octets_, this refers to the process of DER-encoding an integer (as that is the only way to turn an ASN.1 integer into an octet sequence). As this includes any padding, to ensure it's a positive number, this means that it always includes the padding.
- Your explanation about why limiting the encoded value to 20 octets (including padding) being a risk does not make sense. The only requirements are to ensure sufficient entropy in the serial. Applying a SHA-1 hash function is _not_ sufficient entropy in the serial, and would violate the Baseline Requirements' requirement to include at least 64-bits of entropy. As such, you are already effectively limited to 15 bytes, or 127 bits, for internal serial number. Any form of "hiding" the serial number (by using hashing, not encryption) is a violation of the Baseline Requirements' entropy requirement.

As such, please describe what steps you are taking to ensure all serial numbers are limited to 20 encoded octets. For the community benefit, it may further help to understand what software you are using that issues such certificates, so that appropriate communication can be made. This helps the broader community by understanding if there is a common, shared implementation flaw that may represent risk to Mozilla users.

If this is an in-house developed software, please describe what steps you have taken and are taking to ensure such software complies with all of the requirements of RFC 5280 and the Baseline Requirements, so that the community can be confident that future issued certificates will comply.
To produce a certificate there are some templates on the CA server that describes where each RA data shall be in the certificate.
For SAN it looks like:
<?xml version="1.0" encoding="UTF-8"?>
<extension xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="schema.xsd">
  <subjectAltName>
    <extnValue>
      <GeneralName>
        <dNSName>%form.commonName%</dNSName>
      </GeneralName>
      <GeneralName>
        <dNSName>%form.fqdn1%</dNSName>
      </GeneralName>
      <GeneralName>
        <dNSName>%form.fqdn2%</dNSName>
      </GeneralName>
	  ...
      <GeneralName>
        <dNSName>%form.fqdn100%</dNSName>
      </GeneralName>
    </extnValue>
  </subjectAltName>
</extension>

There was a bad configuration on 2 templates, where there was a leading:

....
      <GeneralName>
        <rfc822Name>%form.emailAddress%</rfc822Name>
      </GeneralName>
    </extnValue>
  </subjectAltName>
</extension>

So when the RA sends no email (which is the common case for TLS certs) even with the bad configuration there is no email in SANs.
But if the RA sends an email value, then it appears in the cert.

The issue was quickly detected in 2014, and the templates have been corrected.
It is no more possible to add an emailAddress within a SAN list of a TLS cert.

========================

Regarding the extra space I checked in the RA database and the FQDN value is "extranet.ch-stdenis.fr" with no space.
As every FQDN is validated with a regex, it is not possible to submit an invalid format.
I checked in the log between the RA and the CA and the data exchanged is "extranet.ch-stdenis.fr" with no space.
I also checked in the CA database, and the value is also "extranet.ch-stdenis.fr" with no space:
Nom Prénom	:	messagerie.ch-stdenis.fr
Direction	:	93205
Identifiant	:	0002 26930101600011
Raison sociale	:	CENTRE HOSPITALIER GENERAL DE ST DENIS
Adresse courriel	:	hsd-info@ch-stdenis.fr
Titre	:	SAINT DENIS
Localité	:	SAINT DENIS
Département	:	93205
Pays	:	FR
Nom serveur	:	autodiscover.ch-stdenis.fr
Nom serveur	:	courrier.ch-stdenis.fr
Nom serveur	:	extranet.ch-stdenis.fr
Nom serveur	:	plugin.ch-stdenis.fr
Domaine	:	Certinomis

I downloaded the certificate from the CA database, and used guiDumpASN-ng to parse the cert, and there is an extra space.
I have today no explanation for this very certificate (1 over 100K issued), a bug within the PKI software apparently.
Since 2015 we have made many PKI software upgrades. We have no more certs with an extra space since 2015. It appears that the PKI vendor has corrected with bug.

When the certificate has been disseminated to the customer (2015-04-19), the customer came back this a 'none working' cert issue.
The customer service has reissued the certificate with no problem (2015-04-21).
The customer service should had revoked this cert as they usually do, but they didn't.

We often make some training to customer service people to review the rules to avoid some mistakes, but zero human error is very challenging.

========================

We are not software editor, I don’t know exactly how the serialNumber value is computed on our system.
I can imagine that there is a sequence in order to ensure the uniqueness. Then some entropy is added. In order to hide how the serialNumber is constructed, there is a SHA1 or an encryption function that is used, producing 20 octets.
Then when encoded an extra 00 may be added by the DER encoder.

If the community think this is a problematic practice, then we have to setup a timescale with a dead line to change it.
Because this would requires to ask to the PKI vendor to change the serialNumber computation process taking into account that the new computation process shall not generate already issued serialNumbers.
Then time is required to test the PKI software upgrade, before putting the new version in production environment.
We cannot simply revoke the certs, and produce new ones, it is really more complicated.

So my question is what is the security impact?
In a java software (sorry this is the only language I know) when I parse the cert and ask the serialNumber I retrieve a BigInteger:

BigInteger serial = cert.getSerialNumber();

If I want to generate an OCSP request or parse a CRL to find this serialNumber, I have no problem as the integer value has been decoded by the DER decoder and the BigInteger value is correct.

X509CRLEntry entry = crl.getRevokedCertificate(serial); // <- BigInteger value (not the binary encoded value with a potential heading 00)

So at the application level (upper DER encoding/decoding) there is no impact how integers are encoded.

The only issue I can imagine is if an application doesn’t decode the bytes and do some BIT STRING comparison...
But as the ASN.1 clearly states that the serialNumber is an integer (not bit string), conforming software shall encode and decode the bytes into integers.

So I do not see security impact (unless with some non conforming softwares) that would required an immediate PKI upgrade.

Best regards
Franck Leroy
(In reply to Franck Leroy from comment #3)
> Regarding the extra space I checked in the RA database and the FQDN value is
> "extranet.ch-stdenis.fr" with no space.
> As every FQDN is validated with a regex, it is not possible to submit an
> invalid format.
> I checked in the log between the RA and the CA and the data exchanged is
> "extranet.ch-stdenis.fr" with no space.
> I also checked in the CA database, and the value is also
> "extranet.ch-stdenis.fr" with no space:
> Nom Prénom	:	messagerie.ch-stdenis.fr
> Direction	:	93205
> Identifiant	:	0002 26930101600011
> Raison sociale	:	CENTRE HOSPITALIER GENERAL DE ST DENIS
> Adresse courriel	:	hsd-info@ch-stdenis.fr
> Titre	:	SAINT DENIS
> Localité	:	SAINT DENIS
> Département	:	93205
> Pays	:	FR
> Nom serveur	:	autodiscover.ch-stdenis.fr
> Nom serveur	:	courrier.ch-stdenis.fr
> Nom serveur	:	extranet.ch-stdenis.fr
> Nom serveur	:	plugin.ch-stdenis.fr
> Domaine	:	Certinomis
> 
> I downloaded the certificate from the CA database, and used guiDumpASN-ng to
> parse the cert, and there is an extra space.
> I have today no explanation for this very certificate (1 over 100K issued),
> a bug within the PKI software apparently.
> Since 2015 we have made many PKI software upgrades. We have no more certs
> with an extra space since 2015. It appears that the PKI vendor has corrected
> with bug.
> 
> When the certificate has been disseminated to the customer (2015-04-19), the
> customer came back this a 'none working' cert issue.
> The customer service has reissued the certificate with no problem
> (2015-04-21).
> The customer service should had revoked this cert as they usually do, but
> they didn't.
> 
> We often make some training to customer service people to review the rules
> to avoid some mistakes, but zero human error is very challenging.

Thanks for responding, and this is a much better (and much more detailed) response that helps the community assess impact and understanding :)

That's concerning about the question of the extra space, and the uncertainty about where or how it was introduced. While it's reassuring that multiple software upgrades have been made, and that this certificate stands out, it's also somewhat concerning regarding bugs. Have you considered instituting an additional review process, such as certlint on the tbsCertificate, to proactively detect and alert on the potential of future software bugs of this nature?

I agree that zero human error is often an unrealistic goal. That's why the push for additional details, because it helps show what systems are in place to catch, detect, or mitigate human error. By looking at the issue holistically - humans to catch and detect technical errors, technical systems to catch and detect human errors - we can build and ensure the necessary robustness.

Understanding the systems you have in place, where they may have failed, or what changes you've made or will make helps provide a picture about how your CA addresses these interrelationships.

> If the community think this is a problematic practice, then we have to setup
> a timescale with a dead line to change it.

Such certificates fail to work in various software (including Mozilla NSS) in various forms. For example, it can interfere with Go's ability to detect revocations, or NSS's ability to parse and trust.

So there is a dimension that is security, and a dimension that is interoperability. Both of these are important concerns of Mozilla's Root CA policy, and so taking steps to resolve this in a timely fashion is important. Further, this requirement (to ensure non-negative integers) was introduced in the subsequent RFCs (3280 and 5280) related to such concerns about interoperability and security, as they require positive numbers, so it's fair to say that there is indeed a concern here.

> Because this would requires to ask to the PKI vendor to change the
> serialNumber computation process taking into account that the new
> computation process shall not generate already issued serialNumbers.
> Then time is required to test the PKI software upgrade, before putting the
> new version in production environment.
> We cannot simply revoke the certs, and produce new ones, it is really more
> complicated.

Certainly; this is why it's important to file the bug and discuss. As you note, serial numbers being compliant require systemic fixes - revocation is not sufficient - and it's good to understand the steps (and the timeline) as to when and how those systemic fixes should be made. This is an important issue to correct, and in a timely fashion, to understanding the steps you can take to ensure such compliance is very important. I would encourage you to discuss this with your vendor.

> So my question is what is the security impact?
> In a java software (sorry this is the only language I know) when I parse the
> cert and ask the serialNumber I retrieve a BigInteger:
> 
> BigInteger serial = cert.getSerialNumber();
> 
> If I want to generate an OCSP request or parse a CRL to find this
> serialNumber, I have no problem as the integer value has been decoded by the
> DER decoder and the BigInteger value is correct.
> 
> X509CRLEntry entry = crl.getRevokedCertificate(serial); // <- BigInteger
> value (not the binary encoded value with a potential heading 00)
> 
> So at the application level (upper DER encoding/decoding) there is no impact
> how integers are encoded.

This is not entirely accurate, although it is true for the Java case. That is, you're absolutely correct that when reading the sequence of bytes, both a BER (as Java's is) and DER decoder can properly interpret the sequence of octets, producing a negative integer (as specified in X.690's encoding rules). Similarly, one can use a DER encoder to take a negative integer and produce a series of bytes equivalent to the original encoded value - that is, they fully round-trip.

However, a negative integer is not valid according to RFC 5280, which states that these integers must be positive. It also limits such octets to 20 bytes. Thus, implementations that require or expect conformance to RFC 5280, either on the positive integer (for example, implemented via an unsigned type) or on the size limit (for example, only allocating 20 bytes to represent a certificate serial, as done by NSS and several other PKI implementations) will cause the certificate to either fail to parse, return improper results, or otherwise impact clients.

Similarly, you can imagine (and I have seen) systems which store their serials as positive integers in an internal database (e.g. not DER encoded). Rather than DER encode (and produce the leading 00 bytes, as necessary, to ensure a positive integer), which may result in 21 bytes, they simply copy the bytes into the serial. When parsing the certificate, this INTEGER field is then decoded to a negative integer. When this negative integer is looked up in systems (such as the revocation system), it's not found - because the actual integer is a positive, not negative, number.

Client software frequently has to work around this and other issues. For example, the lack of consistent encoding of serial numbers by CAs has resulted in workarounds in both Chrome and Firefox, to ensure their systems are properly able to detect and handle revocation information.

For these reasons, it is desired for the interoperability and security of your system to plan an upgrade. We understand this may take time, and new development, and that's why request 6 tries to understand a timeline and request 7 tries to understand regular updates on that progression, to minimize the systemic and holistic risk.
(In reply to Franck Leroy from comment #3)
> ....
>       <GeneralName>
>         <rfc822Name>%form.emailAddress%</rfc822Name>
>       </GeneralName>
>     </extnValue>
>   </subjectAltName>
> </extension>

Thanks for providing more detail.

This explanation does not match the three certificates, which each had one email address in a dnsName SAN. They did not have any rfc822Name SANs.

https://crt.sh/?id=4925505&opt=cablint
https://crt.sh/?id=42495728&opt=cablint
https://crt.sh/?id=19404500&opt=cablint
>Thanks for responding, and this is a much better (and much more detailed) response that helps the community assess impact and understanding :)
>That's concerning about the question of the extra space, and the uncertainty about where or how it was introduced. While it's reassuring that multiple software upgrades have been made, and that this certificate stands out, it's also somewhat concerning regarding bugs. Have you considered instituting an additional review process, such as certlint on the tbsCertificate, to proactively detect and alert on the potential of future software bugs of this nature?
>I agree that zero human error is often an unrealistic goal. That's why the push for additional details, because it helps show what systems are in place to catch, detect, or mitigate human error. By looking at the issue holistically - humans to catch and detect technical errors, technical systems to catch and detect human errors - we can build and ensure the necessary robustness.
>Understanding the systems you have in place, where they may have failed, or what changes you've made or will make helps provide a picture about how your CA addresses these interrelationships.

We have an internal tool that generate all type of certificates on the staging environment in order to validate any upgrade.
The SAN email was not detected because this scenario was not implemented; the unit test does not send any email for TLS certs, so no bug...
I can easily add this test case in order to detect any future bad TLS cert template.

>Such certificates fail to work in various software (including Mozilla NSS) in various forms. For example, it can interfere with Go's ability to detect revocations, or NSS's ability to parse and trust.
>So there is a dimension that is security, and a dimension that is interoperability. Both of these are important concerns of Mozilla's Root CA policy, and so taking steps to resolve this in a timely fashion is important. Further, this requirement (to ensure non-negative integers) was introduced in the subsequent RFCs (3280 and 5280) related to such concerns about interoperability and security, as they require positive numbers, so it's fair to say that there is indeed a concern here.

I will not discuss on this as it could last eternity.
My thought is that if we use a proper user implementation, then there is no issue for example:

	X509Certificate x509 = PKCS07.loadX509Certificate("D:\\long-positive-serial.der");
	BigInteger serial = x509.getSerialNumber();
	int length = serial.bitLength() /8;
	System.out.println("SerialNumber length:"+ length + " octets");
	String hexString = serial.toString(16).toUpperCase();
	System.out.println("SerialNumber value :"+ hexString);
	if (serial.compareTo(new BigInteger("0")) == 1 )
	{
	    System.out.println("SerialNumber is non null and positive");
	}
	
Outputs:
SerialNumber length:20 octets
SerialNumber value :DE3C5FFED6E95962C9E2F307415B8B433361C6E5
SerialNumber is non null and positive

So in this point of view everything is fine, and totally compliant with RFC5280.

>"fail to work in various software"
My regrets is that we blame CA for "bad practices" because of some bad user implementations in the market.

But again I will not fight for this, and if any upgrade is needed, I'd like to see a Mozilla communication about "problematic practices" and a delay to resolve it.
This is necessary in order to ask an upgrade to the editor and to argue that the product is not compliant with Mozilla expectations.

>I would encourage you to discuss this with your vendor.
Yeah with a Mozilla communication on the subject ;-)



Best regards (or regrets ? ;-))
Franck Leroy
>Thanks for providing more detail.
You're welcome.

>This explanation does not match the three certificates, which each had one email address in a dnsName SAN. They did not have any rfc822Name SANs.
>https://crt.sh/?id=4925505&opt=cablint
>https://crt.sh/?id=42495728&opt=cablint
>https://crt.sh/?id=19404500&opt=cablint

As you can see there is no Rfc822Name in SAN so the CA configuration is OK.
You are right, for this ones it was a bug in the RA software.
To understand the origin of the bug you have to know that we issue some eSeals certs (smime,tsu...), that could need an email address in SANs.
So the RA software has the capability in case of "server" certificates to send the serverEmail into a SAN form of the CA software.
But the result is that when a serverEmail is in the RA application for a TLS certs, it produces a DNSname with an email... 
So the RA software has quickly been patched (and then corrected) in order to only send serverEmail in SAN, not only in "server" workflows but also by checking that the cert profile is not a TLS certificates.

I have the RA java code ;-)
The very first patch was quick and dirty... but it works

	if (createHolder.getSanNames() != null)
	{
	    StringTokenizer sanST = new StringTokenizer(createHolder.getSanNames(), ",");
	    int index = 0;
	    while (sanST.hasMoreTokens())
	    {
		index++;

		String dnsName = sanST.nextToken();
		result += createXmlNvpDn("param_fqdn" + index, dnsName);
	    }
	    while (index < 98) // purge des anciens SAN résiduels
	    {
		index++;
		result += createXmlNvpDn("param_fqdn" + index, "");
	    }
	    // if FQDN then no serverEmail
	    createHolder.setServerEmail(null);
	}

Last line added to quickly correct: serverEmail was set to null in case of FQDN values...

On the next version, with is more accurate (totally new implementation...) : 

	if (this.profileName.contains("_SSL") || this.profileName.contains("_TLS") || this.profileName.contains("_WEB") || this.profileName.contains("_SRV"))
	{ // only DNSNames is TLS servers
	    san += "DNSNAME=" + commonName;
	    if (this.sanNames != null)
	    {
		Iterator sanST = this.sanNames.iterator();
		while (sanST.hasNext())
		{
		    String dnsName = (String) sanST.next();
		    san += ",DNSNAME=" + dnsName;
		}
	    }
	} else
	{
	    if (this.profileName.contains("_CAC") || this.profileName.contains("_SMIM"))
	    {
		if (this.serverEmail != null)
		{
		    san += "RFC822NAME=" + this.serverEmail;
		}
	    }
	    else if ((this.profileName.contains("_CLT") == false) && (this.profileName.contains("_TSU") == false))
	    {
		san += "RFC822NAME=" + this.email;
	    }
	    if (this.extraName != null)
	    {
		san += ",DIRECTORYNAME=" + this.extraName;
	    }
	}
serverEmail is only for non TLS cert profiles.

Best regards
Franck Leroy
(In reply to Franck Leroy from comment #6)
> I will not discuss on this as it could last eternity.
> My thought is that if we use a proper user implementation, then there is no
> issue for example:
> 
> 	X509Certificate x509 =
> PKCS07.loadX509Certificate("D:\\long-positive-serial.der");
> 	BigInteger serial = x509.getSerialNumber();
> 	int length = serial.bitLength() /8;
> 	System.out.println("SerialNumber length:"+ length + " octets");
> 	String hexString = serial.toString(16).toUpperCase();
> 	System.out.println("SerialNumber value :"+ hexString);
> 	if (serial.compareTo(new BigInteger("0")) == 1 )
> 	{
> 	    System.out.println("SerialNumber is non null and positive");
> 	}
> 	
> Outputs:
> SerialNumber length:20 octets
> SerialNumber value :DE3C5FFED6E95962C9E2F307415B8B433361C6E5
> SerialNumber is non null and positive
> 
> So in this point of view everything is fine, and totally compliant with
> RFC5280.

It is an incorrect point of view. RFC 5280 explicitly states octets. X.690 is precise in that it describes "Contents octets" - which include any necessary padding to ensure a positive number (pursuant with both Section 8.1.4 and Section 8.3 of X.690)

Your measure, via .bitLength / 8, is not a measure of content octets. It is a measure of the number of bits in the underlying big integer. This distinction is meaningful and important.

A certificate whose serial number content octets represent a length of 21 is NOT compliant with https://tools.ietf.org/html/rfc5280#section-4.1.2.2 and invalid.

> 
> >"fail to work in various software"
> My regrets is that we blame CA for "bad practices" because of some bad user
> implementations in the market.

My regret is that we trust CAs to properly implement the RFCs, and then blame implementations for not ignoring their issues. As 5280 states, "Conforming CAs MUST NOT use serialNumber values longer than 20 octets". When encoded, the serialNumber value is longer than 21 octets, and encoding is relevant in this case, because we are discussing "contents octets".

The ASN.1 encoding of such a 'long serial' clearly shows a length of 21 octets for the serialNumber value. It is misissued.

> But again I will not fight for this, and if any upgrade is needed, I'd like
> to see a Mozilla communication about "problematic practices" and a delay to
> resolve it.
> This is necessary in order to ask an upgrade to the editor and to argue that
> the product is not compliant with Mozilla expectations.

Please point your vendor to this bug, and please provide a timeline for remediation of this non-compliance.
(In reply to Franck Leroy from comment #7)
> As you can see there is no Rfc822Name in SAN so the CA configuration is OK.

I can't agree that the configuration is "OK", as it let a certificate with an invalid dnsName be issued.

> 	if (createHolder.getSanNames() != null)

Are you saying that getSanNames() also returned an email address?


> 	if (this.profileName.contains("_SSL") || this.profileName.contains("_TLS")
> || this.profileName.contains("_WEB") || this.profileName.contains("_SRV"))
> 	{ // only DNSNames is TLS servers
> 	    san += "DNSNAME=" + commonName;
> 	    if (this.sanNames != null)
> 	    {
> 		Iterator sanST = this.sanNames.iterator();
> 		while (sanST.hasNext())
> 		{
> 		    String dnsName = (String) sanST.next();
> 		    san += ",DNSNAME=" + dnsName;

This code raises a few questions:

- Are the commonName and sanNames fully sanitized (using the LDH rule at least) before this code is reached?
- Where do you verify that duplicate SANs are not included in a certificate?
(In reply to Franck Leroy from comment #3)
> To produce a certificate there are some templates on the CA server that
> describes where each RA data shall be in the certificate.
> For SAN it looks like:
> <?xml version="1.0" encoding="UTF-8"?>
> <extension xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:noNamespaceSchemaLocation="schema.xsd">
>   <subjectAltName>
>     <extnValue>
>       <GeneralName>
>         <dNSName>%form.commonName%</dNSName>
>       </GeneralName>
>       <GeneralName>
>         <dNSName>%form.fqdn1%</dNSName>
>       </GeneralName>
>       <GeneralName>
>         <dNSName>%form.fqdn2%</dNSName>
>       </GeneralName>
> 	  ...
>       <GeneralName>
>         <dNSName>%form.fqdn100%</dNSName>
>       </GeneralName>
>     </extnValue>
>   </subjectAltName>
> </extension>

Thank yuo for providing an excerpt of your actual CA templates. Could you provide some detail about whether these values are escaped? For example, what would happen if form.commonName contained the value "domain.com</dNSName></GeneralName><GeneralName><dNSName>google.com"? Does escaping (if it exists) properly handle escaping doctype/entity directives in addition to the specific XML node names in your sample template?
(In reply to Jonathan Rudenberg from comment #9)
>Are you saying that getSanNames() also returned an email address?

No in this version SanNames is a list of FQDN, but the code below may had an RFC822Name.
I dont have the buggy code of 2014/2015 that had generated the @ in a SAN, if was something like :

 result += createXmlNvpAtt("param_fqdn" + index, createHolder.getServerEmail());
 
>This code raises a few questions:
>- Are the commonName and sanNames fully sanitized (using the LDH rule at least) before this code is reached?
>- Where do you verify that duplicate SANs are not included in a certificate?

This part of code is just the part that takes the validated datas in the RA database in order to POST to the CA software.
Data are sanitized by the RA before validation and is also sanitized by the CA before storing in the database.

Franck
(In reply to Franck Leroy from comment #11)
> This part of code is just the part that takes the validated datas in the RA
> database in order to POST to the CA software.
> Data are sanitized by the RA before validation and is also sanitized by the
> CA before storing in the database.

What are the sanitization steps in that are applied by the RA software?
(In reply to Paul Kehrer from comment #10)
(In reply to Jonathan Rudenberg from comment #12)
> Thank yuo for providing an excerpt of your actual CA templates. Could you
> provide some detail about whether these values are escaped? For example,
> what would happen if form.commonName contained the value
> "domain.com</dNSName></GeneralName><GeneralName><dNSName>google.com"? Does
> escaping (if it exists) properly handle escaping doctype/entity directives
> in addition to the specific XML node names in your sample template?


1/ When a customer do a cert application, the RA forms validate that the data is correctly formatted according to the validation rules (with regex for instance).

2/ Before any communication from the RA to the CA, as required by CEN and ETSI standard, any application shall be validated by an Registration Officer.
   So the SANs are displayed and the Officer (a human person) who has to validate any entry.
   So such value (in the hypothesis someone could enter it) "domain.com</dNSName></GeneralName><GeneralName><dNSName>google.com" should raise some questions to the Officer when validating it...
   No certificates are automatically generated, any Certinomis certificate is validated by at least one Registration Officer.
   
3/ When the CA receives the data, the XML documents are validated by a piece of software called DataValidationService, that will reject any XML documents containing unexpected data/format.

Let's try it on staging CA directly on the CA API to bypass any RA validation: 

...
values.put("form.commonName", "domain.com</dNSName></GeneralName><GeneralName><dNSName>google.com");
...

CA logs:
21-08-2017 17:02:59	AEL_A2M_EASY	Enregistrement	Operation CREATE_HOLDER: Parameter values error commonName

client side logs:
com.certinomis.bo.exception.BoException: CreateHolder Error:Operation CREATE_HOLDER: Parameter values error commonName


Franck(In reply to Jonathan Rudenberg from comment #12)
> (In reply to Franck Leroy from comment #11)
> > This part of code is just the part that takes the validated datas in the RA
> > database in order to POST to the CA software.
> > Data are sanitized by the RA before validation and is also sanitized by the
> > CA before storing in the database.
> 
> What are the sanitization steps in that are applied by the RA software?

(In reply to Jonathan Rudenberg from comment #12)
> (In reply to Franck Leroy from comment #11)
> > This part of code is just the part that takes the validated datas in the RA
> > database in order to POST to the CA software.
> > Data are sanitized by the RA before validation and is also sanitized by the
> > CA before storing in the database.
> 
> What are the sanitization steps in that are applied by the RA software?
To make sure we've got clear documented answers, here's my current set of answers from Certinomis:

Issue #1 - Invalid DNS names
> 1) How your CA first became aware of the problems listed below (e.g. via a Problem Report, via the discussion in mozilla.dev.security.policy, or via this Bugzilla Bug), and the date.

Per Comment #1:
> > Certinomis has been aware of 4 invalid certificates containing non conformant DNSnames by Jonathan Rudenberg via email on Sunday 2017-08-13.

> 2) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates with the problems listed below.

Per Comment #1:
> > A miss configuration between RA and CA (september 2014) made that some TLS certificates were created with the contact email address added as a RFC822name in the SAN list. 
> > An non-repeatable issuance bug made a certificate with an extra space in the DNSname (March 2015). The certificate has been reissued for the customer, but the invalid certificate has not been revoked.

> 3) Complete list of certificates that your CA finds with each of the listed issues during the remediation process. The recommended way to handle this is to ensure each certificate is logged to CT and then attach a CSV file/spreadsheet of the fingerprints or crt.sh IDs, with one list per distinct problem.

It is unclear that this has been answered.

> 4) Summary of the problematic certificates. For each problem listed below: number of certs, date first and last certs with that problem were issued.

It is unclear that this has been answered.

> 5) Explanation about how and why the mistakes were made, and not caught and fixed earlier.

It is unclear that this has been answered.

> 6) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

It is unclear that this has been answered.

> 7) Regular updates to confirm when those steps have been completed.

It is unclear that this has been answered.

Issue #3 - Invalid serial numbers
> 1) How your CA first became aware of the problems listed below (e.g. via a Problem Report, via the discussion in mozilla.dev.security.policy, or via this Bugzilla Bug), and the date.

It is unclear that this has been answered.

> 2) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates with the problems listed below.

It is unclear that this has been answered.

> 3) Complete list of certificates that your CA finds with each of the listed issues during the remediation process. The recommended way to handle this is to ensure each certificate is logged to CT and then attach a CSV file/spreadsheet of the fingerprints or crt.sh IDs, with one list per distinct problem.

It is unclear that this has been answered.

> 4) Summary of the problematic certificates. For each problem listed below: number of certs, date first and last certs with that problem were issued.

It is unclear that this has been answered.

> 5) Explanation about how and why the mistakes were made, and not caught and fixed earlier.

It is unclear that this has been answered.

> 6) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

It is unclear that this has been answered.

> 7) Regular updates to confirm when those steps have been completed.

It is unclear that this has been answered.
Flags: needinfo?(franck.leroy)
(In reply to Ryan Sleevi from comment #14)
> Issue #1 - Invalid DNS names
> > 3) Complete list of certificates that your CA finds with each of the listed issues during the remediation process. The recommended way to handle this is to ensure each certificate is logged to CT and then attach a CSV file/spreadsheet of the fingerprints or crt.sh IDs, with one list per distinct problem.
 
The remediation process took place in 2014.
At this time there was no such requirements to list all certificates.
I have no such list available today.

> > 4) Summary of the problematic certificates. For each problem listed below: number of certs, date first and last certs with that problem were issued.
 
The SHA1 PKI is now offline and no more trusted by browsers.
Only 1 valid SHA2 certificate has been identified and revoked on 2017-08-17  08:37:12 UTC
https://crt.sh/?id=4925505&opt=cablint
 
> > 5) Explanation about how and why the mistakes were made, and not caught and fixed earlier.
 
The bug was not identified by the quality process as it appears when providing unexpected email value in the RA/CA protocol.
It has been caught and fixed in September 2014, but a few certificates were not revoked due to some human mistakes.
 
> > 6) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Unit tests have been added in order to test such case (email send by RA for TLS cert) in order to detect bad CA template configuration or bad RA/CA API.

> > 7) Regular updates to confirm when those steps have been completed.

The bug has been identified September 15th 2014.
It has been corrected in the API version 3.3.1.4 of September 23th 2014.
The patch has been applied September 26th 2014 but only on the SHA2 PKI, as the SHA1 PKI was end of life and now stopped.
But unfortunately some invalid certificates were not revoked by the customer service.


> Issue #3 - Invalid serial numbers
> > 1) How your CA first became aware of the problems listed below (e.g. via a Problem Report, via the discussion in mozilla.dev.security.policy, or via this Bugzilla Bug), and the date.
 
Certinomis has been aware of invalid certificates containing long serialNumbers by Jonathan Rudenberg via email on Sunday 2017-08-06.
 
> > 2) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates with the problems listed below.
 
Not stopped has it is a native feature of the PKI software. A request to correct it has to be done to the PKI vendor.
 
> > 3) Complete list of certificates that your CA finds with each of the listed issues during the remediation process. The recommended way to handle this is to ensure each certificate is logged to CT and then attach a CSV file/spreadsheet of the fingerprints or crt.sh IDs, with one list per distinct problem.
 
As it depends of the first bit of the serialNumber value, statistically 50% of issued cert are starting with a heading 0x00 byte.
 
> > 4) Summary of the problematic certificates. For each problem listed below: number of certs, date first and last certs with that problem were issued.
 
SerialNumbers computation by the PKI is 20 bytes long.
But when encoded into DER, a heading 0x00 byte may be added, then the DER encoded length increases to 21 bytes.
 
> > 5) Explanation about how and why the mistakes were made, and not caught and fixed earlier.
 
The RFC requirements are not crystal clear (may be obvious for native English speakers..?) and the French PKI vendor understood that length limitation is on the significant value not the encoded one.
 
> > 6) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
 
A patch has to be provided by the vendor.
Quality testing is required to check that there are no impact on previously issued certificates (no value collision for instance), no impact on CRL and OCSP.
Today we have to timeline from the vendor.
 
> > 7) Regular updates to confirm when those steps have been completed.

Not completed today

Regards
Franck Leroy
Flags: needinfo?(franck.leroy)
To confirm:

- Certinomis does not have a timeline for when it can expect to remedy the issue of overly long serial numbers (which will cause interoperability issues with Mozilla NSS-based products)?

I'll set Need-Info to Kathleen to find out how she would like to address this, and I'll leave the Need-Info for you in place so that you'll receive periodic reminders to update this bug once you have a timeline in place for remediation.
Flags: needinfo?(kwilson)
Flags: needinfo?(franck.leroy)
(In reply to Ryan Sleevi from comment #16)
> To confirm:
> 
> - Certinomis does not have a timeline for when it can expect to remedy the
> issue of overly long serial numbers (which will cause interoperability
> issues with Mozilla NSS-based products)?
> 
> I'll set Need-Info to Kathleen to find out how she would like to address
> this, and I'll leave the Need-Info for you in place so that you'll receive
> periodic reminders to update this bug once you have a timeline in place for
> remediation.

It would be best for end users if these certificates are revoked and replaced, but I don't necessarily see a security concern here.

Users running into sites with this type of certificate will get confusing errors. There's some context in Bug #1229014 and Bug #1139205.

So, from a Mozilla standpoint I'm going to say that this CA must fix their certificate issuance to prevent issuing such certs in the future. And the CA needs to provide a prompt timeline for implementing this change.

Also, I recommend that the CA revoke/replace the existing certs with this problem, but revoking/replacing these certs is not urgent.
Flags: needinfo?(kwilson)
Hello

I have a feedback from the PKI vendor that they are working on the issue, but not yet anytime line.

They need more time to evaluate the impact of this update to be able to give a timeline.

Regards

Franck
Flags: needinfo?(franck.leroy)
Franck: It is now >3 weeks since your last update. Do you have an updated status?

It is concerning that a CA would not be able to provide a timeline of implementing BR compliance for at least a month, let alone concerning if a CA was not able to implement compliance. This should ideally be a trivial change, and so it is important to understand why your PKI vendor is taking so long.

Alternatively, if you forgot to provide an update, it would be good to understand why, and what process changes are being made to ensure the community is kept informed in a timely fashion.
Flags: needinfo?(franck.leroy)
Hello

A pki vendor patch is ready.

The workaround is not trivial as the pki test the serial generated and if it starts with a '1' then the certificate is not signed and an error is returned.

So we have to implement a workaround in our RA software to do some 'retry'...

I hope to have the workaround implemented by the end of the month.

A stable solution is to change the pki vendor, will allready start to deploy a new PKI.
Will we swap to this new pki after the ETSI audit planned before the end of the year.


Franck.
Flags: needinfo?(franck.leroy)
Hello

The certificat issuance tests are finished (about 2000 tests).

So we plan to update our production system on October 25th.

If there is some error alerts after the patch, we would step back to the previous version.

I'll let you know if the patch is ok on production.

Regards
Franck Leroy
Hello

Our production system has been patched today.

I will check the serialNumber generated in the incoming days.

I have send an audit request to LSTI for our new PKI environnement, the related CA generated have been added in the CCADB.

Regards
Franck.
Franck: can you confirm that Certnomis is now issuing all certificates with serial numbers of the correct length?

Gerv
Flags: needinfo?(franck.leroy)
Hello

I can confirm that the production system is patched and produces "correct" lenght certificates.

I did not have done a full scan on the database to check issued certificates, only on a few days production, and I confirm that I did not found incorrect lenght certificates on this extract.

Best regards
Franck Leroy
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [ov-misissuance]
You need to log in before you can comment on or make changes to this bug.