Closed Bug 1620772 Opened 4 years ago Closed 4 years ago

SSL.com: Issued precertificate with Debian Weak Key

Categories

(CA Program :: CA Certificate Compliance, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: ryan.sleevi, Assigned: support)

Details

(Whiteboard: [ca-compliance] [dv-misissuance])

Matt Palmer reported on mozilla.dev.security.policy the following:

(Pre) Certificate https://crt.sh/?id=2531502044 has been issued with a known
weak key, specifically Debian weak key 2048/i386/rnd/pid17691. I believe
this issuance to be in contravention of SSL.com's CPS, version 1.8, section
6.1.1.2, which states "SSL.com shall reject a certificate request if the
request has a known weak Private Key".

This does appear to be a violation of the BRs and the stated policy. Please provide an incident report evaluating how this happened and the steps being taken to prevent it, and future errors of this type, from happening again.

Flags: needinfo?(support)

This is a preliminary report.

We received a Certificate Problem Report that claimed issuance of a Certificate that contains a weak Debian key. An investigation was triggered according to our CP/CPS.

Based on section 4.9.1.1 of our CP/CPS, we revoked the certificate within 24 hours and responded to both the Subscriber and the reporter per section 4.9.5 of our CP/CPS.

SSL.com is using a weak key blacklist that contains a prepopulated list of Debian weak keys. The fingerpint of the claimed Debian weak key was not included in our database.

The investigation is still in progress and we will be able to provide more information in the week ending March 13, 2020.

This is the first update to this Preliminary Report

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

SSL.com became aware of this issue via submission of a Certificate Resolution Request (see timeline).

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

Dates in ISO8601 format in UTC timezone.

  • 2020-03-07T03:18Z: SSL.com received a Certificate Resolution Request requesting the revocation of a certificate that was considered to be using a weak private key.
  • 2020-03-07T03:23Z: Revocation Request ticket was created.
  • 2020-03-07T03:44Z: Investigation for key compromise begun.
  • 2020-03-07T11:15Z: The weak key detection engine was checked and found to be in place and functioning.
  • 2020-03-07T13:13Z: Confirmed that the certificate mentioned in the Certificate Resolution Request uses a key not included in the weak keys blacklist that SSL.com maintains.
  • 2020-03-07T13:28Z: As the reporter demonstrated control of the private key associated with the issued certificate, the certificate was revoked and both the Subscriber and reporter were notified.
  • 2020-03-07T14:14Z: Full scan of certificate database started (in parallel with other research) to reveal if any blacklisted keys were used in issued certificates. No certificate was found using a key included in the blacklist.
  • 2020-03-07T19:20Z: The reported key was added to the weak keys blacklist.
  • 2020-03-07T20:47Z: Preliminary report added to bug.
  • 2020-03-10T13:28Z: CA Software vendor was contacted in search for an updated blacklist key information that contains more "flavors" of Debian weak keys.
  • 2020-03-10T17:08Z: This update.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

SSL.com rejects any certificate request if the request has a known weak Private Key. For the purpose of identifying whether a Private Key is weak, SSL.com uses a set of Debian weak keys that was provided by our CA software vendor as the basis for our blacklist. We identified an additional source of Debian weak keys (https://github.com/g0tmi1k/debian-ssh) that already included all 2048-bit keys identified by our CA software vendor but had additional 4096-bit keys. We incorporated the 4096 bit keys from that source into our blacklist as well. This information was disclosed on 2020-03-07 to the person that submitted the Certificate Problem Report.

Similarly, SSL.com also checks for keys that are affected by the ROCA vulnerability.

The above practices comply with our CP/CPS, version 1.8, section 6.1.1.2, which states:

"SSL.com shall reject a certificate request if the request has a known weak Private Key".

We consider that we have made reasonable efforts to fulfill both our CP/CPS and the Baseline Requirements Section 6.1.1.3:

"The CA SHALL reject a certificate request if the requested Public Key does not meet the requirements set forth in Sections 6.1.5 and 6.1.6 or if it has a known weak Private Key (such as a Debian weak key, see http://wiki.debian.org/SSLkeys)"

Our understanding is that there is no single or complete list with all known Debian weak keys, either one that is normative for use by the CAs included in the Mozilla Root Program, nor one specified in the Baseline Requirements.

As an example, our engineering team performed additional research and discovered that the "certwatch" database (used by crt.sh) contains a table of weak keys with a different set of keys compared to the set that SSL.com is using in its blacklist. It appears that at least some keys from the SSL.com weak key database are not included in "certwatch". This can be demonstrated by submitting the following CSR using https://secure.comodo.net/utilities/decodeCSR.html (assuming this page uses the Debian weak key table of the certwatch database used by crt.sh):

-----BEGIN CERTIFICATE REQUEST-----
MIICiDCCAXACAQAwRTELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUx
ITAfBgNVBAoMGEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDCCASAwDQYJKoZIhvcN
AQEBBQADggENADCCAQgCggEBANU56hG9cS5aDxgCCpncSVx7jWQInMI/yA40lrIb
IKp342EAh/B8kE/fpeEWJggQyv7rG6oKcOPs4eT4NGcfHZv4aPgda+c+Fb/aZOUi
4nnjPp8TNW312uaM/Eybs7SB8/C5kKdwcgGpfZHOB01wwjKGlgJhY7vpn5p6Rvqf
rfr3O9O7sySS6iMAqGaPyQg3S9/ivXx6HgpT/tTHltUp1sjfCK0Jl6jGKg4iMzCu
XJDIU1fIGveRu+w1v8WBnS6fiHOTmj36NWxPlTUnvNqEm8Gk3JjktoFy/Ufa7O9I
DRriUW7A71TNdy4yoS+Jr49QATh6smkemcm63TbvS1ZDXskCASOgADANBgkqhkiG
9w0BAQsFAAOCAQEAE2LTq/P0nsYLhgnrRLK6LJ/glXJnAyalkvGg+UsUZRkTS28H
zPkmZ044TlZP8LJKjjJxB/dhlmzt9VtvA7ee0IHQcK750VWHqU0FNZ6XSIZvyjhP
6TpfoPUeGLLOJ4wlIA5LXevUdAX7zapLfzbmHyU5C9x0Od35vjgeLx5Pw+31WREw
x47T0lylrfoTQSZuz+uRP48eBkbhTvqV7gBb2nijaK3lzah4fKKgXcieMNZdJXT+
kM+tqcyWBnzRlWcR5Qcq23+MvfKOsTU3Sl/ebmJV51nN70W189ptyx9Y2SXmPRmT
nFo1dUMmQNqUBXNhpn1+WBVKXVj/lz9G0Y7d/A==
-----END CERTIFICATE REQUEST-----

We have strong indications that there are several different lists of precalculated vulnerable keys whose precise populations depend on combinations of:

  • architecture
  • key size
  • process ID
  • (under certain conditions) application (openssh, openvpn, openssl)
  • openssl environment (for example, the use of .rnd file in the user's home directory and whether openssl can read this file).

Each different combination of the above elements produces a set of 32768 weak keys. These variables seem to affect the key generation process, producing different results, but this is not mentioned in https://wiki.debian.org/SSLkeys. We were not aware that the application and variables produce different keys. It would require someone to open the blacklist packages, read and understand the source code, in order to generate a complete list of Debian weak keys. The lists provided by our CA software vendor and https://github.com/g0tmi1k/debian-ssh were probably produced using openssh. While openssh uses libcrypto/ssl for the key generation, it seems to generate different keys than openssl itself.

We welcome any positive input by the community as an opportunity to improve our practices and communal PKI operations generally. Our team is evaluating possible improvements for this process, such as acquiring actual public keys created by the weak Debian RNG and using those keys to produce fingerprints compatible with our CA software. We have also contacted our CA software vendor regarding an improved blacklist.

Once we successfully augment our blacklist of Debian weak keys, and according to our existing practices, we will rescan our certificate corpus.

In our understanding, the events detailed in this bug do not constitute a compliance violation that rises to the level of an incident. However, our investigation will continue and we intend to share with the community our findings regarding the best solution to extending blacklist databases with more Debian weak keys.

Flags: needinfo?(support)

(In reply to Chris Kemmerer from comment #2)

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

SSL.com rejects any certificate request if the request has a known weak Private Key. For the purpose of identifying whether a Private Key is weak, SSL.com uses a set of Debian weak keys that was provided by our CA software vendor as the basis for our blacklist. We identified an additional source of Debian weak keys (https://github.com/g0tmi1k/debian-ssh) that already included all 2048-bit keys identified by our CA software vendor but had additional 4096-bit keys. We incorporated the 4096 bit keys from that source into our blacklist as well. This information was disclosed on 2020-03-07 to the person that submitted the Certificate Problem Report.

While incorporating the 4096-bit keys is useful, this seems to be a 2048-bit key, that's part of the canonical Debian package mentioned at https://wiki.debian.org/SSLkeys - namely, part of the openssl-blacklist package, as mentioned in Comment #0.

"The CA SHALL reject a certificate request if the requested Public Key does not meet the requirements set forth in Sections 6.1.5 and 6.1.6 or if it has a known weak Private Key (such as a Debian weak key, see http://wiki.debian.org/SSLkeys)"

Our understanding is that there is no single or complete list with all known Debian weak keys, either one that is normative for use by the CAs included in the Mozilla Root Program, nor one specified in the Baseline Requirements.

This is useful, and I agree, worth clarifying. That said, that page does provide a reference of keys, and this key appears to be in that set, so I'm a bit more concerned than if this was, say, a key outside of that set.

We have strong indications that there are several different lists of precalculated vulnerable keys whose precise populations depend on combinations of:

  • architecture
  • key size
  • process ID
  • (under certain conditions) application (openssh, openvpn, openssl)
  • openssl environment (for example, the use of .rnd file in the user's home directory and whether openssl can read this file).

Each different combination of the above elements produces a set of 32768 weak keys. These variables seem to affect the key generation process, producing different results, but this is not mentioned in https://wiki.debian.org/SSLkeys. We were not aware that the application and variables produce different keys.

This seems mentioned at https://wiki.debian.org/SSLkeys#How_weak.3F (for architecture/pid variations, as well as number of keys)

The application variable is mentioned at https://wiki.debian.org/SSLkeys#PEM_keys_.28SSL_certificates.29 and onward.

It would require someone to open the blacklist packages, read and understand the source code, in order to generate a complete list of Debian weak keys. The lists provided by our CA software vendor and https://github.com/g0tmi1k/debian-ssh were probably produced using openssh. While openssh uses libcrypto/ssl for the key generation, it seems to generate different keys than openssl itself.

Yes, that's mentioned on the page.

We welcome any positive input by the community as an opportunity to improve our practices and communal PKI operations generally. Our team is evaluating possible improvements for this process, such as acquiring actual public keys created by the weak Debian RNG and using those keys to produce fingerprints compatible with our CA software. We have also contacted our CA software vendor regarding an improved blacklist.

Once we successfully augment our blacklist of Debian weak keys, and according to our existing practices, we will rescan our certificate corpus.

It seems like a good starting point is the openssl-blacklist mentioned on that page, and can be a short-term scan, while working out additional sources.

Flags: needinfo?(support)

Unfortunately, the format of the blacklists in openssl-blacklist is incompatible with our CA software, which expects the SHA256 hash of the modulus. This is why we mention acquiring the actual public keys. While we have used the openssl-blacklist package to conduct a short-term scan, as you've stated, this would be inefficient compared to the long term solution.

We believe it would be to the benefit of the community if there was a common base list for all CAs, as that would prevent similar issues from occurring, at least involving keys in that common list. If we end up independently regenerating the keys included in the openssl-blacklist package, we pledge to share the public part of those keys with the whole community. In our understanding, many individuals are in possession of similar lists and it would certainly expedite this effort, and reduce waste from duplicate work, if these were shared with the broader community.

In the meantime, we have calculated the fingerprints of the public keys used in our issued Certificates using the algorithm found in openssl-blacklist. We compared these fingerprints against the openssl-blacklist and found only one match, which is the certificate issued to the security researcher that filed the Certificate Problem Report, and which was revoked within 24 hours of reporting. Therefore, our focus is now to find a permanent and efficient method of detecting weak Debian keys created by vulnerable versions of OpenSSL in popular architectures, as this seems to be the best way to meet the community's needs.

Flags: needinfo?(support)

(In reply to Chris Kemmerer from comment #4)

Unfortunately, the format of the blacklists in openssl-blacklist is incompatible with our CA software, which expects the SHA256 hash of the modulus. This is why we mention acquiring the actual public keys. While we have used the openssl-blacklist package to conduct a short-term scan, as you've stated, this would be inefficient compared to the long term solution.

I'm sorry you came away with the impression that I was finding SSL.com's answers reassuring. I find them deeply disconcerting, and continue to do so. For example, this feels very much like an attempt to foist the blame on the CA vendor, especially on the basis that the requirements weren't precise enough for SSL.com to have known better, despite evidence to the contrary.

Rather than being reassuring, that feels deeply negligent and dismissive. I don't disagree that more specificity gets more consistency, but I also don't see anything in this response that suggests SSL.com is taking this matter seriously, as opposed to just shrugging and going "Well, what ya gonna do"?

My previous response tried to point out how even SSL.com's assertions of doing the minimum were, in fact, highly suspect. The canonical place, mentioned in the BRs itself, references the use of this package, and so any question of "how should we do this" should make "regenerating the keys included in the openssl-blacklist package" seem like an unnecessary delay. That SSL.com chose to go a different route (whether directly or through their selection of a CA provider) still brings a question about /why/ they would choose to ignore existing databases and prior art, and importantly, what steps its taking to resolve this /and other future matters of confusion/.

For example, how SSL.com handles RoCA, or other vulnerabilities, is now called into question. Similarly, the use of other data sources, such as the Public Suffix List, referenced in the BRs, raises similar questions. I'm not suggesting that SSL.com should have perfect knowledge of all expectations, but any semblance of taking this matter seriously does not seem to be coming across, as the current impression seems to be that there was nothing wrong except, for, well there was.

Flags: needinfo?(support)

We regret your impression that we take this issue with anything less than the utmost seriousness.

We have opened a ticket and are actively working with our CA software vendor to address the underlying issue.

Rather than stopping there, we have been working concurrently to put into place the necessary checks against openssl-blacklist independently of the CA software vendor.

Whether through our CA software vendor or independently, we are committed to finding a long-term solution that is effective and efficient.

We will provide regular updates of our progress to this bug.

Flags: needinfo?(support)

This is the final Report

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

SSL.com became aware of this issue via submission of a Certificate Resolution Request (see timeline).

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

Dates in ISO8601 format in UTC timezone.

2020-03-07T03:18Z: SSL.com received a Certificate Resolution Request requesting the revocation of a certificate that was considered to be using a weak private key.
2020-03-07T03:23Z: Revocation Request ticket was created.
2020-03-07T03:44Z: Investigation for key compromise begun.
2020-03-07T05:15Z: The weak key detection engine was checked and found to be in place and functioning.
2020-03-07T11:15Z: Confirmed that the certificate mentioned in the Certificate Resolution Request uses a key not included in the weak keys blacklist that SSL.com maintains.
2020-03-07T13:28Z: As the reporter demonstrated control of the private key associated with the issued certificate, the certificate was revoked
2020-03-07T14:14Z: Full scan of certificate database started (in parallel with other research) to reveal if any blacklisted keys were used in issued certificates. No certificate was found using a key included in the blacklist.
2020-03-07T19:20Z: The reported key was added to the weak keys blacklist.
2020-03-07T20:41Z: Both the Subscriber and reporter were notified.
2020-03-07T20:47Z: Preliminary report added to bug.
2020-03-10T13:28Z: CA Software vendor was contacted in search for an updated blacklist key information that contains more "flavors" of Debian weak keys.
2020-03-11T08:00Z: Full scan of certificate database against the "openssl-blacklist" set of weak debian keys generated by OpenSSL in various architectures and parameters. Only the affected certificate of this incident was detected.
2020-03-11T10:00Z: Additional weak Debian key detection engine was developed to test against the "openssl-blacklist" package.
2020-03-13T16:35Z: Completed testing of additional weak Debian key detection engine and pushed to production.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

SSL.com rejects any certificate request if the request has a "known weak Private Key". For the purpose of identifying whether a Private Key is weak, SSL.com used a set of Debian weak keys that was provided by our CA software vendor as the basis for our blacklist. We identified an additional source of Debian weak keys (https://github.com/g0tmi1k/debian-ssh) that already included all 2048-bit keys identified by our CA software vendor but had additional 4096-bit keys. We incorporated the 4096 bit keys from that source into our blacklist as well. This information was disclosed on 2020-03-07 to the researcher who submitted the Certificate Problem Report.

Similarly, SSL.com also checks for keys that are affected by the ROCA vulnerability.

The above practice complies with our CP/CPS, version 1.8, section 6.1.1.2, which states:

"SSL.com shall reject a certificate request if the request has a known weak Private Key".

Our understanding was that there is no single or complete list with all known Debian weak keys, either one that is normative for use by the CAs included in the Mozilla Root Program, nor one specified in the Baseline Requirements. After a discussion in m.d.s.p. we realized that the community's expectation was for CAs to be able to block issuance of weak Debian keys produced by OpenSSL at least, as captured in the openssl-blacklist package that is described in http://wiki.debian.org/SSLkeys.

We used this package to scan our certificate database against weak Debian keys produced by vulnerable versions of OpenSSL and no certificates were detected using such weak keys, except for the certificate issued to the security researcher who tested our weak Debian key detection mechanism. That certificate was revoked within 24 hours in accordance with all applicable requirements (CP/CPS, BRs).

Our CA software contains a key blacklist engine into which the CA operator can "feed" public keys (RSA/ECDSA). These keys can then be identified in any future CSR that contains such a public key. Our long-term solution would be to use this "native" CA software engine to feed all the weak Debian keys produced by vulnerable versions of OpenSSL to meet this requirement at least up to the community's expectations. We have reached out to the community in search for such keys. This would improve the quality and speed of the detection mechanism and at the same time remove the need for additional code to be developed, thus minimizing the complexity of the remediation.

Our CA software vendor was contacted and informed about the request to augment the blacklist database with the weak keys produced by OpenSSL. We intend to remain in communication with them regarding improvements to their blacklist engine, which at time of filing this report are in progress.

While waiting for the vendor's response, our engineering team intensified its efforts and developed , tested and deployed a new code module to our RA application which now successfully rejects the additional keys included in the openssl-blacklist package from being signed.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

See below.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

One certificate:

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

We considered that we had made reasonable efforts to fulfil both our CP/CPS and the Baseline Requirements Section 6.1.1.3:

"The CA SHALL reject a certificate request if the requested Public Key does not meet the requirements set forth in Sections 6.1.5 and 6.1.6 or if it has a known weak Private Key (such as a Debian weak key, see http://wiki.debian.org/SSLkeys)"

Our understanding was that there is no single or complete list with all known Debian weak keys, either one that is normative for use by the CAs included in the Mozilla Root Program, nor one specified in the Baseline Requirements. After a discussion in m.d.s.p. we realized that the expectation was for CAs to block issuance of weak Debian keys produced by OpenSSL at least, as captured in the openssl-blacklist package that is described in http://wiki.debian.org/SSLkeys.

Other CAs had a similar interpretation, e.g. https://community.letsencrypt.org/t/2017-09-09-late-weak-key-revocation/42519, which is a sign that the language of this requirement as presented in the BRs doesn't clearly capture community expectations. We will support an update of this requirement so that all CAs in the Root Program clearly and unambiguously understand the minimum (baseline) expectations for detecting Debian weak keys.

At the same time, we acknowledge that we could do better in interpreting BRs. We should have considered that OpenSSL is the reference implementation that most of Subscribers default to, and search for detection solutions that are specific to that. To minimize the risk of such misinterpretations, we will enhance the role of our compliance team, so that it has the capacity to engage more actively in the implementation process, provide an independent interpretation of requirements and provide deeper evaluation of solutions proposed by the engineering team. The compliance team shall also extend our internal audit program and conduct retrospective evaluation of our policies and practices, in accordance with our objective to meet and exceed all applicable expectations.

As already mentioned, SSL.com remains committed to improving security across the community.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

In order to prevent the problem from happening again in the future, and as a parallel effort while waiting for the vendor's response using the "native" blacklist engine, our engineering team developed and deployed the new module to the RA application mentioned above. This module rejects the additional Debian weak keys produced by OpenSSL that are included in the openssl-blacklist package.

In addition to the remediation related to this issue, we will initiate a discussion at the CA/B Forum to improve the language of section 6.1.1.3 of the BRs clarifying the minimum expectations for the Debian weak key detection engine that CAs must implement.

It appears that all questions have been answered and remediation is complete.

Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED

Wayne: I don’t think this was ready to be closed. SSL.com has committed, but not delivered, on additional remediations. While I can understand wanting to close this, due to the disappointment in the response, it seems like:

  • Updating their CA software to support sensible blocklisting
  • Proposing more effective language that they believe would have prohibited their creative and problematic interpretation

SSL.com hasn’t committed to dates on delivering either, and a lack of weekly updates is concerning, but I don’t think this is ready to close out.

Flags: needinfo?(wthayer)

I've reopened this. However, my understanding is that SSL.com has already updated their software:

While waiting for the vendor's response, our engineering team intensified its efforts and developed , tested and deployed a new code module to our RA application which now successfully rejects the additional keys included in the openssl-blacklist package from being signed.

My understanding is that the vendor's enhancement will do the same thing.

I'm not opposed to keeping this open until a proposal is made to the CAB Forum, but I also think one could argue that it's beyond the scope of this bug.

Status: RESOLVED → REOPENED
Flags: needinfo?(wthayer)
Resolution: FIXED → ---

Comment #4 highlights the difference between the short-term response versus what their CA is providing, which SSL.com will hopefully provide an ETA for. The current system design limits SSL.com’s ability to effectively blocklist keys.

We actually patched the RA to block any keys listed in openssl-blacklist on 03-13-2020 within 1 week of opening the bug so users were not able to submit nor get certs issued for those weak keys.

Due to the urgency of the matter we wanted to be able to block that list while we worked with the CA vendor to send us a list update.

As for the BR language update, with the COVID19 outbreak we weren't sure if it was a good time to begin this process.

We can certainly provide something this week to get the ball rolling but we're concerned about participation from the community due to COVID19.

In any case since we've put the remediation in place, we'd like to consider this incident closed. We expect the BR language discussion to ongoing considering the current circumstances.

Leo: I think it would be helpful to state to complete set of steps SSL.com is taking. I’m having difficulty squaring Comment #12 with Comment #4. If the answer is that SSL.com views the total remediation as being using openssl-blacklist, as was already called out in the BRs, then we can certain close this, but with a note that it’s a very disappointing incident response.

I’m hoping you can carefully evaluate Questions 6 and 7 in the template, and make sure it’s clear the steps SSL.com has or is proposing to take, and when, because it seems there’s been some contradictory, or at least confusing, statements here.

(In reply to Ryan Sleevi from comment #9)

Wayne: I don’t think this was ready to be closed. SSL.com has committed, but not delivered, on additional remediations. While I can understand wanting to close this, due to the disappointment in the response, it seems like:

  • Updating their CA software to support sensible blocklisting

For completeness sake, on March 25, 2020 we implemented an external validator in our CA software provided by the vendor that checks against the openssl-blacklist package in addition to the existing key blacklist database. Since we didn't have access to the public keys, we could not update the blacklist of our CA software directly.

We view both the RA and CA checks, both currently in production, as long term solutions

  • Proposing more effective language that they believe would have prohibited their creative and problematic interpretation

As mentioned in comment #12 we will proceed this week to get the discussion started

SSL.com hasn’t committed to dates on delivering either, and a lack of weekly updates is concerning, but I don’t think this is ready to close out.

We were under the impression the report provided in comment #7 was satisfactory and that further updates were not needed. We'll try to do better next time by posting to the bug to see if there is anything else required of us to close the incident.

Ryan, I hope this addresses your concerns. If not, I would appreciate some additional guidance from you so we can close this incident response in a satisfactory way.

I can only conclude that, despite Comment #4, SSL.com has not, and does not plan, to address the following issues:

  • That their system for blocklisting ”expects the SHA256 hash of the modulus.”

    • This is clearly distinct from incorporating openssl-blacklist
  • They do not plan to develop or support a common baselist of keys, nor have they developed one.

    We believe it would be to the benefit of the community if there was a common base list for all CAs, as that would prevent similar issues from occurring, at least involving keys in that common list. If we end up independently regenerating the keys included in the openssl-blacklist package, we pledge to share the public part of those keys with the whole community.

  • They they are no longer focused on how:

    to find a permanent and efficient method of detecting weak Debian keys created by vulnerable versions of OpenSSL in popular architectures, as this seems to be the best way to meet the community's needs.

  • That their investigation is complete and they no longer:

    intend to share with the community our findings regarding the best solution to extending blacklist databases with more Debian weak keys.

  • That they do not plan to share

    how SSL.com handles RoCA, or other vulnerabilities

    other than a self-attested statement that likely shares the same interpretative issues we see here, that

    SSL.com also checks for keys that are affected by the ROCA vulnerability

    Without any description that might highlight similar misapplication

  • That they have not examined, or do not plan to share how SSL.com handles

    the use of other data sources, such as the Public Suffix List, referenced in the BRs,

Status: REOPENED → RESOLVED
Closed: 4 years ago4 years ago
Resolution: --- → FIXED

Ryan, thank you for providing more clarity. SSL.com intends to honor all commitments from comment #4. I will provide a followup during the week after consulting with our PKI team.

(In reply to Ryan Sleevi from comment #15)

I can only conclude that, despite Comment #4, SSL.com has not, and does not plan, to address the following issues:

  • That their system for blocklisting ”expects the SHA256 hash of the modulus.”
    • This is clearly distinct from incorporating openssl-blacklist
  • They do not plan to develop or support a common baselist of keys, nor have they developed one.

    We believe it would be to the benefit of the community if there was a common base list for all CAs, as that would prevent similar issues from occurring, at least involving keys in that common list. If we end up independently regenerating the keys included in the openssl-blacklist package, we pledge to share the public part of those keys with the whole community.

We appealed to the Mozilla community and to anyone with access to the public keys behind the openssl-blacklist package to disclose this information publicly, but we didn't hear anything back, otherwise we would have definitely shared that.

Since we were able to build a different, equally effective process to block these specific keys, we don't think it necessary to duplicate work already done by others.

  • They they are no longer focused on how:

    to find a permanent and efficient method of detecting weak Debian keys created by vulnerable versions of OpenSSL in popular architectures, as this seems to be the best way to meet the community's needs.

The changes we applied to both the RA and CA software are permanent and efficient for us to detect weak Debian keys created by vulnerable versions of OpenSSL in popular architectures.

  • That their investigation is complete and they no longer:

    intend to share with the community our findings regarding the best solution to extending blacklist databases with more Debian weak keys.

In Comment #7 we mentioned that the community's minimum expectation was for CAs to block vulnerable keys created by OpenSSL in various architectures and parameters. We also disclosed a source for additional Debian weak keys that were not included in the openssl-blacklist package, although the community doesn't consider this list very useful.

We also concluded that there is no single or complete list with all known Debian weak keys, either one that is normative for use by the CAs included in the Mozilla Root Program, nor one specified in the Baseline Requirements. However, I think once we help tighten up the language in the BRs as we have pledged to do, we can have more constistency among Publicly-Trusted CAs.

  • That they do not plan to share

    how SSL.com handles RoCA, or other vulnerabilities

    other than a self-attested statement that likely shares the same interpretative issues we see here, that

    SSL.com also checks for keys that are affected by the ROCA vulnerability

    Without any description that might highlight similar misapplication

Our focus was on answering the Debian weak keys issue. Our CA software uses https://github.com/crocs-muni/roca/blob/master/java/BrokenKey.java for RoCA weak key detection.

  • That they have not examined, or do not plan to share how SSL.com handles

    the use of other data sources, such as the Public Suffix List, referenced in the BRs,

SSL.com uses regularly updated lists from the Public Suffix List as documented in https://publicsuffix.org/list/ to block issuance of wildcard certificates in accordance with section 3.2.2.6

We hope this response is satisfactory. If not, let me know what else we can provide.

(In reply to Leo Grove from comment #17)

Since we were able to build a different, equally effective process to block these specific keys, we don't think it necessary to duplicate work already done by others.

Trying to square this reply:

  • Original Pledge: "If we end up independently regenerating the keys included in the openssl-blacklist package, we pledge to share the public part of those keys with the whole community."
  • Status: SSL.com did not end up independently regenerating the keys, thus does not having anything to share.

Is that a correct understanding? And if so, is the statement below simply aspirational in nature, rather than any specific commitment to any action or remediation?

We believe it would be to the benefit of the community if there was a common base list for all CAs, as that would prevent similar issues from occurring, at least involving keys in that common list.

  • That they do not plan to share

    how SSL.com handles RoCA, or other vulnerabilities

    other than a self-attested statement that likely shares the same interpretative issues we see here, that

    SSL.com also checks for keys that are affected by the ROCA vulnerability

    Without any description that might highlight similar misapplication

Our focus was on answering the Debian weak keys issue. Our CA software uses https://github.com/crocs-muni/roca/blob/master/java/BrokenKey.java for RoCA weak key detection.

Thanks, this is helpful.

  • That they have not examined, or do not plan to share how SSL.com handles

    the use of other data sources, such as the Public Suffix List, referenced in the BRs,

SSL.com uses regularly updated lists from the Public Suffix List as documented in https://publicsuffix.org/list/ to block issuance of wildcard certificates in accordance with section 3.2.2.6

We hope this response is satisfactory. If not, let me know what else we can provide.

This least addresses that you're not using some fork of the Public Suffix List, as one might conclude that SSL.com would see as perfectly fine and allowed.

Overall, I'm not terribly encouraged by this incident response. This seems to go in the opposite direction of many of the good practices captured on m.d.s.p.. Of course, you can lead a horse to water, but you can't make him drink; these are good practices, not mandatory practices. I would strongly encourage SSL.com to evaluate that thread in dealing with future incident responses, so we can look at both systemic flaws and opportunities for ecosystem-wide improvements. When interpretative issues arise that do not seem to affect other CAs, that's concerning. When issues arise with the CA implementation itself, sharing precise and technical details to facilitate independent implementations is equally valuable. While I appreciate the attentiveness to "fix SSL.com", I think it's equally important to fix the ecosystem; from both a technical and a compliance side.

I'm kicking this back over to Wayne, despite my serious reservations about some of the replies here and how well they inspire confidence. This hardly seems the hill to plant a flag on arguing that the existing implementation was OK, but that's apparently the tactic still taken.

Flags: needinfo?(wthayer)

SSL.com has (kicked off a discussion)[https://cabforum.org/pipermail/servercert-wg/2020-April/001821.html] of the BR updates mentioned in comment #7.

I have no further questions.

Flags: needinfo?(wthayer)
Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [dv-misissuance]
You need to log in before you can comment on or make changes to this bug.