1904399 - CommScope: OCSP responses contain issuer certificate

Status: RESOLVED INVALID

(CA Program :: CA Certificate Compliance, task)

Description

[Andrew Ayer]

OCSP Watch is reporting 20 certificates issued by CommScope whose OCSP response signatures cannot be verified. crt.sh also reports failures, e.g.:

https://crt.sh/?sha256=7da29d598908b73e3f2c39f5205da22021af947f3448ebde3af2eed1854fd789&opt=ocsp

https://crt.sh/?sha256=03d47fec41ec4dd5c63a036d6b80a1bdf67b6c3ee37c25c1dcb81f5acc852e80&opt=ocsp

Comment 1

[Wayne]

I can confirm this is still true, and I'm not seeing a response from Commscope regarding this Certificate Problem Report over 4 days in?
https://crt.sh/?sha256=7da29d598908b73e3f2c39f5205da22021af947f3448ebde3af2eed1854fd789&opt=ocsp

Mechanism	Provider	Status	Revocation Date	Last Observed in CRL	Last Checked (Error)
OCSP	The CA	ocsp.ParseResponseForCert => bad OCSP signature: x509: ECDSA verification failure	n/a	n/a	2024-06-28 19:12:05 UTC

Comment 2

[Wayne]

This is still happening and I am not seeing any acknowledgement?

Comment 3

[Nicol So]

(In reply to Andrew Ayer from comment #0)

OCSP Watch is reporting 20 certificates issued by CommScope whose OCSP response signatures cannot be verified. crt.sh also reports failures, e.g.:

https://crt.sh/?sha256=7da29d598908b73e3f2c39f5205da22021af947f3448ebde3af2eed1854fd789&opt=ocsp

https://crt.sh/?sha256=03d47fec41ec4dd5c63a036d6b80a1bdf67b6c3ee37c25c1dcb81f5acc852e80&opt=ocsp

We started investigating the cause of your observations shortly after your report. We held off posting a reply until we felt quite certain we understood the cause.

We have done testing of our own using OpenSSL, Firefox (configured to require OCSP), an open source OCSP checker, and several web-based OCSP checking tools operated by third parties. Outside of OCSP Watch and crt.sh, which your report cites, none of the tests we performed reported an issue with OCSP signature verification.

We have traced the issue to what we consider a bug in a Go package (golang.org/x/crypto/ocsp), which we believe is used by both OCSP Watch and crt.sh. In particular, the package makes assumptions [1] about the content of the "certs" field in a BasicOCSPResponse structure that are not supported by RFC 6960. In the OCSP responses for certificates issued by our public CAs, we include the issuer's certificate in the "certs" field in a BasicOCSPResponse structure. The "ocsp" Go package incorrectly assumes that, if "certs" is present, the first certificate in "certs" must be the certificate of a (delegated) OCSP responder and handles the certificate as such without checking. This will result in a signature verification failure when the certificate is not what the "ocsp" package assumes it to be.

Our usage of "certs" is consistent with the relevant language in [2] and what it implies.

[1] https://cs.opensource.google/go/x/crypto/+/master:ocsp/ocsp.go, line 554--580 (Retrieved 2024-07-01)
[2] "The responder MAY include certificates in the certs field of BasicOCSPResponse that help the OCSP client verify the responder's signature. If no certificates are included, then certs SHOULD be absent." (RFC 6960, Section 4.2.1)

Comment 4

[Nicol So]

(In reply to Wayne from comment #2)

This is still happening and I am not seeing any acknowledgement?

Please see my response to the report in comment #3. (The comment was about to be submitted when your question was posted.)

Comment 5

[Andrew Ayer]

Thank you for the analysis in Comment 3. It is correct that OCSP Watch and crt.sh both use the Go standard library to verify OCSP responses. However, I do not agree with the conclusion.

Including the certificate issuer's certificate in the OCSP response is superfluous since the client already knows this certificate, and therefore doesn't need to get it from the response to verify the responder's signature. The certs field is definitely not intended to contain it, and I believe RFC 6960's use of the word "help" implies that the certs field should contain only helpful certificates, not unnecessary ones. I think it's reasonable for the Go standard library to make the assumption that it does. Note that no other publicly-trusted CA is doing what CommScope is doing.

Comment 6

[Aaron Gable]

I agree that CommScope should stop including this superfluous certificate in their OCSP responses, as it drastically increases the size of the response. However, I do not agree that including it is in violation of RFC 6960.

The go library's job is to determine if the OCSP response is valid. To do that, it needs to validate the signature on the OCSP response as either a) coming from the issuer of the certificate itself, or b) coming from a delegated responder. By my reading, it fails in this task in at least two ways:

• it does not take into account the possibility that the embedded certificate is superfluous, or provided to help clients that don't already have the original issuer on hand; and
• it does not verify that the delegated responder (if present) contains the id-kp-OCSPSigning EKU, as required by RFC 6960 Section 4.2.2.2.

To be clear, the x/crypto/ocsp library does document this behavior:

// If the response contains an embedded certificate, then that certificate will
// be used to verify the response signature. If the response contains an
// embedded certificate and issuer is not nil, then issuer will be used to verify
// the signature on the embedded certificate.

But that does not mean that all OCSP producers are required to abide by this library's behavior. An equally valid implementation could use the embedded certificate to verify the response signature, and then confirm that the embedded certificate and the original certificate's issuer are the same.

Comment 7

[Corey Bonnell]

I agree with Aaron that Golang OCSP response parsing behavior does not establish any sort of standard in regard to proper handling of OCSP responses. From a brief read of the code, there are several missing validation steps in addition to the shortcomings that Aaron mentioned that render the OCSP parsing API as wholly unsuitable for use as a OCSP verifier API as-is. Any application using the parsing API as a OCSP verifier without performing supplementary checks on its own is insecure in that OCSP responses that would fail the PKIX validation algorithm would be accepted without error. For example, it appears that the API would accept a OCSP response that is signed by the subject certificate (or any other certificate issued by the same issuer) specified in the CertID provided that the subject certificate is included in the certs bag.

As for this issue, I think it should be resolved as INVALID. The addition of the CA certificate in the certs bag is potentially sub-optimal from a size standpoint, unless it's being added to convey the CA certificate to a RP that does not yet have it. It's possible to construct a valid OCSP request using data solely present in the subject certificate in many cases, so this mechanism can serve as a form of AIA chasing. Regardless, there is no prohibition in RFC 2560 or 6960 on including the subject certificate's issuer, so I don't see how the OCSP responder in question is non-compliant.

Comment 8

[Andrew Ayer]

The addition of the CA certificate in the certs bag is potentially sub-optimal from a size standpoint, unless it's being added to convey the CA certificate to a RP that does not yet have it.

In what situation would the RP not already have this certificate, considering it needs the issuer's public key to calculate the CertID in the OCSP request?

RFC 6960 section 4.1.1 (ASN.1 Specification of the OCSP Request): "issuerKeyHash is the hash of the issuer's public key. The hash shall be calculated over the value (excluding tag and length) of the subject public key field in the issuer's certificate."

It's possible to construct a valid OCSP request using data solely present in the subject certificate in many cases, so this mechanism can serve as a form of AIA chasing.

What are these "many cases"? If you're suggesting that the client could use the AKI under the assumption that it is calculated a certain way, that is extremely un-robust and out-of-spec of RFC 5280 and RFC 6960. It would fail, for instance, with Let's Encrypt, which uses truncated SHA-256 hashes for SKI/AKIs. Do you have any evidence that OCSP was intended to be used this way, or that any clients do this?

Regardless, there is no prohibition in RFC 2560 or 6960 on including the subject certificate's issuer, so I don't see how the OCSP responder in question is non-compliant.

Does RFC 6960 place any restriction on the contents of the certs field, or can CAs put any certificate in it?

Every other publicly-trusted CA has taken the position that this field should not contain this certificate. It would be interesting to hear why CommScope has chosen to include it, ideally citing some real-world scenarios where including this extra certificate is helpful to clients.

Comment 9

[Corey Bonnell]

In what situation would the RP not already have this certificate, considering it needs the issuer's public key to calculate the CertID in the OCSP request?

A TLS server serving an incomplete chain is an example that immediately comes to mind. Also, one necessarily does not need the issuer certificate if the SKI calculation method is known (see below).

What are these "many cases"? If you're suggesting that the client could use the AKI under the assumption that it is calculated a certain way, that is extremely un-robust and out-of-spec of RFC 5280 and RFC 6960.

RFC 6960 does not specify how exactly or who is calculating the issuerKeyHash. If the RP has a priori knowledge that the CA uses a given SKI calculation method, it's perfectly acceptable to use the AKI keyIdentifier value as the issuerKeyHash. Even absent a guarantee of a consistent algorithm, this approach works well for many CAs, although it does not work anymore for Let's Encrypt due to the switch to RFC 7093 method 1 SKIs, as you mentioned.

Do you have any evidence that OCSP was intended to be used this way

No, but I don't see how using it this way violates any standard. Protocol standards generally do not exhaustively enumerate every acceptable implementation of the protocol.

or that any clients do this?

Yes, I wrote one, which is why I mentioned this method.

Does RFC 6960 place any restriction on the contents of the certs field, or can CAs put any certificate in it?

The only guidance that 6960 provides is: The responder MAY include certificates in the certs field of BasicOCSPResponse that help the OCSP client verify the responder's signature.

If the responder operator believes a certificate helps a OCSP client to verify the response signature, then it is free to add it. If a further restriction is desired, then I think a further profiling of acceptable OCSP responses needs to be added to policy.

It would be interesting to hear why CommScope has chosen to include it

I believe they're not the first CA to do this judging from some old Golang issues I stumbled across, but I'd also be interested to hear the rationale.

Comment 10

[Rob Stradling]

In what situation would the RP not already have this certificate, considering it needs the issuer's public key to calculate the CertID in the OCSP request?

A TLS server serving an incomplete chain is an example that immediately comes to mind. Also, one necessarily does not need the issuer certificate if the SKI calculation method is known (see below).

If the RP doesn't already have the issuing CA certificate, then that RP won't yet have built and verified a chain to a trusted root. Why then would that RP want to check the revocation status of that (possibly fake) certificate's serial number via OCSP?

Comment 11

[Corey Bonnell]

If the RP doesn't already have the issuing CA certificate, then that RP won't yet have built and verified a chain to a trusted root. Why then would that RP want to check the revocation status of that (possibly fake) certificate's serial number via OCSP?

As far as I'm aware, there's no defined order of chain discovery and retrieval of associated revocation artifacts relative to certification path validation. For example, OpenSSL fetches CRLs prior to verifying signatures on a candidate certification path: https://medium.com/@sleevi_/path-building-vs-path-verifying-implementation-showdown-39a9272b2820.

Comment 12

[Nicol So]

(In reply to Andrew Ayer from comment #5)

Thank you for the analysis in Comment 3. It is correct that OCSP Watch and crt.sh both use the Go standard library to verify OCSP responses. However, I do not agree with the conclusion.

Including the certificate issuer's certificate in the OCSP response is superfluous since the client already knows this certificate, and therefore doesn't need to get it from the response to verify the responder's signature. The certs field is definitely not intended to contain it, and I believe RFC 6960's use of the word "help" implies that the certs field should contain only helpful certificates, not unnecessary ones. I think it's reasonable for the Go standard library to make the assumption that it does. Note that no other publicly-trusted CA is doing what CommScope is doing.

Our usage of “certs” is conformant to RFC 6960. On that point, Aaron Gable and Corey Bonnell agree. MAY is permissive and does not mean that any usage not explicitly included in a MAY clause is prohibited; the issuer certificate is helpful for validating an OCSP response. An OCSP client is just an entity that queries the revocation status of a certificate; depending on the nature of the application it is a part of, it may or may not already have the issuer certificate of the certificate being queried. The “certs” structure in RFC 6960 generally may contain multiple certificates. RFC 6960 does not specify the exact nature of what those certificates must be, and more importantly, it specifies no particular order in which those certificates must be presented. This means a client cannot assume that the first certificate in “certs” is a certificate that plays a particular role. Any client that wants to use the optional information present in “certs” must (as a necessity) be prepared to examine each certificate and analyze its relevance.

Although our existing implementation is allowed by RFC6960, in light of the discussion here we will follow other CAs' practice not to include the issuer cert in a response signed by the issuer. We’ll provide an update when that change is in effect.

Comment 13

[Nicol So]

Our work to implement the change mentioned in comment #13 is in progress. We’ll provide an update when that change has been deployed.

Comment 14

[Nicol So]

We have deployed the change we stated in comment #13 that we would make. The change is intended to avoid triggering the bug we identified in the Go package golang.org/x/crypto/ocsp and to align with common CA practices. After the deployment, we checked OCSP Watch and crt.sh, the two web-based tools that previous reported signature verification failures when querying our OCSP responder. The two websites did not report any signature verification failures.

Comment 15

[Nicol So]

It has been 9 days since we reported that we had completed the change we said we would make in comment #13. There has been no follow-up questions. We have no outstanding action items. CommScope would like to request that the issue be treated as resolved and closed.

Comment 16

[Ben Wilson]

Has CommScope considered posting an Incident Report as part of completing / closing this bug?
Thanks,
Ben

Comment 17

[Nicol So]

We do not think an incident report is warranted because this is not a true incident in our system. In this case, the observations of the original reporter were traced back to a bug in an OCSP library used by third-party applications (OCSP Watch and crt.sh). Our OCSP usage was conformant albeit uncommon. Our findings and reasoning have already been stated and explained in previous comments. We do not think we will have more significant insights to include in a formal report.

Comment 18

[Ben Wilson]

Are there any other comments or questions? If not, I will close this on or about Wednesday, Aug. 7, 2024.

Comment 19

[Andrew Ayer]

I think it's fine to close this out. That said, I'm highly incredulous of the rationales presented in this bug for including the issuer certificate. The only use case presented is an OCSP client that wrongly assumes that AKI is computed a certain way. I doubt the authors of RFC 6960 had that use case in mind when they wrote Section 4.2.1. Rather, they were probably trying to support a PKI in which OCSP responses are issued from a completely separate hierarchy (see this comment by Adam Langley). That's inapplicable to the WebPKI.

Any WebPKI CA that includes the issuer certificate in the OCSP response is doing so by mistake, and it's a mistake that wastes bandwidth and requires clients to be more complex. Go is unlikely to change their OCSP implementation to support CAs doing this. As AGL said, "the PKIX RFCs err strongly on the side of complexity, but that results in an over-complex mess and must be subsetted. The trick is pick the right battles and, over time, try to bring the complexity down."

The WebPKI is currently in a good state where no CAs are making this mistake. To prevent a regression, I recommend adding an entry to https://wiki.mozilla.org/CA/Required_or_Recommended_Practices that says:

If a CA includes a certs field in a BasicOCSPResponse, it MUST contain exactly one Authorized Responder certificate.

Longer term, assuming OCSP has a future in the WebPKI, OCSP responses should be precisely profiled in the BRs in the same way that certificates have been.

Comment 20

[Roman Fischer]

Putting this clarification (which I support) in the Mozilla recommended practises is IMHO not good. Wouldn't it be better to crate a git-issue for the TLS BR and track it there?

Comment 21

[Ben Wilson]

As a placeholder, I added this - https://wiki.mozilla.org/CA/Required_or_Recommended_Practices#OCSP_Responses - "Including additional, superfluous certificates in an OCSP response wastes bandwidth. Therefore, CA operators should limit certificates delivered to a single Authorized Responder certificate, when that is necessary." We can edit that language as necessary or advisable.

Comment 22

[Ben Wilson]

I will close this as Invalid unless someone can explain why I shouldn't.