Closed Bug 1398261 Opened 7 years ago Closed 6 years ago

Visa: Non-BR-Compliant OCSP Responders

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: kathleen.a.wilson, Assigned: masilva, NeedInfo)

References

Details

(Whiteboard: [ca-compliance] [ocsp-failure]) 

Problems have been found with OCSP responders for this CA, and reported in the mozilla.dev.security.policy forum here:

https://groups.google.com/d/msg/mozilla.dev.security.policy/o1MX07iWDco/RuM1NK_0AQAJ

As per section 4.9.10 of the BRs, OCSP responders MUST NOT respond with a “good” status for unissued certificates. The effective date for this requirement was 2013-08-01.

Please provide an incident report in this bug, as described here:
https://wiki.mozilla.org/CA/Responding_To_A_Misissuance#Incident_Report
Marcelo: it has now been 13 days. Please respond ASAP to tell us how you plan to address this issue.

Gerv
Hi Kathleen/Gerv, after numerous tests, we were unable to replicate this issue. We have then escalated this issue to our vendor to review, confirm and remediate any issues with the OCSP responder. They were able to reproduce the tests using the MS certutil command using a "fake" certificate including our OCSP URL, and got the expected result: Request unauthorized (6).

Please see below the test results:

=============
50          54          1:54:18 PM 9/22/2017   2.5227428          certutil.exe              10.86.141.251   ocsp.visa.com   OCSP     OCSP:Request, Version = v1 (0) {HTTP:32, TCP:31, IPv4:30}
  Frame: Number = 54, Captured Frame Length = 154, MediaType = WiFi
+ WiFi: [Unencrypted Data] .T....., (I) 
+ LLC: Unnumbered(U) Frame, Command Frame, SSAP = SNAP(Sub-Network Access Protocol), DSAP = SNAP(Sub-Network Access Protocol)
+ Snap: EtherType = Internet IP (IPv4), OrgCode = XEROX CORPORATION
+ Ipv4: Src = 10.86.141.251, Dest = 198.241.174.58, Next Protocol = TCP, Packet ID = 26918, Total IP Length = 90
+ Tcp: Flags=...AP..., SrcPort=55640, DstPort=HTTP(80), PayloadLen=50, Seq=694651240 - 694651290, Ack=3077059779, Win=258
+ Http: HTTP Payload, URL: /ocsp 
- OCSP: Request, Version = v1 (0)
  - OcspRequest: 0x1
   + SequenceHeader: 
   - TbsRequest: 
    + SequenceHeader: 0x1
      DefaultVersion: v1 (0)
    + SequenceofHeader: RequestList
    - Request: 
     - SequenceHeader: 
      + AsnId: Sequence and SequenceOf types (Universal 16)
      + AsnLen: Length = 42, LengthOfLength = 0
     - ReqCert: 
      + SequenceHeader: 
      + HashAlgorithm: IdSha1 (1.3.14.3.2.26)
      + IssuerNameHash: Hash of Issuer's DN
      - IssuerKeyHash: Hash of Issuer's public key
       + AsnOctetStringHeader: 
      - SerialNumber: 65568
       - SerialNumber: 65568
        + AsnIntegerHeader: 
          AsnInt: 65568 (0x10020)
              0            55          1:54:18 PM 9/22/2017   2.8383556              certutil.exe        ocsp.visa.com   10.86.141.251   TCP              TCP:Flags=...A...., SrcPort=HTTP(80), DstPort=55640, PayloadLen=0, Seq=3077059779, Ack=694651290, Win=513              {TCP:31, IPv4:30}
              383        56          1:54:18 PM 9/22/2017   2.8383556              certutil.exe        ocsp.visa.com   10.86.141.251   OCSP              OCSP:Response, Status = Request unauthorized (6)              {HTTP:32, TCP:31, IPv4:30}


=============

We are also testing a different vendor solution and planning to replace it soon.
Thanks,
Marcelo
I get an unauthorized response if I send a nonce, but that's appears to be true for any serial number, including valid ones (e.g. https://crt.sh/?q=329cf245b61e639b759b048b8f5c8399fe676be17766d38e18c23aac6a6e50e8. When passing no nonce the problem is clearly demonstrable.

Here's an easy way to replicate it using the OpenSSL CLI:

1) Download a copy of the Visa eCommerce Issuing CA (https://crt.sh/?id=12629307). You can click "download certificate PEM" on the left side of the page.
2) Within the directory you saved the file: openssl ocsp -issuer 12629307.crt -serial 1010101010101010101010101011010101011010 -url http://ocsp.visa.com/ocsp -no_nonce

You will get a response that looks like:

Response verify OK
1010101010101010101010101011010101011010: good
	This Update: Sep 29 17:00:01 2017 GMT
	Next Update: Oct  7 17:00:02 2017 GMT
(In reply to Paul Kehrer from comment #3)
> I get an unauthorized response if I send a nonce, but that's appears to be
> true for any serial number, including valid ones (e.g.
> https://crt.sh/
> ?q=329cf245b61e639b759b048b8f5c8399fe676be17766d38e18c23aac6a6e50e8. When
> passing no nonce the problem is clearly demonstrable.
> 
> Here's an easy way to replicate it using the OpenSSL CLI:
> 
> 1) Download a copy of the Visa eCommerce Issuing CA
> (https://crt.sh/?id=12629307). You can click "download certificate PEM" on
> the left side of the page.
> 2) Within the directory you saved the file: openssl ocsp -issuer
> 12629307.crt -serial 1010101010101010101010101011010101011010 -url
> http://ocsp.visa.com/ocsp -no_nonce
> 
> You will get a response that looks like:
> 
> Response verify OK
> 1010101010101010101010101011010101011010: good
> 	This Update: Sep 29 17:00:01 2017 GMT
> 	Next Update: Oct  7 17:00:02 2017 GMT

Thank you Paul.
I was able to run the test now, and actually I've noticed some issues on the results. 
I can see the "Good" response as you have indicated, but I can also notice that our OCSP service has indicated a “Response verify failure” message, indicating that the Issuer Certificate wasn’t verified, of course because our OCSP doesn’t recognized any info from this fake certificate.
By using the Windows Certutil tool to perform the same test we didn't get a good response, and we were wondering whether the openssl doesn't have an appropriate behavior when this type of test is done. Therefore, we have escalated it to the vendor again looking for some clarification.

Please see below the complete results:

C:\OpenSSL\bin>openssl ocsp -issuer eCommIssuingCA.crt -serial 1010101010101010101010101011010101011010 -url http://ocsp.visa.com/ocsp -no_nonce
Response Verify Failure
15440:error:27069065:OCSP routines:OCSP_basic_verify:certificate verify error:ocsp_vfy.c:138:Verify error:unable to get local issuer certificate
1010101010101010101010101011010101011010: good
        This Update: Oct  1 17:00:01 2017 GMT
        Next Update: Oct  9 17:00:02 2017 GMT

C:\OpenSSL\bin>
Hi Marcelo,

For clarity, the "response verify failure" message is being emitted by OpenSSL because OpenSSL is failing to build a trust chain locally to validate the response. It is not being emitted by your OCSP service, which is, indeed, completely bamboozled.

You may get a clean response from OpenSSL by by fetching a Visa eCommerce Root certificate from e.g. https://crt.sh/?d=896972, concatenating it with the Visa eCommerce Issuing CA certificate (in my example, 12629307.crt), and passing it to openssl, like:

$ cat 12629307.crt 896972.crt > combined
$ openssl ocsp -issuer 12629307.crt -serial 1234 -url http://ocsp.visa.com/ocsp -no_nonce -CAfile combined
Response verify OK
1234: good
	This Update: Oct  2 17:00:00 2017 GMT
	Next Update: Oct 10 17:00:01 2017 GMT


I expect you will find this is not a fault with the openssl client.
As Tim already mentioned, the error you're seeing is a local validation error from OpenSSL and is not germane to the issue.

Frankly, the responses we've received to date do not demonstrate the technical acumen that is required to operate a CA with public trust, and the lackadaisical manner in which VISA has approached responding to this (and other) issues is deeply concerning.
The responses to date have to be vetted and require a level of scrutiny and approval prior to posting on any public forum.  The OCSP issue identified is actively being researched by our engineers and with the Vendor of our OCSP service.  We are also pursuing a replacement, as alternative providers are being validated and tested.  If the current vendor is unable to support the required fix, or provide an acceptable explanation, a replacement solution will be introduced and implemented. Visa has not approached this in a lackadaisical manner, but additionally has had to focus on the impact to a complex infrastructure and systems that changes could introduce. Visa is committed to partnering with Mozilla and the CA/Browser forum with their desire to better secure browsers, users, and the infrastructure we all use.
We reached a conclusion that our current vendor is unable to provide us a workable solution that supports our desire to successfully mitigate this issue.  We are  moving our OCSP infrastructure to a new platform that meets the requirements of the Mozilla Root Program and CA/Browser Forum Baseline Requirements.  We are currently identifying risk and a implementation plan, and expect to complete the transition shortly.  We will provide a delivery date once we have finalized the transition plan.
(In reply to Jason Crawford from comment #8)
> We reached a conclusion that our current vendor is unable to provide us a
> workable solution that supports our desire to successfully mitigate this
> issue.  We are  moving our OCSP infrastructure to a new platform that meets
> the requirements of the Mozilla Root Program and CA/Browser Forum Baseline
> Requirements.  We are currently identifying risk and a implementation plan,
> and expect to complete the transition shortly.  We will provide a delivery
> date once we have finalized the transition plan.

While it's encouraging to see that Visa recognizes the need to transition its OCSP infrastructure, there's still some concerning issues at play here.

1) The response status code requirement was widely discussed, given the roll it played in Diginotar. As noted in Comment #0, this has been a requirement since 2013. Why did Visa not detect or implement these changes, and not discover its vendor was not capable of providing this until 3 days ago (Comment #8)?
2) Visa's issues in testing (Comment #2) don't align with the responses in Comment #7. That is, the matter is not just the timeliness (which seems to be what Comment #7 refers to), but the degree of understanding of the underlying issue - again, widely discussed within the industry.
3) This doesn't provide any sort of information into the post-mortem response process - understanding what procedures Visa already had in place to measure and assess compliance, what steps it took (in 2013) to ensure compliance, why those steps failed, and how those steps are being remediated. Concretely, without understanding these issues, it's unclear that the solution proposed in Comment #8 will meaningfully address the issues - the community has no knowledge that Visa's future selection of a vendor will be able to ensure compliance, nor that its compliance staff will review changes to the BRs, nor that its incident response team will be able to suitably respond to incidents.

These are reasons that the discussion on https://bugzilla.mozilla.org/show_bug.cgi?id=1391087#c19 become all the more pertinent, and information that Visa can share to this specific incident can significantly help or hinder those discussions.
Flags: needinfo?(masilva)
(In reply to Ryan Sleevi from comment #9)
> (In reply to Jason Crawford from comment #8)
> > We reached a conclusion that our current vendor is unable to provide us a
> > workable solution that supports our desire to successfully mitigate this
> > issue.  We are  moving our OCSP infrastructure to a new platform that meets
> > the requirements of the Mozilla Root Program and CA/Browser Forum Baseline
> > Requirements.  We are currently identifying risk and a implementation plan,
> > and expect to complete the transition shortly.  We will provide a delivery
> > date once we have finalized the transition plan.
> 
> While it's encouraging to see that Visa recognizes the need to transition
> its OCSP infrastructure, there's still some concerning issues at play here.
> 
> 1) The response status code requirement was widely discussed, given the roll
> it played in Diginotar. As noted in Comment #0, this has been a requirement
> since 2013. Why did Visa not detect or implement these changes, and not
> discover its vendor was not capable of providing this until 3 days ago
> (Comment #8)?

The original implementation of our OCSP was conducted and implemented under the assumption that the solution provided met the necessary requirements (2014), unfortunately I cannot assume what the engineer at that time concluded.  As we have focused on aligning our PKI offering with Baseline requirements, our legacy solutions are undergoing re-evaluation and testing. With the decision to move to a new solution, this is intended to include the OCSP solution as well. The new dedicated technical staff supporting the PKI solution is also focused on Mozilla, BR and industry requirements as part of the long term strategy. 


> 2) Visa's issues in testing (Comment #2) don't align with the responses in
> Comment #7. That is, the matter is not just the timeliness (which seems to
> be what Comment #7 refers to), but the degree of understanding of the
> underlying issue - again, widely discussed within the industry.

We have the upmost confidence that the staff supporting our systems today have a deep understanding of the underlying issue. Having inherited the infrastructure, there was an assumption that existing services were complaint as the focus was on moving to newer technology. The gap Mozilla identified would have been replaced with the new solution, in which all compliance requirements would be tested, along with the annual audit, but has now been accelerated to mitigate the issue.

> 3) This doesn't provide any sort of information into the post-mortem
> response process - understanding what procedures Visa already had in place
> to measure and assess compliance, what steps it took (in 2013) to ensure
> compliance, why those steps failed, and how those steps are being
> remediated. Concretely, without understanding these issues, it's unclear
> that the solution proposed in Comment #8 will meaningfully address the
> issues - the community has no knowledge that Visa's future selection of a
> vendor will be able to ensure compliance, nor that its compliance staff will
> review changes to the BRs, nor that its incident response team will be able
> to suitably respond to incidents.

> These are reasons that the discussion on
> https://bugzilla.mozilla.org/show_bug.cgi?id=1391087#c19 become all the more
> pertinent, and information that Visa can share to this specific incident can
> significantly help or hinder those discussions.

We will provide post mortem details along with our targeted delivery date once we have finalized the transition. Where it will be difficult to provide details for the legacy systems, the PKI teams goal to align to and stay in alignment with baseline requirements, as well as dedicated SMEs with Vendor and compliance requirements in scope will ensure the continued testing and validation of compliance.
(In reply to Jason Crawford from comment #10)

> We will provide post mortem details along with our targeted delivery date
> once we have finalized the transition. Where it will be difficult to provide
> details for the legacy systems, the PKI teams goal to align to and stay in
> alignment with baseline requirements, as well as dedicated SMEs with Vendor
> and compliance requirements in scope will ensure the continued testing and
> validation of compliance.

It has now been a month and a half since this comment and three months have elapsed since the issue was reported. The "Visa eCommerce Issuing CA" OCSP responder is still not in compliance with the Baseline Requirements. When will this be fixed?
(In reply to Jonathan Rudenberg from comment #11)
> (In reply to Jason Crawford from comment #10)
> 
> > We will provide post mortem details along with our targeted delivery date
> > once we have finalized the transition. Where it will be difficult to provide
> > details for the legacy systems, the PKI teams goal to align to and stay in
> > alignment with baseline requirements, as well as dedicated SMEs with Vendor
> > and compliance requirements in scope will ensure the continued testing and
> > validation of compliance.
> 
> It has now been a month and a half since this comment and three months have
> elapsed since the issue was reported. The "Visa eCommerce Issuing CA" OCSP
> responder is still not in compliance with the Baseline Requirements. When
> will this be fixed?

The vendor solution we had previously been testing failed to meet our requirements.  We have developed an in house solution that will meet our internal requirements and resolve this issue.  In order to avoid any potential impact to services, our current timeline for deploying the solution into our production environment is January.  At this current moment, we are continuing to test and validate the solution in our non-production environment in preparation for the production deployment after the holidays.
The internal solution that was deployed in our non-production environment has been tested and validated successfully.  We remain on schedule for our production deployment.
This issue has been resolved as our new solution has been successfully deployed and validated. The Visa OCSP Service is responding properly to all validation requests.
(In reply to Jason Crawford from comment #14)
> This issue has been resolved as our new solution has been successfully
> deployed and validated. The Visa OCSP Service is responding properly to all
> validation requests.

Great news!

When will we see the post-mortem details that you promised in comment 10? I hope you realize that comment 12 does nothing to address the underlying systemic cause of this issue and what Visa has done or will do to prevent a similar problem from occurring in the future.
Changing QA contact per https://bugzilla.mozilla.org/show_bug.cgi?id=1438254
QA Contact: gerv → wthayer
(In reply to Wayne Thayer [:wayne] from comment #15)
> (In reply to Jason Crawford from comment #14)
> > This issue has been resolved as our new solution has been successfully
> > deployed and validated. The Visa OCSP Service is responding properly to all
> > validation requests.
> 
> Great news!
> 
> When will we see the post-mortem details that you promised in comment 10? I
> hope you realize that comment 12 does nothing to address the underlying
> systemic cause of this issue and what Visa has done or will do to prevent a
> similar problem from occurring in the future.

Once notified by Mozilla, our investigation confirmed Mozilla’s finding and identified shortcomings with our vendor’s documentation and procedures, both from architecture and operational perspectives.   After further discussions with the vendor, we decided that the product in place wasn't adequate for the integration with our Certificate Management software,  and decided to replace the product.   Our initial implementation of OCSP services was focused on OCSP functionality, prior to our efforts in becoming BR compliant.  The role for validating compliance with BR has been re-assigned a dedicated resources, and the original resources has been removed.  The realigned role will now work directly with our technical engineers on technical validations and compliance requirements.  This measures put into place to ensure that BR and Root Program compliance would be maintained moving forward.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [ocsp-failure]
You need to log in before you can comment on or make changes to this bug.