Closed Bug 1485851 Opened Last year Closed Last year

Visa: Qualified Audit Statements

Categories

(NSS :: CA Certificate Compliance, task)

task
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: wayne, Assigned: pkipolicy)

Details

(Whiteboard: [ca-compliance])

Visa has supplied the following audit statements in bug 1485777:

WTCA:https://bug1485777.bmoattachments.org/attachment.cgi?id=9003607
WTBR: https://bug1485777.bmoattachments.org/attachment.cgi?id=9003608

The WTCA report contains one qualification, and the WTBR report contains four qualifications including two (the first two) that document misissuance.

The management response to the second WTBR qualification indicates that there is no record of domain validation on 5 certificates. Have these certificates been revalidated or revoked? Has Visa identified all certificates for which they have no domain validation records and remediated them?

Please provide an incident report for these issues, as described here:
https://wiki.mozilla.org/CA/Responding_To_A_Misissuance#Incident_Report
The incident report should be posted to the mozilla.dev.security.policy forum and added to this bug.

The required scope of the incident report is:
1. the misissuances described in the first two qualifications in the WTBR report. (if any of these were the subject of a previous bug, please reference the bug)
2. Visa's plan to remediate all of the issues listed in these audit reports (not just the misissuances)
3. An explanation of why these audit reports were delivered almost two months late and how Visa intends to prevent that from happening again.
Hello Wayne,

Sorry for the delay. We have restored access to our historical tracking system and are reviewing the items, listed in the post.
We are preparing a detailed response and we will respond shortly. 

Thanks

Anup Chauhan
Hello,
The following a response to the https://bugzilla.mozilla.org/show_bug.cgi?id=1485851,  
Note that subscribers of Visa’s CA are organizations, not individuals, that have a financial, legal and contractual relationship with Visa.   Before certificates are issued to any Visa subscriber, the organization has undergone risk determinations, financial and legal validation, as well as having contractual legal documents in place.  The partnership is well established prior to certificates being requested.  Visa’s enrollment and certificate submission is managed through client management teams, and all certificate request are made with Visa staff involved.  There is no external access to the certificate management system. Certificate request are part of an onboarding sequence.  Certificate requests are validated via client portal, validating need and active status.  Visa’s inclusion in root programs, provides a different use case than your standard web use case.  The team has worked to migrate from WTCA to a more web-focused Baseline Requirement, as we don’t feel Baseline has optimized for non-web requirements, we agree with the vision of increasing the security profile of internet usage and will continue to partner with those working to do so.  All issues identified to date have been addressed and resolved.
Issues listed in WTC and WTBR

WTCA issues: -
6.6: For 5 of the 5 certificate revocations selected, we were unable to obtain evidence of a revocation request in writing or authentication of the revocation request in compliance with the CP and CPS. 
Response to the issue: 
1)	How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting. Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
a)	The issue was brought to our attention by our auditors during the annual WTCA/BR audit that started April 2018

2)	A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
a)	Timeline of certificates revocation was performed on these dates
i)	Certificate revoked on 6/20/2017
ii)	Certificate revoked on 08/01/2017 
iii)	Certificate revoked on 8/3/2017
iv)	Certificate revoked on 10/18/2017Certificate revoked on 11/02/2017

3)	Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
a)	N/A this is not related to certificate issuance

4)	A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
a)	The certificates revocation was performed on these dates
i)	Certificate revoked on 6/20/2017
ii)	Certificate revoked on 08/01/2017 
iii)	Certificate revoked on 8/3/2017
iv)	Certificate revoked on 10/18/2017
v)	Certificate revoked on 11/02/2017

5)	Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
a)	Given the irreversible impact of revoking certificates, and possible service and financial impact, certificate revocation was done once the client had successfully migrated to new certificates and services were running.  The certificate revocation request was not requested due to any compromise, and were replacement for expiring certificates, or re-issuance of a certificate.  

6)	List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
a)	The Client support team and vettors have been advices to include communication from the clients in the tickets as part of validation of the requests
b)	Certificate revocations request are now escalated to the team to perform the validation, revocation.  A proper workflow for requesting certificate revocations has now been established and revocation will be requested when service replacements are completed and not as part of the request.

WTBR issues: -
2-4.1 : For 5 of the 45 certificate issuance requests selected, we were unable to obtain domain validation evidence in accordance with the Baseline Requirements.
Response to the issues
1)	How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
a)	The issue was brought to our attention by our auditors during the annual WTCA/BR audit that started April 2018

2)	A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
a)	The issue was not related to any bug.  The domain validation letters were received at the time the certificates were requested (April 2017 till Dec 2017). The vettors had performed the validation while issuing the certificate. The actual evidence of the domain validation letters was un-available to be provide to auditors for our WTCA/BR audit in April 2018. This was due to the fact that the ticketing system was decommissioned and historical data was unavailable. We have since restored access and retrieved the domain validation letters and forwarded to the auditors

3)	Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
a)	There was no need to stop issuing certificates because we have a new ticketing system and all domain validation evidences are recorded in it. 

4)	A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
a)	There are 5 certificates (1 of the 5 certificate belongs to VISA’s own domain)
i)	Certificate 1 was used on 12/4/2017
ii)	Certificate 2 was issued on 26/07/2017
iii)	Certificate 3 was issued on 12/10/2017 
iv)	Certificate 4 was issued on 19/10/2017 
v)	Certificate 5 was issued on 28/12/2017

5)	Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
a)	The issue was in providing documentary evidence of domain validation to the auditors because old ticketing system was decommissioned in end of December 2017. Hence our inability to access historical data when oud audit started in April 2017. Our new ticketing system is functional starting January 2018 and records all requests, vetting process and evidence  for certificates stating January 2018 

6)	List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
a)	This item was resolved , We already have a new companywide ticketing system implemented in January 2018 and are confident that the process/steps for domain validation is being performed and recorded in it. We do not foresee this issue arising in the future. 

2-5.10: The OCSP responder was configured to allow a response of "good" for a certificate that was not issued by the CA, until February 13, 2018.
Response to the issue: The issue was resolved Bug# 1398261 
1)	How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
a)	The problem was identified by Bugzilla report 1398261 that in September 2017 it was reported that the Visa OCSP was responding with "good" to certificates that were not issued by them

2)	A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
a)	09/2017 – Bug reported 
b)	10/2017 – unacceptable results from Vendor , new solution developed
c)	11/2017 – New solution implementation, removing Non-production infrastructure supporting OCSP
d)	12/2017 – New Production OCSP infrastructure staged, continued testing in non-production
e)	01/2018 – New OCSP implemented post PEAK Holiday Season, dry run and burn in period performed
f)	02/2018 – Published to Mozilla new compliant OCSP was put into place.

3)	Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
a)	N/A not a certificate issuance issue

4)	A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued. 
a)	N/A not a certificate issuance issue

5)	Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
a)	The requirement was a BR requirement that Visa has adopted for compliance, following maintaining compliance with BR is a priority for Visa.  Annual compliance audits will determine areas of improvement as well as continuing to be active in CA/Browser discussions.

6)	List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
a)	This item was resolved, Visa has invested in modernization a PKI infrastructure to disallow products unable of meeting compliance requirements.
  
2-7.3: We were unable to obtain evidence that firewall and router activity logs were being retained for the required seven year period.
Response to the issue:
1)	How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date
a)	The issue was brought to our attention by our auditors during the annual WTCA/BR audit that started April 2018

2)	A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
a)	07/2018 – BR Audit reported issue
b)	08/2018 – Updates requested to Document Retention Policy

3)	Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
a)	N/A not a certificate issuance issue

4)	A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
a)	N/A not a certificate issuance issue

5)	Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
a)	Visa PKI systems have a 7 year retention for logs and systems audit records, the firewalls are managed by our corporate firewall team and policies adhere to Visa Key Controls that specifically identify that logs cannot be stored for over 12 months.  This is a miss-alignment of BR requirements and Visa documents retention policies.

6)	List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
a)	This item was resolved, We have updated the Firewall log retention for the PKI systems to be updated and reflected in Visa’s global Document Retention policy.  This update has been completed and the issue now resolved.  The team is also validating existing administrative requirements and confirming alignment with BR and Visa policy.

From Bug (1485851)
An explanation of why these audit reports were delivered almost two months late and how Visa intends to prevent that from happening again.:
Visa migration to a new Global ticketing system had its unintended consequence on the PKI and Vetting teams.    The team struggled to regain access to tickets that had been archived in 2017. The delay of the report was caused by Visa’s attempt to produce vetting evidence contained in an archived Ticketing system.  The team worked to restore access to the Global ticketing system Visa had migrated away from at the end of 2017.  After a period of waiting, and queries for the Audit Report, it was decided to move forward with the audit and not delay the audit any longer. Next year’s audit will overlap this time frame but will include the period from which the original audit date is required. 

2-2.25: For 1 of the 45 certificate issuances selected, we noted the extended key usage field was not present in accordance with the Baseline Requirements.
Response to the issue:
A detailed response will follow on this issue
Hello,

2-2.25: For 1 of the 45 certificate issuances selected, we noted the extended key usage field was not present in accordance with the Baseline Requirements.

Response to the issue:

Visa has researched this certificate and determined it was issued using a legacy profile that was migrated for use of a closed visa service.  The certificate identified is not used as a Web server certificate.  The certificate internet facing certificate used by the  fqdn issued by commercial ca. We are researching the working with the subscriber and determining the best replacement opportunity with the end-entity, that will not impact production services and cause any disruption  to  the client.  Visa plans to replace the certificate with a corrected certificate.  We can provide a more detailed timeline once we formalize the process with the end entity.  The profile had already been corrected prior to the audit, and the possibility of this re-occurrence has been eliminated.

The certificate was issued ( ‎Wednesday, ‎April ‎12, ‎2017 7:27:04 AM) 
The profile was corrected (‎Friday, ‎April ‎12, ‎2019 7:27:04 AM) 

Thank you
Could you clarify the date the profile was corrected, please?
Flags: needinfo?(pkipolicy)
Hello,

The profile/template to include the required EKU was corrected as of August 23, 2017
Ref; https://bugzilla.mozilla.org/show_bug.cgi?id=1391087#c9 

Thank you.
Flags: needinfo?(pkipolicy)
This root is being removed from the program via bug 1493822.
Status: NEW → RESOLVED
Closed: Last year
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.