1391087 - Visa: Non-BR-Compliant Certificate Issuance

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Kathleen Wilson]

The following problems have been found in certificates issued by your CA, and reported in the mozilla.dev.security.policy forum. Direct links to those discussions are provided for your convenience.

To continue inclusion of your CA’s root certificates in Mozilla’s Root Store, you must respond in this bug to provide the following information:
1) How your CA first became aware of the problems listed below (e.g. via a Problem Report, via the discussion in mozilla.dev.security.policy, or via this Bugzilla Bug), and the date.
2) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates with the problems listed below.
3) Complete list of certificates that your CA finds with each of the listed issues during the remediation process. The recommended way to handle this is to ensure each certificate is logged to CT and then attach a CSV file/spreadsheet of the fingerprints or crt.sh IDs, with one list per distinct problem.
4) Summary of the problematic certificates. For each problem listed below: number of certs, date first and last certs with that problem were issued.
5) Explanation about how and why the mistakes were made, and not caught and fixed earlier.
6) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
7) Regular updates to confirm when those steps have been completed.

Note Section 4.9.1.1 of the CA/Browser Forum’s Baseline Requirements, which states:
“The CA SHALL revoke a Certificate within 24 hours if one or more of the following occurs: …
9. The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement; 
10. The CA determines that any of the information appearing in the Certificate is inaccurate or misleading; …
14. Revocation is required by the CA’s Certificate Policy and/or Certification Practice Statement; or 
15. The technical content or format of the Certificate presents an unacceptable risk to Application Software Suppliers or Relying Parties (e.g. the CA/Browser Forum might determine that a deprecated cryptographic/signature algorithm or key size presents an unacceptable risk and that such Certificates should be revoked and replaced by CAs within a given period of time).

However, it is not our intent to introduce additional problems by forcing the immediate revocation of certificates that are not BR compliant when they do not pose an urgent security concern. Therefore, we request that your CA perform careful analysis of the situation. If there is justification to not revoke the problematic certificates, then explain those reasons and provide a timeline for when the bulks of the certificates will expire or be revoked/replaced. 

We expect that your forthcoming audit statements will indicate the findings of these problems. If your CA will not be revoking the certificates within 24 hours in accordance with the BRs, then that will also need to be listed as a finding in your CA’s BR audit statement.

We expect that your CA will work with your auditor (and supervisory body, as appropriate) and the Root Store(s) that your CA participates in to ensure your analysis of the risk and plan of remediation is acceptable. If your CA will not be revoking the problematic certificates as required by the BRs, then we recommend that you also contact the other root programs that your CA participates in to acknowledge this non-compliance and discuss what expectations their Root Programs have with respect to these certificates.


The problems reported for your CA in the mozilla.dev.security.policy forum are as follows:

** Failure to respond within 24 hours after Problem Report submitted
https://groups.google.com/d/msg/mozilla.dev.security.policy/PrsDfS8AMEk/w2AMK81jAQAJ
The problems were reported via your CA’s Problem Reporting Mechanism as listed here:
https://ccadb-public.secure.force.com/mozilla/CAInformationReport
Therefore, if this is the first time you have received notice of the problem(s) listed below, please review and fix your CA’s Problem Reporting Mechanism to ensure that it will work the next time someone reports a problem like this.

** Invalid dnsNames (e.g. invalid characters, internal names, and wildcards in the wrong position)
https://groups.google.com/d/msg/mozilla.dev.security.policy/CfyeeybBz9c/lmmUT4x2CAAJ
https://groups.google.com/d/msg/mozilla.dev.security.policy/D0poUHqiYMw/Pf5p0kB7CAAJ

Comment 1

[Marcelo B. Silva]

Hi Kathleen,
We have identified the issue, and have validated that both certificates were issued for our internal Corporate IT department.  We are currently working to remediate this issue with the assistance from our compliance lead to ensure we maintain compliance moving forward.
Marcelo

Comment 2

[Jonathan Rudenberg]

I emailed a problem report about these certificates, can you please investigate and explain why it was not received?

Here are the relevant message headers:

  From: Jonathan Rudenberg <[e-mail]>
  Subject: Misissuance - invalid dnsNames
  Message-Id: <[e-mail]>
  Date: Sun, 13 Aug 2017 00:35:47 -0400
  To: [e-mail]

Comment 3

[Ryan Sleevi]

Marcelo,

Your reply looks like it's incomplete and doesn't address the items requested. Could you ensure you provide the details requested (e.g. to each of the specific items highlighted)? You can see https://bugzilla.mozilla.org/show_bug.cgi?id=1391054#c2 as an example.

Comment 4

[Marcelo B. Silva]

Hi Ryan, following below the answers for the questions:

1) How your CA first became aware of the problems listed below (e.g. via a Problem Report, via the discussion in mozilla.dev.security.policy, or via this Bugzilla Bug), and the date.

 - Unqualified domain names in SAN reported to Visa via the Mozilla Dev Security Policy Forum on August 16th, 2017   
 - Inclusion of serverAuth key purpose in extended key usage via the Mozilla Dev Security Policy Forum on August 16th, 2017.

2) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates with the problems listed below.
We have ceased issuing certificates with the abovementioned problems.


3) Complete list of certificates that your CA finds with each of the listed issues during the remediation process. The recommended way to handle this is to ensure each certificate is logged to CT and then attach a CSV file/spreadsheet of the fingerprints or crt.sh IDs, with one list per distinct problem.
We are currently researching to identify certificates issued with the abovementioned problems.

4) Summary of the problematic certificates. For each problem listed below: number of certs, date first and last certs with that problem were issued.

We are currently researching to identify certificates issued with the abovementioned problems.

5) Explanation about how and why the mistakes were made, and not caught and fixed earlier.

The certificates in question were issued as part of our legacy process which was prior to our alignment with the CA/B Forum Baseline Requirements.  The two legacy certificates listed in bugzilla 1391087 have been replaced with certificates with FQDN’s that include the serverAuth key purpose in extended key usage, which aligns with the CA/B Forum Baseline Requirements.  We are continuing to research if there are any other certificates with the abovementioned problems.  

6) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
 1.Identify certificates with problems as detailed in bugzilla 1391087.
 2.Inform Certificate owner who will enroll for new certificate(s), with acceptable FQDN’s and certificate attributes.
 3.Certificate owner will replace previously issued certificate(s), with the newly issued certificates that align with the 
   CA/Browser Forum Baseline Requirements.
 4.Revoke legacy certificates once certificate owner has confirmed successful installation of replacement certificates.

7) Regular updates to confirm when those steps have been completed.
Updates to be provided once research has been completed.

Thanks you,

Comment 5

[Jonathan Rudenberg]

Hi Marcelo,

Can you please provide your expected timeline for revocation of the misissued certificates and reasoning for it? Based on when you learned about this issue, 24 hours has already passed. What was the analysis and conclusion that led you to decide not to revoke the certificates immediately?

Comment 6

[Ryan Sleevi]

(In reply to Marcelo B. Silva from comment #4)
> 7) Regular updates to confirm when those steps have been completed.
> Updates to be provided once research has been completed.

It's now been 4 days since this reply was posted. Can you please indicate when you expect to provide a further update? Many CAs have been able to complete this research within 24 hours, so it would be useful if, as part of your post-mortem, the timeline of the investigation, and the challenges encountered, was also shared.

Comment 7

[Marcelo B. Silva]

(In reply to Ryan Sleevi from comment #6)
> (In reply to Marcelo B. Silva from comment #4)
> > 7) Regular updates to confirm when those steps have been completed.
> > Updates to be provided once research has been completed.
> 
> It's now been 4 days since this reply was posted. Can you please indicate
> when you expect to provide a further update? Many CAs have been able to
> complete this research within 24 hours, so it would be useful if, as part of
> your post-mortem, the timeline of the investigation, and the challenges
> encountered, was also shared.

Hi Ryan,
We have replaced both certificates within 24 hours, and they are no longer active. 
One of them (CN=mimdmoce.visa.com, S/N: 46534D43101303058C52F9A7EA59678A)was already revoked on 08/18/2017.
For the second one, we are waiting some further validation so we can avoid any impact on our systems in other regions. It will be revoked by 08/29/2017.
Thanks,

Comment 8

[Jonathan Rudenberg]

(In reply to Marcelo B. Silva from comment #7)
> For the second one, we are waiting some further validation so we can avoid
> any impact on our systems in other regions. It will be revoked by 08/29/2017.

I don't believe this is justified given the details you have provided. The certificate in question (https://crt.sh/?id=42651061&opt=cablint) contains Internal Names and should be revoked immediately, as it is a security risk. Bug 1390974, comment 3 has details about this risk.

Additionally, can you please explain what the status of your investigation is and when a substantive update with the results will be posted?

Comment 9

[Ryan Sleevi]

(In reply to Marcelo B. Silva from comment #7)
> Hi Ryan,
> We have replaced both certificates within 24 hours, and they are no longer
> active. 
> One of them (CN=mimdmoce.visa.com, S/N: 46534D43101303058C52F9A7EA59678A)was
> already revoked on 08/18/2017.
> For the second one, we are waiting some further validation so we can avoid
> any impact on our systems in other regions. It will be revoked by 08/29/2017.
> Thanks,

Marcelo,

Please carefully review Comment #0 for all the things requested of Visa. Please carefully review each question that has been asked, and ensure you provide complete and thorough answers. The continued trust of your root is predicated on how effectively, comprehensively, and transparently respond to issues once highlighted.

I hope you can appreciate the gravity of this request and come back with a much more detailed response. The best way to ensure you've met every obligation/request is to use the "Reply" button in the top-right next to each message. That will copy the text into the comment bubble, and allow you to respond inline, point-by-point, to the requested items, from Kathleen, myself, and from those who have further commented.

Thank you for your complete and thorough attention to this.

Comment 10

[Ryan Sleevi]

Marcelo,

It's nearly been one week since the last request, and 10 days since you last provided a substantive update. When can we expect the next update?

Please review Comment #9, Comment #8, and Comment #5.

Comment 11

[Marcelo B. Silva]

(In reply to Ryan Sleevi from comment #9)
> (In reply to Marcelo B. Silva from comment #7)
> > Hi Ryan,
> > We have replaced both certificates within 24 hours, and they are no longer
> > active. 
> > One of them (CN=mimdmoce.visa.com, S/N: 46534D43101303058C52F9A7EA59678A)was
> > already revoked on 08/18/2017.
> > For the second one, we are waiting some further validation so we can avoid
> > any impact on our systems in other regions. It will be revoked by 08/29/2017.
> > Thanks,
> 
> Marcelo,
> 
> Please carefully review Comment #0 for all the things requested of Visa.
> Please carefully review each question that has been asked, and ensure you
> provide complete and thorough answers. The continued trust of your root is
> predicated on how effectively, comprehensively, and transparently respond to
> issues once highlighted.
> 
> I hope you can appreciate the gravity of this request and come back with a
> much more detailed response. The best way to ensure you've met every
> obligation/request is to use the "Reply" button in the top-right next to
> each message. That will copy the text into the comment bubble, and allow you
> to respond inline, point-by-point, to the requested items, from Kathleen,
> myself, and from those who have further commented.
> 
> Thank you for your complete and thorough attention to this.

Hi Ryan, following below more detailed answers for the initial questions:

1) How your CA first became aware of the problems listed below (e.g. via a Problem Report, via the discussion in mozilla.dev.security.policy, or via this Bugzilla Bug), and the date.

- Unqualified domain names in SAN reported to Visa via the Mozilla Dev Security Policy Forum on August 16th, 2017   
- Inclusion of serverAuth key purpose in extended key usage via the Mozilla Dev Security Policy Forum on August 16th, 2017.

2) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates with the problems listed below.

We have ceased issuing certificates with the abovementioned problems.

- The certificates that were initial reported have been remediated as of August 23, 2017. More details in the answer for the question #6 below.
- We are currently discussing internally our approach for remediating other legacy certificates missing the serverAuth OID.

3) Complete list of certificates that your CA finds with each of the listed issues during the remediation process. The recommended way to handle this is to ensure each certificate is logged to CT and then attach a CSV file/spreadsheet of the fingerprints or crt.sh IDs, with one list per distinct problem.

a)  Unqualified domain names in SAN reported: 
No issue was found after remediating the two findings and further investigation.
b)  Inclusion of serverAuth key purpose in extended key usage:
The complete list of certificates will be provided upon approval by our Legal and Compliance department.  


4) Summary of the problematic certificates. For each problem listed below: number of certs, date first and last certs with that problem were issued.

- The complete list of certificates will be provided upon approval by our Legal and Compliance department.  

5) Explanation about how and why the mistakes were made, and not caught and fixed earlier.

The certificates in question were issued as part of our legacy process which was prior to our alignment with the CA/B Forum Baseline Requirements.  The two legacy certificates listed in bugzilla 1391087 have been replaced with certificates with FQDN’s that include the serverAuth key purpose in extended key usage, which aligns with the CA/B Forum Baseline Requirements.

6) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

a)Identify certificates with problems as detailed in bugzilla 1391087.
b)Inform Certificate owner who will enroll for new certificate(s), with acceptable FQDN’s and certificate attributes.
c) Certificate owner will replace previously issued certificate(s), with the newly issued certificates that align with the CA/Browser Forum Baseline Requirements.
d) Provide replacement certificates to certificate owner.
e) Revoked legacy certificates after successful installation of replacement certificates.
f) Both certificates were replaced and revoked as per below dates:
  • 08/17/2017 – New certificates issued
  • 08/18/2017 – Certificate for CN=mimdmoce.visa.com was revoked (Internal Incident #: 5624015)
  • 08/23/2017 - Certificate for CN=mimdmoce.visa.com was revoked (Internal Incident #: 5636868)
g) We have already updated all certificate templates in the past to include the required EKU, and changed our policy and procedures to not allow any internal hostnames nor internal IP addresses in any TLS certificate issued by our CA.
h) To maintain compliance, we have included the adopted process into our quarterly 3% self-audit checks.  This will ensure that we are, and will remain aligned to the CA/Browser Forum Baseline Requirements.

7) Regular updates to confirm when those steps have been completed.
For now we have remediated the findings indicated in Bugzilla #1391087;
Revoked and replaced both certificates; generated detailed report and investigated this issue further; Found no other TLS certificate with internal hostname/IP address as part of the SAN names nor Common Names; started working with Legal department to release a complete list of certificates missing the serverAuth OIDs.

Comment 12

[Marcelo B. Silva]

(In reply to Jonathan Rudenberg from comment #2)
> I emailed a problem report about these certificates, can you please
> investigate and explain why it was not received?
> 
> Here are the relevant message headers:
> 
>   From: Jonathan Rudenberg <[e-mail]>
>   Subject: Misissuance - invalid dnsNames
>   Message-Id: <[e-mail]>
>   Date: Sun, 13 Aug 2017 00:35:47 -0400
>   To: [e-mail]

Hi Jonathan,
I was able to check it and your email was received properly, but our employee responsible to read and answer all emails had to take a medical leave and that's why you didn't get a reply to your email. Our sincerely apologies for that.
And thank you for reporting us the issue you noticed.
Marcelo

Comment 13

[Marcelo B. Silva]

(In reply to Jonathan Rudenberg from comment #8)
> (In reply to Marcelo B. Silva from comment #7)
> > For the second one, we are waiting some further validation so we can avoid
> > any impact on our systems in other regions. It will be revoked by 08/29/2017.
> 
> I don't believe this is justified given the details you have provided. The
> certificate in question (https://crt.sh/?id=42651061&opt=cablint) contains
> Internal Names and should be revoked immediately, as it is a security risk.
> Bug 1390974, comment 3 has details about this risk.
> 
> Additionally, can you please explain what the status of your investigation
> is and when a substantive update with the results will be posted?

Hi Jonatan,
Both certificates were replaced 10 days ago, on 08/18/2017, and revoked as per below dates:
  • 08/18/2017 – Certificate for CN=mimdmoce.visa.com was revoked (Internal Incident #: 5624015)
  • 08/23/2017 - Certificate for CN=mimdmoce.visa.com was revoked (Internal Incident #: 5636868)
They were installed in Visa internal hosted systems, and without the private key the certificates are useless. 
In case of a key compromise any Visa certificate is revoked within 24 hours, as per Baseline Requirements and Visa CP/CPS.

You can see additional details on the investigation in my reply to Comment #9.

Thanks,
Marcelo

Comment 14

[Ryan Sleevi]

Thanks for responding. I think it's still necessary to provide additional detail.

That is, we can look at the problem from two dimensions: The problem itself, and the systemic issues that allowed the problem to manifest. Your description focuses on the resolution of the problem, but doesn't indicate any systemic changes have been made. As a consequence, it does not help the community feel that your CA has taken steps to reduce the risk of future violations (of any requirement) in the future. That is, one dimension is "Did you fix the bug", but another dimension is "How was the bug introduced, why was it not detected, and what steps are you taking to prevent future bugs"

To understand how you might approach this problem, consider https://groups.google.com/d/msg/mozilla.dev.security.policy/vl5eq0PoJxY/W1D4oZ__BwAJ and how it provided a timeline of events, the steps that were already in place (and with substantial detail), where there were controls missing or mistakes made, details about the steps being taken (e.g. "We fixed the bug" is not sufficient detail to understand), and a holistic, systemic awareness of how the CA is managed and the opportunities for errors.

(In reply to Marcelo B. Silva from comment #11)
> 3) Complete list of certificates that your CA finds with each of the listed
> issues during the remediation process. The recommended way to handle this is
> to ensure each certificate is logged to CT and then attach a CSV
> file/spreadsheet of the fingerprints or crt.sh IDs, with one list per
> distinct problem.
>
> b)  Inclusion of serverAuth key purpose in extended key usage:
> The complete list of certificates will be provided upon approval by our
> Legal and Compliance department.  

When can we expect an update? Please see https://bugzilla.mozilla.org/show_bug.cgi?id=1391074#c12 as an example of how such requests are being handled.
 
> 4) Summary of the problematic certificates. For each problem listed below:
> number of certs, date first and last certs with that problem were issued.
> 
> - The complete list of certificates will be provided upon approval by our
> Legal and Compliance department.  

When can we expect an update? Please see https://bugzilla.mozilla.org/show_bug.cgi?id=1391074#c12 as an example of how such requests are being handled.

> 5) Explanation about how and why the mistakes were made, and not caught and
> fixed earlier.
> 
> The certificates in question were issued as part of our legacy process which
> was prior to our alignment with the CA/B Forum Baseline Requirements.  The
> two legacy certificates listed in bugzilla 1391087 have been replaced with
> certificates with FQDN’s that include the serverAuth key purpose in extended
> key usage, which aligns with the CA/B Forum Baseline Requirements.

https://crt.sh/?id=42651061&opt=cablint was issued in 2016-06-17
https://crt.sh/?id=12797320&opt=cablint was issued in 2016-01-15

The Baseline Requirements have been in force since 2012, and within Mozilla policy since at least 2014.

As part of providing the necessary transparency and detail for public trust, can you please provide a timeline about your transition from your legacy process to your new process, and how your new process provides sufficient mitigations for such issues?

That is, the goal of this reply should be able to help the community understand how Visa has ensured such problems are not possible anymore and that they have a sufficient and in-depth knowledge of how the problems happened (both technically and systemically). This requires substantially more detail - including about the controls you have in place at present - to help provide that assurance.

For example, the Baseline Requirements required that all such certificates were to be revoked on 2016-10-01. Understanding how and why Visa failed to do so is crucial to understanding how Visa will ensure its compliance in the future, and is part of the systemic and holistic analysis of failures.

> h) To maintain compliance, we have included the adopted process into our
> quarterly 3% self-audit checks.  This will ensure that we are, and will
> remain aligned to the CA/Browser Forum Baseline Requirements.

Are you stating this was not already part of your compliance checks, despite having been explicitly required in the Baseline Requirements and with significant fanfare (e.g. https://cabforum.org/internal-names/ ) ? 

Publicly documenting what checks you do perform as your self-audit, for example, can help provide the necessary public assurance.

Understanding why this wasn't detected during your audits is equally important.

Comment 15

[Jason Crawford]

(In reply to Ryan Sleevi from comment #14)

(In reply to Ryan Sleevi from comment #9)

Hi Ryan,

I would like to share with you some details regarding our PKI System and our position within the CA/Browser Forum.  Visa is one of the oldest operating Certificate Authorities and is currently a non-voting member within the CA/Browser Forum.  Visa has been operating a closed PKI system prior to the inception of the Baseline Requirements, which we had a number of legacy processes for the issuance and fulfillment of our certificates to our clients.  Certificates that are issued by Visa public CA’s are issued only to our clients for interconnectivity purposes.  Unlike other CA’s and particularly those that have undergone your Blink Process, our core business is not PKI.  The certificates that were impacted with the noted issues were not issued erroneously to a bad actor(s) nor do we issue certificates to the open public. Due to our unique PKI system, we are not at liberty to divulge with the public our list of impacted clients and their certificates without our Legals' consent.

Regarding BR compliance, we completed our initial BR audit in September of 2016.  Since that time, we have been addressing the observations noted by our external auditors.  This also would encompass any certificate issues that have been publically reported.  Understanding that such changes in adopting a new process will have business impact, it is difficult to provide an accurate timeline of complete compliance as we are required to assess the impact to our client and payment systems to avoid any operational impact.  We are committed to aligning with BR and Mozilla requirements as we have continuously move forward in making the necessary changes .


As stated previously, the two certificates reported have been revoked and replaced with new certificates.  We are diligently working with our impacted clients to remediate all other certificates that may have been effected, without causing major impact to their systems.  We have taken the necessary action both systemically and procedurally to ensure future certificates issued from our closed PKI shall no longer be issued with the previously reported issues.  In addition, we have altered our 3% (random) self-audit process to focus on the noted issues with prejudice.   

We hope that the brevity of this response provides yourself and the public transparency and clarity to the questions you have asked regarding this reported issue and our previous responses. 


Best regards,

Jason



> Thanks for responding. I think it's still necessary to provide additional
> detail.
> 
> That is, we can look at the problem from two dimensions: The problem itself,
> and the systemic issues that allowed the problem to manifest. Your
> description focuses on the resolution of the problem, but doesn't indicate
> any systemic changes have been made. As a consequence, it does not help the
> community feel that your CA has taken steps to reduce the risk of future
> violations (of any requirement) in the future. That is, one dimension is
> "Did you fix the bug", but another dimension is "How was the bug introduced,
> why was it not detected, and what steps are you taking to prevent future
> bugs"
> 
> To understand how you might approach this problem, consider
> https://groups.google.com/d/msg/mozilla.dev.security.policy/vl5eq0PoJxY/
> W1D4oZ__BwAJ and how it provided a timeline of events, the steps that were
> already in place (and with substantial detail), where there were controls
> missing or mistakes made, details about the steps being taken (e.g. "We
> fixed the bug" is not sufficient detail to understand), and a holistic,
> systemic awareness of how the CA is managed and the opportunities for errors.
> 
> (In reply to Marcelo B. Silva from comment #11)
> > 3) Complete list of certificates that your CA finds with each of the listed
> > issues during the remediation process. The recommended way to handle this is
> > to ensure each certificate is logged to CT and then attach a CSV
> > file/spreadsheet of the fingerprints or crt.sh IDs, with one list per
> > distinct problem.
> >
> > b)  Inclusion of serverAuth key purpose in extended key usage:
> > The complete list of certificates will be provided upon approval by our
> > Legal and Compliance department.  
> 
> When can we expect an update? Please see
> https://bugzilla.mozilla.org/show_bug.cgi?id=1391074#c12 as an example of
> how such requests are being handled.
>  
> > 4) Summary of the problematic certificates. For each problem listed below:
> > number of certs, date first and last certs with that problem were issued.
> > 
> > - The complete list of certificates will be provided upon approval by our
> > Legal and Compliance department.  
> 
> When can we expect an update? Please see
> https://bugzilla.mozilla.org/show_bug.cgi?id=1391074#c12 as an example of
> how such requests are being handled.
> 
> > 5) Explanation about how and why the mistakes were made, and not caught and
> > fixed earlier.
> > 
> > The certificates in question were issued as part of our legacy process which
> > was prior to our alignment with the CA/B Forum Baseline Requirements.  The
> > two legacy certificates listed in bugzilla 1391087 have been replaced with
> > certificates with FQDN’s that include the serverAuth key purpose in extended
> > key usage, which aligns with the CA/B Forum Baseline Requirements.
> 
> https://crt.sh/?id=42651061&opt=cablint was issued in 2016-06-17
> https://crt.sh/?id=12797320&opt=cablint was issued in 2016-01-15
> 
> The Baseline Requirements have been in force since 2012, and within Mozilla
> policy since at least 2014.
> 
> As part of providing the necessary transparency and detail for public trust,
> can you please provide a timeline about your transition from your legacy
> process to your new process, and how your new process provides sufficient
> mitigations for such issues?
> 
> That is, the goal of this reply should be able to help the community
> understand how Visa has ensured such problems are not possible anymore and
> that they have a sufficient and in-depth knowledge of how the problems
> happened (both technically and systemically). This requires substantially
> more detail - including about the controls you have in place at present - to
> help provide that assurance.
> 
> For example, the Baseline Requirements required that all such certificates
> were to be revoked on 2016-10-01. Understanding how and why Visa failed to
> do so is crucial to understanding how Visa will ensure its compliance in the
> future, and is part of the systemic and holistic analysis of failures.
> 
> > h) To maintain compliance, we have included the adopted process into our
> > quarterly 3% self-audit checks.  This will ensure that we are, and will
> > remain aligned to the CA/Browser Forum Baseline Requirements.
> 
> Are you stating this was not already part of your compliance checks, despite
> having been explicitly required in the Baseline Requirements and with
> significant fanfare (e.g. https://cabforum.org/internal-names/ ) ? 
> 
> Publicly documenting what checks you do perform as your self-audit, for
> example, can help provide the necessary public assurance.
> 
> Understanding why this wasn't detected during your audits is equally
> important.

Comment 16

[Jonathan Rudenberg]

(In reply to Jason Crawford from comment #15)
> Due to our unique PKI system, we are not at liberty to divulge
> with the public our list of impacted clients and their certificates without
> our Legals' consent.

In comment #11 (17 days ago) Marcelo said:

> - The complete list of certificates will be provided upon approval by our Legal and Compliance department.

When will these certificates be provided?

> Regarding BR compliance, we completed our initial BR audit in September of
> 2016.  Since that time, we have been addressing the observations noted by
> our external auditors.  This also would encompass any certificate issues
> that have been publically reported.

What other issues that have not yet been publicly reported have you discovered?

> Understanding that such changes in
> adopting a new process will have business impact, it is difficult to provide
> an accurate timeline of complete compliance as we are required to assess the
> impact to our client and payment systems to avoid any operational impact.

In what areas are you currently out of compliance with Mozilla's Root Store Policy and the Baseline Requirements?
 
> We hope that the brevity of this response provides yourself and the public
> transparency and clarity to the questions you have asked regarding this
> reported issue and our previous responses. 

I believe this response is missing answers to multiple questions asked in comment #14. It would be useful if you could reply to that comment with inline responses to each question so that the context is provided and you can ensure that no answers are missed.

Comment 17

[Gervase Markham [:gerv]]

(In reply to Marcelo B. Silva from comment #12)
> I was able to check it and your email was received properly, but our
> employee responsible to read and answer all emails had to take a medical
> leave and that's why you didn't get a reply to your email. Our sincerely
> apologies for that.
> And thank you for reporting us the issue you noticed.

Please can you either:
a) update your internal processes so that reports to this email address are handled with speed, no matter who is ill or on holiday; or
b) provide an alternate email address that we can document in the CCADB as your Problem Reporting Mechanism, which does have some guarantees that it will be looked at in an timely manner?

Gerv

Comment 18

[Gervase Markham [:gerv]]

(In reply to Jason Crawford from comment #15)
> I would like to share with you some details regarding our PKI System and our
> position within the CA/Browser Forum.  Visa is one of the oldest operating
> Certificate Authorities and is currently a non-voting member within the
> CA/Browser Forum.  Visa has been operating a closed PKI system prior to the
> inception of the Baseline Requirements, which we had a number of legacy
> processes for the issuance and fulfillment of our certificates to our
> clients.  Certificates that are issued by Visa public CA’s are issued only
> to our clients for interconnectivity purposes.  Unlike other CA’s and
> particularly those that have undergone your Blink Process, our core business
> is not PKI.

The Mozilla Root Store Policy, section 2.1, states:

2.1 CA Operations. CAs whose certificates are included in Mozilla's root program MUST: 
1) provide some service relevant to typical users of our software products;

> The certificates that were impacted with the noted issues were
> not issued erroneously to a bad actor(s) nor do we issue certificates to the
> open public. Due to our unique PKI system, we are not at liberty to divulge
> with the public our list of impacted clients and their certificates without
> our Legals' consent.

Given this, it seems that Visa's PKI is more appropriately considered a private PKI and not a public one. I will start a discussion in mozilla.dev.security.policy about whether it is appropriate for Visa to continue to hold public trust.

Gerv

Comment 19

[Gervase Markham [:gerv]]

Discussion started:
https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/NNV3zvX43vE

Gerv

Comment 20

[Jason Crawford]

(In reply to Ryan Sleevi from comment #14)
> Thanks for responding. I think it's still necessary to provide additional
> detail.
> 
> That is, we can look at the problem from two dimensions: The problem itself,
> and the systemic issues that allowed the problem to manifest. Your
> description focuses on the resolution of the problem, but doesn't indicate
> any systemic changes have been made. As a consequence, it does not help the
> community feel that your CA has taken steps to reduce the risk of future
> violations (of any requirement) in the future. That is, one dimension is
> "Did you fix the bug", but another dimension is "How was the bug introduced,
> why was it not detected, and what steps are you taking to prevent future
> bugs"
> 
> To understand how you might approach this problem, consider
> https://groups.google.com/d/msg/mozilla.dev.security.policy/vl5eq0PoJxY/
> W1D4oZ__BwAJ and how it provided a timeline of events, the steps that were
> already in place (and with substantial detail), where there were controls
> missing or mistakes made, details about the steps being taken (e.g. "We
> fixed the bug" is not sufficient detail to understand), and a holistic,
> systemic awareness of how the CA is managed and the opportunities for errors.

We adopted the Baseline Requirements in late 2016 and while operating our legacy PKI Infrastructure, we wanted to ensure that there were no major systemic impact to our clients.  As listed in this bug, we fell short in quickly making updates to our legacy issuance processes to ensure the appropriate certificate attributes and certificates with non-FQDN’s as common names were addressed.  As previously mentioned we have addressed this legacy process as listed in this bug.  Moving forward, we have dedicated resources to closely monitor the CAB Forum ballots and votes to ensure we can update our clients as we adopt any new requirements so that we can make changes to our PKI to maintain our compliance with the Baseline Requirements.

(In reply to Jonathan Rudenberg from comment #16) 
> What other issues that have not yet been publicly reported have you discovered?
> In what areas are you currently out of compliance with Mozilla's Root Store Policy and the Baseline Requirements?

After completion of our internal review, we have determined that this was not a security issue, and have made changes to ensure certificates no longer fail the requirement.  We will track and re-issue the issued certificates through a process that does not interrupt services or cause potential outages.  All compliance issues in respect to Mozilla Root Program and Baseline Requirements have been communicated through the appropriate channels for public consumption.

(In reply to Gervase Markham from comment #17) 

> Please can you either:
> a) update your internal processes so that reports to this email address are handled with speed, no matter who is ill or on > > > holiday; or
> b) provide an alternate email address that we can document in the CCADB as your Problem Reporting Mechanism, which does have > > some guarantees that it will be looked at in an timely manner?

Our Internal processes have been updated for which we have sufficient coverage to monitor such reported issues in a timely manner for faster turnaround times.

Best regards,

Jason

Comment 21

[Firefox Bug Husbandry Bot]

Changing QA contact per https://bugzilla.mozilla.org/show_bug.cgi?id=1438254

Comment 22

[Wayne Thayer]

This root is being removed from the program via bug 1493822.