Closed Bug 1668331 Opened 6 years ago Closed 5 years ago

Camerfirma: Delayed revocations related to Invalid stateOrProvinceName field

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: martin_ja, Assigned: martin_ja)

Details

(Whiteboard: [ca-compliance] [leaf-revocation-delay])

This bug is opened because of the delay in the revocation of the certificates affected for the bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1667430.
Please, find below the incident report:

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

This bug has been opened because of a delay in the revocation of the certificates affected for the problems detected in the bug https://bugzilla.mozilla.org/show_bug.cgi?id=1667430 opened on September 25th.

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

The information related to this point is in the bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1667430

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

The information related to this point is in the bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1667430

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

The certificates that were revoked with delay are listed below:
All certificates affected for the bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1667430

  1. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

We’ll add this info this week.

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

As we told in the bug https://bugzilla.mozilla.org/show_bug.cgi?id=1667430, we try not to impact in our customers activity, we need their collaboration to replace the SSL certificates before being revoked. It is difficult to explain them why we need to revoke so urgently when there is not a security issue.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We are studying what we can do so that the situation does not repeat itself and we will inform you throughout this week

Assignee: bwilson → martin_ja
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance]
  1. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

We’ll add this info this week.

The data about the problematic certificates was uploaded (2020-10-02) in bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1667430#c20

7 List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We are studying what we can do so that the situation does not repeat itself and we will inform you throughout this week

We updated the info about this point (2020-10-202) in https://bugzilla.mozilla.org/show_bug.cgi?id=1667430#c23

Whiteboard: [ca-compliance] → [ca-compliance] [delayed-revocation-leaf]

I'd like to use this bug to focus on the delays here, although I realize there's been some conversation about this on Bug 1667430

I'm concerned that https://bugzilla.mozilla.org/show_bug.cgi?id=1667430#c9 stated it'd offer more detail, but comments appear to simply say "refer back to the bug", specifically:

we are going to open a new bug for the delay, giving more details about the reason and the plan,

In that same comment, Ana Lopes stated:

The revocation of this kind of certificates is more complicated than in other cases because we need to define the substitution plan and it involves more people and resources. Those kinds of SSL and the number of certificates affected take us to think that the 5-day period is not enough. We try not to impact in our customers activity.

My concern here is this has been a long-standing requirement, going back to the very first version of the Baseline Requirements, which itself only permitted 24 hours. The reason this requirement is so unambiguously stated in the Baseline Requirements is to ensure that CAs design their processes, procedures, and business to ensure that this control is met.

So I struggle to see how this is more complicated, unless it is the fact that Camerfirma has not designed their business and operations to actually comply with the Baseline Requirements. And if that is the case, then it seems like the path forward is to be discussing removing trust in Camerfirma, since this is a requirement for nearly 7 years.

Comment 23 on that issue further contains details more appropriate for this. Specifically:

Revocation in a short time in these two customers could cause an incident with a relevant impact on the customer's business. Safe replacement couldn’t be done in a shorter period of time.

However, zero details have been provided by Camerfirma, which is unacceptably not in line with the expectations for CAs.

Flags: needinfo?(martin_ja)

Revocation progress update:

The status of the revocation progress today is:

  1. Infocert subCAs:
    InfoCert Organization Validation 2019 CA 3 & InfoCert Organization Validation CA 3:
    (229 revoked + 1 expired )/301 misissued certificates

  2. Intesa Sanpaolo subCAs:
    Intesa Sanpaolo Organization Validation 2019 CA & Intesa Sanpaolo Organization Validation CA
    ( 93 revoked + 3 expired ) / 871 misissued certificates

We will update this information in the next few days.

https://bugzilla.mozilla.org/show_bug.cgi?id=1667430#c29 stated

Regarding the revocation status:

  1. 172 of the 202 certificates issued by "InfoCert Organization Validation 2019 CA 3" and "InfoCert Organization Validation CA 3" has been revoked. For the revocation of the remaining certificates we are in contact with customers who have asked us to wait so that they can proceed with the replacement of certificates without undermining the availability of their processes and the security of their users.
  2. 109 of the 871 certificates issued by "Intesa Sanpaolo Organization Validation 2019 CA" and "Intesa Sanpaolo Organization Validation CA" has already been revoked.

Eusebio: This doesn't meet the expectations called out in https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation

Please ensure Camerfirma carefully reviews that, because the responses to date are non-responsive to the concerns raised in Comment #2.

Flags: needinfo?(eusebio.herrera)

Hi Ryan,

In reply to Comment 2

As we explained in the bug https://bugzilla.mozilla.org/show_bug.cgi?id=1667430
We are very conscious about the importance of the BR and the deadlines defined by them.

In the case of Camerfirma, we have a procedure, and we apply the measures to avoid delays in the revocation like the definition of a substitution plan that we shared with you in the bug https://bugzilla.mozilla.org/show_bug.cgi?id=1623384 and also commented in the bug https://bugzilla.mozilla.org/show_bug.cgi?id=1668331
We defined this process to have an effective strategy to renew affected certificates and can revoke the affected ones in time.

In the case of CAs that use external platforms to issue the certificates, we have established the addendums with the clients that let us revoke all the certificates with error without the authorization of the clients. Besides. we are studying add legal penalty clauses in those addendums in case of delay.

Despite the addendum, it’ s true that in few cases the business impact of the revocation would be extraordinarily high and the revocation without a previous substitution would have very bad consequences for the business. In those cases, we need to establish plans with the clients to solve the situation in the best way possible for everyone.

Besides, we want to highlight the pressure that the important clients with strategic business make on Camerfirma in terms of claim for damages, even turning to their legal departments because of possible abusive clauses.

For that reason, as we told you before, our strategy is also focused on avoid errors because we think that it is the only way to avoid delays at all. Nevertheless errors happens and we keep on working with our clients to be aware that the certificates we provide can be revokes in 24h or 5 days. Camerfirma can provide with a new one in less that 24 hours, they should be ready to install it before revocation.

We are working on the improvement of the control in the design of profiles and issuance of all CAs controlled by Camerfirma in different ways as we detailed in other bugs like https://bugzilla.mozilla.org/show_bug.cgi?id=166833

Apart from that, we will continue giving details about the state of progress of revocation status related to this bug.

Flags: needinfo?(martin_ja)

Revocation progress update:

  1. Infocert subCAs:
    InfoCert Organization Validation 2019 CA 3 & InfoCert Organization Validation CA 3:
    ( 297 revoked + 2 expired) = 299 / 301 misissued certificates

  2. Intesa Sanpaolo subCAs:
    Intesa Sanpaolo Organization Validation 2019 CA & Intesa Sanpaolo Organization Validation CA
    ( 165 revoked + 6 expired ) = 171 / 871 misissued certificates

Flags: needinfo?(eusebio.herrera)

Revocation progress update:

Today has been revoked the last of the Infocert (InfoCert Organization Validation 2019 CA 3 & InfoCert Organization Validation CA 3) misissued certificates.

  1. Infocert subCAs:
    InfoCert Organization Validation 2019 CA 3 & InfoCert Organization Validation CA 3:
    ( 299 revoked + 2 expired) = 301 / 301 misissued certificates

  2. Intesa Sanpaolo subCAs:
    Intesa Sanpaolo Organization Validation 2019 CA & Intesa Sanpaolo Organization Validation CA
    ( 203 revoked + 6 expired ) = 209 / 871 misissued certificates

Revocation progress update:

Infocert subCAs:
InfoCert Organization Validation 2019 CA 3 & InfoCert Organization Validation CA 3:
( 299 revoked + 2 expired) = 301 / 301 misissued certificates

Intesa Sanpaolo subCAs:
Intesa Sanpaolo Organization Validation 2019 CA & Intesa Sanpaolo Organization Validation CA
( 275 revoked + 9 expired ) = 284 / 871 misissued certificates

Revocation progress update:

  1. Infocert subCAs:
    InfoCert Organization Validation 2019 CA 3 & InfoCert Organization Validation CA 3:
    ( 299 revoked + 2 expired) = 301 / 301 misissued certificates

  2. Intesa Sanpaolo subCAs:
    Intesa Sanpaolo Organization Validation 2019 CA & Intesa Sanpaolo Organization Validation CA
    ( 275 revoked + 9 expired ) = 284 / 871 misissued certificates

Revocation progress update:

  1. Infocert subCAs:
    InfoCert Organization Validation 2019 CA 3 & InfoCert Organization Validation CA 3:
    ( 299 revoked + 2 expired) = 301 / 301 misissued certificates
  2. Intesa Sanpaolo subCAs:
    Intesa Sanpaolo Organization Validation 2019 CA & Intesa Sanpaolo Organization Validation CA
    ( 386 revoked + 10 expired ) = 396 / 871 misissued certificates

Revocation progress update:

  1. Infocert subCAs:
    InfoCert Organization Validation 2019 CA 3 & InfoCert Organization Validation CA 3:
    ( 299 revoked + 2 expired) = 301 / 301 misissued certificates
  2. Intesa Sanpaolo subCAs:
    Intesa Sanpaolo Organization Validation 2019 CA & Intesa Sanpaolo Organization Validation CA
    ( 631 revoked + 10 expired ) = 641 / 871 misissued certificates

Revocation progress update:

Infocert subCAs:
InfoCert Organization Validation 2019 CA 3 & InfoCert Organization Validation CA 3:
( 299 revoked + 2 expired) = 301 / 301 misissued certificates

Intesa Sanpaolo subCAs:
Intesa Sanpaolo Organization Validation 2019 CA & Intesa Sanpaolo Organization Validation CA
( 844 revoked + 10 expired ) =  854 / 871 misissued certificates

Revocation progress update:

Infocert subCAs:
InfoCert Organization Validation 2019 CA 3 & InfoCert Organization Validation CA 3:
( 299 revoked + 2 expired) = 301 / 301 misissued certificates

Intesa Sanpaolo subCAs:
Intesa Sanpaolo Organization Validation 2019 CA & Intesa Sanpaolo Organization Validation CA
( 857 revoked + 10 expired ) = 867 / 871 misissued certificates

Revocation progress:

Infocert subCAs:
InfoCert Organization Validation 2019 CA 3 & InfoCert Organization Validation CA 3:
( 299 revoked + 2 expired) = 301 / 301 misissued certificates

Intesa Sanpaolo subCAs:
Intesa Sanpaolo Organization Validation 2019 CA & Intesa Sanpaolo Organization Validation CA
( 857 revoked + 10 expired ) = 867 / 871 misissued certificates

Revocation progress:

Infocert subCAs:
InfoCert Organization Validation 2019 CA 3 & InfoCert Organization Validation CA 3:
( 299 revoked + 2 expired) = 301 / 301 misissued certificates

Intesa Sanpaolo subCAs:
Intesa Sanpaolo Organization Validation 2019 CA & Intesa Sanpaolo Organization Validation CA
( 859 revoked + 10 expired ) = 869 / 871 misissued certificates

The last two certificates has been revoked, but this weekend after some aditional checks we have detected 27 aditional certificates with this issue.
We keep the revocation deadline of January 15th for revoke these aditional certificates.

So the revocation progress is:

Infocert subCAs:
InfoCert Organization Validation 2019 CA 3 & InfoCert Organization Validation CA 3:
( 299 revoked + 2 expired) = 301 / 328 misissued certificates

Intesa Sanpaolo subCAs:
Intesa Sanpaolo Organization Validation 2019 CA & Intesa Sanpaolo Organization Validation CA
( 861 revoked + 10 expired ) = 871 / 871 misissued certificates

There is no update regarding this matter from our part since 2020-12-22.

Do you consider this bug could be closed or do you need extra information about it?

I'm sorry, I've updated this bug incorrectly. This is the correct information:

Revocation progress:

Infocert subCAs:
InfoCert Organization Validation 2019 CA 3 & InfoCert Organization Validation CA 3:
( 299 revoked + 2 expired) = 313 / 328 misissued certificates

Intesa Sanpaolo subCAs:
Intesa Sanpaolo Organization Validation 2019 CA & Intesa Sanpaolo Organization Validation CA
( 861 revoked + 10 expired ) = 871 / 871 misissued certificates

(In reply to Juan Angel Martin from comment #0)

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

The information related to this point is in the bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1667430

I disagree with your (implicit) statement that that bug report contains the information and details that are expected from a CA when they decide to not revoke or to delay revocation[0].

1.) Could you please provide a timeline of what actions were taken, that resulted in your decision to not revoke the certificates that are in the scope of this issue on time?

As we told in the bug https://bugzilla.mozilla.org/show_bug.cgi?id=1667430, we try not to impact in our customers activity, we need their collaboration to replace the SSL certificates before being revoked. It is difficult to explain them why we need to revoke so urgently when there is not a security issue.

For this, I quote:

The rationale must include an explanation for why the situation is exceptional.
Responses similar to “we do not deem this non-compliant certificate to be a security risk” are not acceptable.
When revocation is delayed at the request of specific Subscribers, the rationale must be provided on a per-Subscriber basis.

- https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation

2.) This situation reminds me a lot of Bug 1623384 and the subsequent delay in revocation that is documented in Bug 1624658. Could you elaborate on how this delay is exceptional (and substantially different from the delay of Bug 1624658)?

3.) What specific reasons are there for each subscriber that the revocation of their set of misissued certificates was delayed for so long? Trouble in understanding revocation is necessary even for non-"security issues" does not sound like a valid answer to me.

4.) Additionally, did you not send a notification [1] to all subscribers that states "[...] In that sense, in the points 21 and 22 of the use conditions, you accepted specifically the possibility for CAMERFIRMA to revoke your certificate 24 hours after detecting a security incident or after 5 calendar days after detecting a format o content problem. [...]", such that subscribers would know about the need for quick replacement even in cases that are not "security incidents", and that delays like this could be prevented by invoking these clauses of the use conditions?

[0] https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation
[1] Bug 1624658, Comment #4

(In reply to Juan Angel Martin from comment #18)

Infocert subCAs:
InfoCert Organization Validation 2019 CA 3 & InfoCert Organization Validation CA 3:
( 299 revoked + 2 expired) = 313 / 328 misissued certificates

These numbers do not quite add up.

Hi Matthias,

Please, find below the answer to your questions in Comment #19

"1.) Could you please provide a timeline of what actions were taken, that resulted in your decision to not revoke the certificates that are in the scope of this issue on time? "

In order to establish the plan for their substitution and revocation that we presented on Septiembre 30th, we were analyzing the magnitude and impact for the clients and established the substitution and revocation plan.
Our SubCAs contacted their clients and established the plan with them, taking into account the revocation deadlines defined by the BR but also the impact that the revocation could cause on their businesses.
As we told in the bug https://bugzilla.mozilla.org/show_bug.cgi?id=1667430, we tried not to impact in our customers activity, and we need their collaboration to replace the SSL certificates before being revoked.

“For this, I quote:
The rationale must include an explanation for why the situation is exceptional.
Responses similar to “we do not deem this non-compliant certificate to be a security risk” are not acceptable.
When revocation is delayed at the request of specific Subscribers, the rationale must be provided on a per-Subscriber basis.

As required, we opened a new incident report before the delays took place as soon as we were conscious about the situation and the impossibility to revoke the certificates in time and we gave all the details, explanations and deadlines.
All this information will appear in the next audit report and the audit will review the risk analysis performed to evaluate it.

"2.) This situation reminds me a lot of Bug 1623384 and the subsequent delay in revocation that is documented in Bug 1624658. Could you elaborate on how this delay is exceptional (and substantially different from the delay of Bug 1624658)?"

The delay was exceptional in both cases, in the two cases the operations in the companies were not normal because of the restricted activity at the offices and the more difficult availability of employees, but in this particular case the number of certificates made the situation even more complicated.
In this particular case, the client is a bank and, as we already mentioned, the problem affected a big number of domains and machines and the revocation of this kind of certificates is more complicated than in other cases because we need to define the substitution plan and it involves more people and resources.

"3.) What specific reasons are there for each subscriber that the revocation of their set of misissued certificates was delayed for so long? Trouble in understanding revocation is necessary even for non-"security issues" does not sound like a valid answer to me. "

The infrastructure of Intesa Sanpaolo is very complicated because of the big number of machines and domains involved and thus, the big number of people who the activity depends on. Besides, the services that they offer, as a very important bank, make all these kind of operations very sensitive for their customers.

"4.) Additionally, did you not send a notification [1] to all subscribers that states "[...] In that sense, in the points 21 and 22 of the use conditions, you accepted specifically the possibility for CAMERFIRMA to revoke your certificate 24 hours after detecting a security incident or after 5 calendar days after detecting a format o content problem. [...]", such that subscribers would know about the need for quick replacement even in cases that are not "security incidents", and that delays like this could be prevented by invoking these clauses of the use conditions? "

Camerfirma has the possibility to revoke unilaterally in terms of law, but we have always studied and valuated the particular case of the client to offer them the best options trying to make the less damage possible in their activity. That is why we have tried to reach a compromise considering the potential risk that the problem registered in the bug can affect and the impact for their businesses following the possibility that the BR offer in the cases that the CA considers extraordinary.
We are working in contingency plans with the SubCAs and clients for possible future occasions.

That is why we have tried to reach a compromise considering the potential risk that the problem registered in the bug can affect and the impact for their businesses following the possibility that the BR offer in the cases that the CA considers extraordinary.

Wait, what? The BRs don't do any such thing. The BRs are the BRs.

Mozilla's policy provides discretion, but clear rationale must be accomplished. I'm not sure how the response from Camerfirma here can be taken as anything other than "We abide by the BRs, except when we don't want do", which is deeply concerning.

At the same time, the lack of faith in Camerfirma here, as evidenced by responses by Comment #20 even in spite of https://wiki.mozilla.org/CA:Camerfirma_Issues , do not give me much hope of there being a suitable remediation other than removing trust in Camerfirma. I'm assigning to Ben, because I don't believe this bug is likely to result in any more positive (for users) result given these fundamental issues with Camerfirma, to see if he'd like to close.

Flags: needinfo?(bwilson)

Revocation progress:

All the certificates has been revoked.

Infocert subCAs:
InfoCert Organization Validation 2019 CA 3 & InfoCert Organization Validation CA 3:
( 299 revoked + 2 expired) = 336 / 336 misissued certificates

Intesa Sanpaolo subCAs:
Intesa Sanpaolo Organization Validation 2019 CA & Intesa Sanpaolo Organization Validation CA
( 861 revoked + 10 expired ) = 871 / 871 misissued certificates

(In reply to Eusebio Herrera from comment #22)

InfoCert Organization Validation 2019 CA 3 & InfoCert Organization Validation CA 3:
( 299 revoked + 2 expired) = 336 / 336 misissued certificates

As already mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=1668331#c19 above, 299 + 2 = 301 != 336.

Camerfirma is not even following comments in this bug anymore, this is so depressing.

As we published earlier, after completing the analysis, we found two additional batches:

So, 301+27+8=336.

We do not have more updates for this bug.

I am going to close this bug on or about next Wed. 27-Jan-2021 with the understanding that Camerfirma will continue to work on addressing the underlying issues along with others raised on the wiki, https://wiki.mozilla.org/CA:Camerfirma_Issues, and in the related thread on the m.d.s.p. list.

Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] [delayed-revocation-leaf] → [ca-compliance] [leaf-revocation-delay]
You need to log in before you can comment on or make changes to this bug.