Closed Bug 1692535 Opened 4 years ago Closed 3 years ago

Camerfirma: Delayed revocations of certificates issued by old CAs with an RSA modulus size of 2047 bits

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: ana.lopes, Assigned: bwilson)

Details

(Whiteboard: [ca-compliance] [leaf-revocation-delay])

Attachments

(1 file)

Claim statistics.png
54.28 KB, image/png
Details

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.146 Safari/537.36

Steps to reproduce:

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
    Since the moment that we received the notification we have been conscious about the impossibility to revoke such a big number of certificates concerning to some citizens and we are already involved in a substitution process (as we explained in the Bug 1692533 ) to revoke and substitute all certificates issued by the final entity CA RACER by the end of the year.

  2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
    2.1. After knowing the situation, we provided the clients with the links to issue their new certificates using a new CA and avoid more new certificates with problems. (February 5-12th)
    2.2. We are managing the revocation process so that we can prioritize the certificates affected with this problem.

  3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
    Find information in the Bug 1692533

  4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
    Find information in the Bug 1692533

  5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
    Find information in the Bug 1692533

  6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
    Find information in the Bug 1692533

  7. List of steps your CA is taking to resolve the situation
    7.1. As we mention in the bug XXXX, we are involved in a substitution and revocation process and we will have all the certificates issued by the CAs AC Camerfirma and RACER substituted by the end of the year.
    7.2. We will prioritize the certificates affected with this problem and will add information here about the revocation deadlines as soon as possible.

(In reply to Ana Lopes from comment #0)

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.146 Safari/537.36

Steps to reproduce:

  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
    Since the moment that we received the notification we have been conscious about the impossibility to revoke such a big number of certificates concerning to some citizens and we are already involved in a substitution process (as we explained in the Bug 1692533 ) to revoke and substitute all certificates issued by the final entity CA RACER by the end of the year.

  2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
    2.1. After knowing the situation, we provided the clients with the links to issue their new certificates using a new CA and avoid more new certificates with problems. (February 5-12th)
    2.2. We are managing the revocation process so that we can prioritize the certificates affected with this problem.

  3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
    Find information in the Bug 1692533

  4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
    Find information in the Bug 1692533

  5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
    Find information in the Bug 1692533

  6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
    Find information in the Bug 1692533

  7. List of steps your CA is taking to resolve the situation
    7.1. As we mentionin the Bug 1692533 , we are involved in a substitution and revocation process and we will have all the certificates issued by the CAs AC Camerfirma and RACER substituted by the end of the year.
    7.2. We will prioritize the certificates affected with this problem and will add information here about the revocation deadlines as soon as possible.

Assignee: bwilson → ana.lopes
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Whiteboard: [ca-compliance][delayed-revocation-leaf]

To perform the susbtitution and revocation process in a organized way, we have divided the total amount of certificates in different batches .
We began the process on February 23rd by notifying this substitution process to the first batch of customers and we gave them one month to replace their certificates issued by this CA. After this month every certificate of that batch will be revoked.
So, the first 13.877 certificates will be revoked on March 23rd, .

According to our predictions, the last batch of certificates will be revoked by September 15th at latest.

We will update this bug with the revocation progress every time that any certificate batch will be revoked.

We are continuing with the plan to carry out the revocation of the first batch of certificates on March, the 23th

We are continuing with the plan to carry out the revocation of the first batch of certificates on March, the 23th

We continue working on our revocation plan trying to follow the planification, but we have had to change the strategy because the defined process caused some problems for the clients.

The first strategy planned for these certificates and described in this bug was the same as the one that we were following for the complete SSL substitution that we are performing.

The first day we started revoking the first batch of certificates that we had planned, the number of client calls received by customer support department asking for help was more than five times bigger than the average in a normal day during a substitution process and the number of tickets opened was increased in more than 300%.

This situation has forced us to stop the revocations that we had planned for the certificates affected by this bug until having a new strategy. So, we could only have 4.703 certificates revoked on March 23rd , that is much less than the number we planned (13.877).

Our new strategy to revoke the certificates in time consists of trying to avoid client problems and complaints following the points detailed below:

  • We will not send the general emails that we sent in the past anymore and the new communications will be more personalized and with more details.
  • We will send the notifications per groups dividing the total in small groups to be able to revoke each batch in the planned deadlines.
  • We will only revoke the certificates contained in those batches when the client substitutes their certificate.
  • If a client does not substitute their certificate within the deadlines, it will be mandatory revoked by September 15th at latest, as we stipulated in this bug.

We will continue informing about the progress every week if we do not have any other important information to add.

Please, find below the updates of the revocation process:

Total affected certificates: 40510
Revoked Certificates: 4703
Alive certificates: 35807

  • Pending to revoke: 23762
  • With an expiration date less than September 1st 2021 (we do not include them in the substitution plan): 12045

We did not revoke during this last two weeks so the number of revoked certificates is the same as in last comment. We stopped the process because most of the clients were on holidays and did not have the possibility to substitute their certificates. We will continue with the process next week.

The following table indicates the dates when we send the notification to the clients to substitute their certificates, but we only revoke them as they substitute their certificates and not after the notification (as we detailed in the previous comment), so we cannot include an exact planification for the revocations but only for the notifications:

STAGE 1: NUMBER OF CERTIFICATES 15, DATE OF NOTIFICATION 09/02/2021
STAGE 2: NUMBER OF CERTIFICATES 2, DATE OF NOTIFICATION 12/02/2021
STAGE 3: NUMBER OF CERTIFICATES 13477, DATE OF NOTIFICATION 18/02/2021
STAGE 4: NUMBER OF CERTIFICATES 425, DATE OF NOTIFICATION 25/02/2021
STAGE 5: NUMBER OF CERTIFICATES 1362, DATE OF NOTIFICATION 03/03/2021
STAGE 6: NUMBER OF CERTIFICATES 66, DATE OF NOTIFICATION 15/03/2021
STAGE 7: NUMBER OF CERTIFICATES 112, DATE OF NOTIFICATION 22/03/2021
STAGE 8: NUMBER OF CERTIFICATES 22, DATE OF NOTIFICATION 16/04/2021
STAGE 9: NUMBER OF CERTIFICATES 661, DATE OF NOTIFICATION 23/04/2021
STAGE 10: NUMBER OF CERTIFICATES 703, DATE OF NOTIFICATION 26/04/2021
STAGE 11: NUMBER OF CERTIFICATES 11620,DATE OF NOTIFICATION 01/05/2021
TOTAL 28465

Revocation progress:

Last Friday 545 certificates were revoked.
In total, 5.248 certificates has been revoked.

Further revocations are expected this week.

Revocation progress:

344 new certificates has been revoked since the last update.
In total, 5592 certificates has been revoked.

We will continue informing about the process.

We want to clarify some aspects regading the new strategy that we follow to revoke and substitute the affected certificates to clarify possible doubts that may arise.

The first approach that we chose that was acting unilaterally did not have good results as we described in comment 5 (you can also find the claim statistics in the image attached).

After realising this situation, we decided to change the strategy and try to let them substitute their certificates before revoking them. This approach does not mean that we are not worry or we give the control of the revocation to the clients, but we focus on providing them with clear notifications and make sure that every client has their certificate active to operate with it before revoking.

We need to consider that most of the certificates to revoke are citizen certificates that they use to present their taxes and apply for grants from the Government. Thus, our intention is to cause the less damage possible trying not to revoke the certificates without a replacement just at the end of that period.

We have established two deadlines to revoke that will take place once we end the substitution for all the clients:

The first deadline is May 31st: By this date, we must have revoked the certificates belonged to the smallest projects that affect councils or small entities (6245 certificates).

The second deadline is July 31st: By this date, we must have revoked the rest of certificates belonged to the biggest projects that affect Autonomous Communities (16628 certificates).

The final deadline is the date that we provided in comment 2 (September 15th) and the period between the second deadline and this date will be used only to revoke the possible remaining active certificates due to incidents or exceptional situations that we could not solve before.

Attached image Claim statistics.png —

Revocation progress:
356 more certificates have been revoked this week.
In total, 5604 certificates have been revoked.
We expect more revocations next week.

Revocation progress:
627 more certificates have been revoked this week.
In total, 6231 certificates have been revoked.
We expect more revocations next week.

Revocation progress:
610 more certificates have been revoked this week.
In total, 6841 certificates have been revoked.
We expect more revocations next week.

Revocation progress:
6.742 more certificates have been revoked this week.
In total, 13.583 certificates have been revoked.
We expect more revocations next week.

Revocation progress:
1216 more certificates have been revoked this week.
In total, 14799 certificates have been revoked.
We expect more revocations next week.

Revocation progress:
719 more certificates have been revoked this week.
In total, 15.518 certificates have been revoked.
We expect more revocations next week.

Revocation progress:
721 more certificates have been revoked this week.
In total, 16.239 certificates have been revoked.
We expect more revocations next week.

Revocation progress:
570 more certificates have been revoked this week.
In total, 16.809 certificates have been revoked.
We expect more revocations next week.

Revocation progress:
961 more certificates have been revoked this week.
In total, 17.770 certificates have been revoked.
We expect more revocations next week.

Revocation progress:
308 more certificates have been revoked this week.
In total, 18.078 certificates have been revoked.
We expect more revocations next week.

Revocation progress:
883 more certificates have been revoked this week.
In total, 18.961 certificates have been revoked.
We expect more revocations next week.

Could you in your next updates also provide how many problematic certificates are still valid? Presumably, some have expired, and some have been revoked, but I'm having trouble seeing the whole picture with the information that is currently being provided in these progress updates.

Revocation progress:
1.483 more certificates have been revoked this week.
In total, 20.444 certificates have been revoked.
We expect more revocations next week.

Matthias, next week we will provide more information on the remaining certificates to be revoked.

Kind regards.

Total affected certificates: 40.510.

  • With an expiration date less than September 1st 2021 (we did not include them in the substitution plan): 12.045.
  • To be subsituted: 28.465 certificates.

This week, 7.350 certificates has been revoked.
In total, 27.794 certificates have been revoked

Pending revocations: 671 certificates.
All of these 671 certificates belongs to doctors from the "Servicio Gallego de Salud", the Health Service of Galicia, the Spanish region. They uses the certificates in order to access medical records from their patiens, access to analysys...
They have told us that due to the COVID they are having logistic difficulties to replace these batch of 671 certificates (these are smartcard certificates).
They have asked us for a extension in order to substitute this certificates.

Nowadays, we are discussing with the heads of "Servicio Gallego de Salud" a replacement plan for this last batch of certificates to be revoked.

We will update this bug with progress on pending revocations.

Whiteboard: [ca-compliance][delayed-revocation-leaf] → [ca-compliance] [delayed-revocation-leaf] Next update 2021-10-01

From the last batch of 671 certificates issued to doctors, 301 has been revoked.
Still have 370 certificates pending to revoke.

From the total amount of 28.465 certificates, 28.095 has been revoked and 370 certificates are pending to revoke.
Our aim is to revoke all of them during the next weeks.

We will update this bug with progress on pending revocations.

Whiteboard: [ca-compliance] [delayed-revocation-leaf] Next update 2021-10-01 → [ca-compliance] [delayed-revocation-leaf] Next update 2021-10-11

Since the last update 9 days ago, 67 more certificates have been revoked.

From the last batch of 671 certificates issued to doctors, 368 has been revoked.
Still have 303 certificates pending to revoke.

From the total amount of 28.465 certificates, 28.162 has been revoked and 303 certificates are pending to revoke.
Our aim is to revoke all of them during the next weeks.

We will update this bug with progress on pending revocations.

Since the last update 14 days ago, 73 more certificates have been revoked.

From the last batch of 671 certificates issued to doctors, 441 has been revoked.
Still have 230 certificates pending to revoke.

From the total amount of 28.465 certificates, 28.235 has been revoked and 230 certificates are pending to revoke.
Our aim is to revoke all of them during the next weeks.

We will update this bug with progress on pending revocations.

Since the last update 17 days ago, 78 more certificates have been revoked.

From the last batch of 671 certificates issued to doctors, 519 has been revoked.
Still have 152 certificates pending to revoke.

From the total amount of 28.465 certificates, 28.313 has been revoked and 152 certificates are pending to revoke.
Our aim is to revoke all of them during the next weeks.

We will update this bug with progress on pending revocations.

Since the last update 18 days ago, 40 more certificates have been revoked.

From the last batch of 671 certificates issued to doctors, 559 has been revoked.
Still have 112 certificates pending to revoke.

From the total amount of 28.465 certificates, 28.353 has been revoked and 112 certificates are pending to revoke.
Our aim is to revoke all of them during the next weeks.

We will update this bug with progress on pending revocations.

Since the last update 14 days ago, 43 more certificates have been revoked.

From the last batch of 671 certificates issued to doctors, 602 has been revoked.
Still have 69 certificates pending to revoke.

From the total amount of 28.465 certificates, 28.396 has been revoked and 69 certificates are pending to revoke.
Our aim is to revoke all of them during the next weeks.

We will update this bug with progress on pending revocations.

Since the last update 14 days ago, 39 more certificates have been revoked.

From the last batch of 671 certificates issued to doctors, 641 has been revoked.
Still have 30 certificates pending to revoke.

From the total amount of 28.465 certificates, 28.435 has been revoked and 30 certificates are pending to revoke.
Our aim is to revoke all of them during the next weeks.

We will update this bug with progress on pending revocations.

Whiteboard: [ca-compliance] [delayed-revocation-leaf] Next update 2021-10-11 → [ca-compliance] [delayed-revocation-leaf] Next update 2022-01-18

Since the last update 25 days ago, 30 more certificates have been revoked.

From the last batch of 671 certificates issued to doctors, 671 has been revoked.

From the total amount of 28.465 certificates, all of them has been revoked.

All,
Can we consider this issue closed? I will schedule this for closure on next Friday, 18-Feb-2022.
Ben

Flags: needinfo?(bwilson)
Flags: needinfo?(bwilson)
Whiteboard: [ca-compliance] [delayed-revocation-leaf] Next update 2022-01-18 → [ca-compliance] [delayed-revocation-leaf] Next update 2022-02-18

The bug assignee didn't login in Bugzilla in the last 7 months.
:kwilson, could you have a look please?
For more information, please visit auto_nag documentation.

Assignee: ana.lopes → bwilson
Flags: needinfo?(kwilson)
Flags: needinfo?(kwilson)
Flags: needinfo?(bwilson)
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] [delayed-revocation-leaf] Next update 2022-02-18 → [ca-compliance] [leaf-revocation-delay]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: