Actalis: Insufficient serial number entropy

ASSIGNED
Assigned to

Status

ASSIGNED
11 days ago
4 days ago

People

(Reporter: adriano.santoni, Assigned: adriano.santoni, NeedInfo)

Tracking

3.22

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [ca-compliance])

(Assignee)

Description

11 days ago

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36

Actual results:

On 3 March 2019 Actalis got aware of issues concerning the entropy in the serial numbers of certificates and started investigating the impacts.

We determined the root cause of the problem to lie in the unexpected and undocumented behavior of the "EJBCA" software.

Having found, on March 4, 2019, about 230,000 active certificates which serial numbers containing 63 bits of entropy, on March 6, 2019, at 7.30 AM (CET) we implemented a fix to remove the problem. Since then, all of our certificates are issued with longer serial numbers containing much more than 64 bits of entropy.

We expect to revoke the majority of impacted certificates within approx 30 days, barring unforeseen circumstances.

We will provide further details in a subsequent update.

Comment 1

11 days ago

Adriano,

Can you please provide the details of the "about 230,000 active certificates" that have now violated the BR revocation timeline?

Further, given that this is not (yet) a preliminary incident report in https://wiki.mozilla.org/CA/Responding_To_An_Incident , can you please provide one in the next 24 hours?

Similarly, when deferring matters to subsequent updates, can you provide clear timelines about when the next update is expected? It's unclear whether you expect to provide an update one day, one week, or one month from now, and providing clear timelines helps manage expectations about that.

Flags: needinfo?(adriano.santoni)

Updated

11 days ago
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Assignee: wthayer → adriano.santoni
Summary: Actalis: serial numbers with 63 bits of entropy → Actalis: Insufficient serial number entropy
Whiteboard: [ca-compliance]
(Assignee)

Comment 2

10 days ago

In the following, we provide a full incident report, still not definitive.

  1. How your CA first became aware of the problem and the time and date.

On March 3, 2019, upon reviewing a discussion in mozilla.dev.security.policy, we got aware of issues concerning the entropy in the serial numbers of certificates.

  1. A timeline of the actions your CA took in response.

(all times below are CET)

2019-03-03 h18:30, upon reviewing a discussion in m.d.s.p. we learned of a problem concerning serial numbers generated by the EJBCA software, that we also use, and started investiga-ting the impacts of said problem on our certificates.
2019-03-04 h10:15, having confirmed the impact and its root cause, we started studying the best course of action in order to fix it at the soonest; we tested a change to the EJBCA configuration so to lengthen serial numbers and verified that such change could be implemented on "live" CAs without unwanted side effects.
2019-03-05 h15.40, having found that the fix was viable and safe, we planned its deployment to our production systems for the next day early morning.
2019-03-06 h07:30, we deployed the fix to our production environment, then we double-checked that the change was effective and that everything else was still in order by closely examining several newly issued certificates and confirming that no one had any problems subsequent to the change.
2019-03-07 h08:30, we started investigating on the best way to re-issue the impacted certificates so to minimize disruption to our users. This task is still in progress.

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

Confirmed. Since March 6, h7.30 CET (see timeline above), all of our certificates are issued with longer serial numbers (16 bytes) containing much more than 64 bits of randomness in them.

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

We issued approx. 350,000 certs with the problem, of which approx. 224,000 are still active to date. We are still collecting and checking data. We will provide more precise figures in a subsequent update, as soon as possible.

  1. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

We will provide a full listing in a subsequent update, as soon as possible.

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The serial numbers generation logic implemented by EJBCA, combined with EJBCA’s default settings, caused serial numbers to have less entropy than we expected. We relied on EJBCA to generate 64-bits random serial numbers, and since the serial numbers had the expected length and did look random we did not realize that they contained less than 64 bits of randomness.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

The defect was fixed on March 6 (see timeline at point #2).

We expect to revoke the majority of impacted certificates within approx. 30 days, barring unforeseen circumstances. We are preparing suitable procedures in order to achieve that with the minimum disruption to our customers. We anticipate that for a fraction of the impacted certificates it will take longer due to the limited agility of certain types of customers in handling certificate re-issuances at unexpected times.

To prevent reoccurrence of the same problem in the future, we adopted the following additional measures:

  • Since industry tools such as CABLint/ZLint do not currently handle this, we modified our own compliance-checking tool so to block issuance of certificates having serials shorter than 16 bytes. This change is already in production to date.

  • We hold an awareness meeting with all internal stakeholders to describe in details what happened, what were the causes, and to underline that we must ensure at all times our compliance with the randomness requirement on serial numbers as an integral part of our compliance with the BRs.

You need to log in before you can comment on or make changes to this bug.