In the following, we provide a full incident report, still not definitive.
- How your CA first became aware of the problem and the time and date.
On March 3, 2019, upon reviewing a discussion in mozilla.dev.security.policy, we got aware of issues concerning the entropy in the serial numbers of certificates.
- A timeline of the actions your CA took in response.
(all times below are CET)
2019-03-03 h18:30, upon reviewing a discussion in m.d.s.p. we learned of a problem concerning serial numbers generated by the EJBCA software, that we also use, and started investiga-ting the impacts of said problem on our certificates.
2019-03-04 h10:15, having confirmed the impact and its root cause, we started studying the best course of action in order to fix it at the soonest; we tested a change to the EJBCA configuration so to lengthen serial numbers and verified that such change could be implemented on "live" CAs without unwanted side effects.
2019-03-05 h15.40, having found that the fix was viable and safe, we planned its deployment to our production systems for the next day early morning.
2019-03-06 h07:30, we deployed the fix to our production environment, then we double-checked that the change was effective and that everything else was still in order by closely examining several newly issued certificates and confirming that no one had any problems subsequent to the change.
2019-03-07 h08:30, we started investigating on the best way to re-issue the impacted certificates so to minimize disruption to our users. This task is still in progress.
- Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
Confirmed. Since March 6, h7.30 CET (see timeline above), all of our certificates are issued with longer serial numbers (16 bytes) containing much more than 64 bits of randomness in them.
- A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
We issued approx. 350,000 certs with the problem, of which approx. 224,000 are still active to date. We are still collecting and checking data. We will provide more precise figures in a subsequent update, as soon as possible.
- The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
We will provide a full listing in a subsequent update, as soon as possible.
- Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
The serial numbers generation logic implemented by EJBCA, combined with EJBCA’s default settings, caused serial numbers to have less entropy than we expected. We relied on EJBCA to generate 64-bits random serial numbers, and since the serial numbers had the expected length and did look random we did not realize that they contained less than 64 bits of randomness.
- List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
The defect was fixed on March 6 (see timeline at point #2).
We expect to revoke the majority of impacted certificates within approx. 30 days, barring unforeseen circumstances. We are preparing suitable procedures in order to achieve that with the minimum disruption to our customers. We anticipate that for a fraction of the impacted certificates it will take longer due to the limited agility of certain types of customers in handling certificate re-issuances at unexpected times.
To prevent reoccurrence of the same problem in the future, we adopted the following additional measures:
Since industry tools such as CABLint/ZLint do not currently handle this, we modified our own compliance-checking tool so to block issuance of certificates having serials shorter than 16 bytes. This change is already in production to date.
We hold an awareness meeting with all internal stakeholders to describe in details what happened, what were the causes, and to underline that we must ensure at all times our compliance with the randomness requirement on serial numbers as an integral part of our compliance with the BRs.