1540961 - Atos: Insufficient Serial Number Entropy

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[u636358]

User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36

Comment 1

[u636358]

How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

When performing self-compliance check on our Trusted Root based on discussions in mozilla.dev.security.policy with similar issues.

A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

2019-03-18: Atos TC self-assessment on certificates issued from Atos Trusted Root CAs. The issued certificates have effective serial number lengths of 63 Bit due to a feature of the used EJBCA (64 bit is configured for serial number length, but the highest bit is always set to zero to avoid misinterpreting as a negative signed long integer value.
2019-03-18: Atos TC installed PKI software upgrade on development environment and offline root-ca environment
2019-03-28: Atos TC Issued new Server-CA to replace Atos TrustedRoot Server-CA 2017 using offline root-ca environment
2019-03-29: Atos TC started software upgrade of EJBCA Version in production environment and changed CA Serial Number Octet Size to 12 for certificates issued by Atos TrustedRoot Server-CA 2019 and started certificate replacement planning.

Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

Atos TC issues new server certificates with certificate serial number length of 96 bit with entropy of 95 bit (highest bit is always zero). Issuance of server certificates with wrong entropie is stopped.

A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

The following certificates are effected:

Atos TrustedRoot Server-CA 2017:

• 271 server certificates
  First: https://crt.sh/?id=521426984 (2018-06-12)
  Last: https://crt.sh/?id=1322580659 (2019-03-27)

Atos TrustedRoot Server-CA 2013:

• 51 server certificates
  First: https://crt.sh/?id=49702396 (2016-11-01)
  Last: https://crt.sh/?id=446717646 (2018-05-07)

The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

The effected certificates are listed at:
https://crt.sh/?Identity=%25&iCAID=57403
https://crt.sh/?Identity=%25&iCAID=1306

Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

There was a misinterpretation of BR 7.1. It was believed a unsigned 64 bit integer was sufficient to satisfy the requirement. Atos TC has used the tools like zlint, CA/B Forum lint and X.509 lint for compliance checking, but the issue was not detected.

List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

2019-04-01: Information to certificate holder about planned certificate renewal process.
2019-05-30: Renewal of all effected server certificates
2019-05-31: Revocation of all effected server certificates

Comment 2

[Julien Cristau [:jcristau]]

Can you explain 1) the 3 weeks between the discussion on mdsp and Atos TC self-assessment, 2) the 2 weeks between said self-assessment and this report, 3) why issuance continued after the problem was detected, 4) the extra 2 months for revocation of misissued certs. It seems there's room for improvement there...

Comment 3

[u636358]

Like you, we see potential for improvement on our side. Since we got problems with the update of the CA Software and the connected RA Interface, we didn’t focus enough on the report. In future, we will keep priority on the incident report, as it will get mandatory like Wayne proposed (https://groups.google.com/forum/m/#!topic/mozilla.dev.security.policy/692gP7HqtPM).
Some days before the disclosure of our incident report we got in contact with Kathleen and Wayne for some clarification about possible discrepancies between the Baseline Requirements and the Mozilla Policy. Wayne yesterday confirmed that S/MIME is also in scope of the Mozilla Policy. After this confirmation we yesterday immediately stopped issuing user certificates.
About 200.000 S/MIME certificates need to be revoked and replaced. As a first step yesterday evening we stopped and updated the S/MIME-CAs. Beginning today we issue S/MIME-certificates only with correct entropy.

Furthermore we are confident that we will revoke most of the server faster as our timeline indicates. The timeline represents the completion of the process.
Regarding the timeline for S/MIME certificates we are currently in contact with our customers and will keep you updated (timeline and exact numbers).

Comment 4

[u636358]

Here the number of S/MIME certificates issued from 2019-09-30 until 2019-04-02 ordered by CA.

Atos TrustedRoot Issuing CA for <CUSTOMERNAME> 2015
5.913

Atos TrustedRoot Issuing CA for <CUSTOMERNAME> 2015
767

Atos TrustedRoot Client-CA 2013
88.198

Atos TrustedRoot Client-CA 2012
5.029

Atos TrustedRoot Client Issuing CA 2015
197

Atos TrustedRoot Client-CA for <CUSTOMERNAME> 2017
135

Note: All certificates issued after 2019-04-03 are correct. All Intermedia CAs, with the exception of “Atos TrustedRoot Client-CA for <CUSTOMERNAME> 2017”, are correct and don’t need be re-issued. And all certificates, with the exception of certificates from “Atos TrustedRoot Client-CA for <CUSTOMERNAME> 2017” and “Atos TrustedRoot Client Issuing CA 2015”, were issued on smartcards. The certificates from “Atos TrustedRoot Client-CA for <CUSTOMERNAME> 2017” have only the EKU id-kp-clientAuth (1.3.6.1.5.5.7.3.2). More Details will follow after further investigation.

Comment 5

[u636358]

Here is an update:
All customers are informed about the current situation. The exchange of the SSL certificates is ongoing. Many certificates are re-issued but not all of them are already installed. The customers have to plan maintenance windows. After the new certificates are in place, the old ones will be immediately revoked. The plan to revoke all misissued SSL-certificates and the Sub-CA Atos TrustedRoot Server-CA 2017 on 2019-05-31 is still in force.

We informed our customers also to renew their user certificates, which will automatically revoke the old certificates. Nearly all of these certificates are issued on smartcards, hence a high support effort is expected. And because of the architecture of the smartcard, which can only store a limited number of certificates, every additional certificate will cause inconvenience for the user when processing existing encrypted data and e-mails. Thus the smartcards are used in customer environments and there is no security issue, we would like to not revoke the certificates , but let the customers revoke them during regular renewals. About 150 user certificates which are issued as Soft-PSE will be revoked or renewed until 2019-05-31.
Nevertheless all affected Sub-CAs will be revoked and replaced at least on 2019-05-31.

In the meantime we enhanced our processes and work instructions and extended our team to track and implement policy changes and keep priority on the incident report.

Comment 6

[u636358]

The next update.
We are on track to revoke most of the mississued certificates on 2019-05-31. But we got the response of some customers that they aren’t able install the new certificates until that deadline. So we want to extend our deadline for the revocation of the remaining certificates to the 2019-06-30. On Friday we will provide the exact number of certificates to be replaced.

Comment 7

[Ryan Sleevi]

That is clearly not on track, and I don’t believe should be acceptable to extend the deadline.

Much more detail is required if that is your plan. Please review other CAs incident reports - such as those regarding underscores, and the need for per-customer thorough analysis and meaningful process changes by the CA. Otherwise, plan to have these revoked when stated.

Comment 8

[Thomas Schwieters]

Hello,

Current situation:
We are making good progress, we have setup a new intermediate CA 'Atos TrustedRoot Server CA 2019' and the old 'Atos TrustedRoot Server CA 2017' is not issuing certificates any more. The 'Atos TrustedRoot Server CA 2019' has a serialnumber of 96bits (entropy for 95 bits). All newly issued certificates also have a serialnumber of 96bits (entropy of 95 bits).

So, there will be no new cases of serialnumber problems. The remaining task is to revoke all affected certificates. For this point we would like to ask for an extension of our deadline until 30.06.2019.
We are facing issues, with the replacements of certificates at various customers.

Here is our current status:
Affected server certificates: 322
Renewed, but not yet revoked certificates: 46
Revoked certificates: 140
open certificates: 136

As you can see a lot of certificates are already renewed and the old ones revoked. We are in constant contact with our customers, to help installing the newly issued certificates.

We are facing two problems, for which we want to ask for an extension of the timeframe:

1. We are struggling with the feedback of some customers. We are currently increasing the pressure on the customers.
2. We have two customers, who have asked to extend the revokation date to at least 14.06.2019. They have planed the exchange for certificates, but since they have very complex systems (Banking systems) with also other customers using it, they need extended planing to replace the certificates.

Our plan is

1. to send a reminder to all affected customers who have not yet done a renewal on monday 03.06.2019.
2. Another final reminder will be sent on 10.06.2019.
3. On 17.06.2019 we will then start to manually revoke certificates.
4. The manual revokation of certificates should then be finished on 30.06.2019

From now on, we will provide weekly updates of the current renewal status.
We have also done the preparation to revoke all affected certificate, in case it is requested.

Comment 9

[Ryan Sleevi]

which we want to ask for an extension of the timeframe

This demonstrates a fundamental lack of awareness or participation in the community discussions for the past year. It has been repeatedly and clearly stated that extensions are not and cannot be granted. The CA is responsible for either adhering to the BRs or, as ATOS has done, violating the BRs. Similarly, the CA is responsible for choosing to revoke or not to revoke. The community reaction to that violation is based on how well a CA demonstrates awareness about the underlying issues, has a meaningful and concrete plan to address and prevent those, and delivers on the binding commitments it makes.

Furthermore, this situation is so remarkably similar to ones recently discussed, and the expectations of what CAs are supposed to do and the level at which they're to rise to, that ATOS' response fails to meet those basic minimums.

If ATOS has not been following the discussions for the past several months, as evidenced by Comment #8, and is not aware of the basic information expected of every CA proposing to further violate the BRs, then I think that calls into question the basic trustworthiness of the CA.

My advice is for ATOS to carefully examine the discussions in m.d.s.p. regarding exceptions and BR violations, such as around underscores, and carefully and thoughtfully examine the expectations for reporting incidents, before replying tomorrow with anything other than the effect that all remaining certificates have been revoked, as ATOS committed to the community to do on 2019-04-02.

Comment 10

[Ryan Sleevi]

And in case this seems as a surprise: Carefully review Comment #2 and Comment #3.

Comment 11

[u636358]

We understand your point and did inform our customers and revoked all remaining certificates which were affected.

Comment 12

[Ryan Sleevi]

To be clear: The decision to revoke or not is the CA's call. Wayne's posts on m.d.s.p. have tried to provide guidance, repeatedly, over the past half year as to the factors the CA needs to weigh. If you make a call to not revoke, you need to provide significantly more detail than presently provided. None of this is with the objective of breaking clients that may be affected - but the CA bears the responsibility to demonstrate the understanding, transparency, and holistic plan for remediation. That's where the current gap is, and that's why I encouraged a review of the past discussions, to understand precisely what it is you need to do. Because it's been hashed out so much (and at such length), and with careful consideration of the end-user impact at the forefront of everyone's mind, it's not something ease and pert to summarize - especially when CAs are, at this point, expected to be aware of those discussions.

But the information is out there, and if you make a call not to revoke, the information about how to ensure that single decision does not fundamentally undermine trust in ATOS is also there and available. You can make the call to rise to that level, or, to avoid the risk of failing to meet those expectations, you can make the call to revoke. Either is at the discretion of the CA.

Comment 13

[Ryan Sleevi]

Any updates?

Comment 14

[u636358]

Hello Ryan,

here is our status:

• All affected CA certificates, server certificates and SMIME certificates (in Software) are revoked (see comment 1 + 4), except SMIME certificates on Smartcard which will not be revoked (see comment 5)
• The CA Serial Number Octet Size is changed to 96bits, this means an entropy of 95 bits (see comment 1)
• To technically prevent miss-issuances because of the serial number, we added a validator script to the CA, next to zlint validator, which was already in place. We suggest to share that script to the community.
• The Atos team has been strengthened and re-trained in terms of fulfilling and implementing BR requirements, especially incident reports.

As a summary: All activities are finished.

Best regards

Michael

Comment 15

[Wayne Thayer]

It appears that remediation of this incident is completed.