1738207 - Telia: Issued three precertificates with non-NIST EC curve

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[pekka.lahtiharju]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36 Edg/95.0.1020.30

Steps to reproduce:

Tried to issue EC certificate with non-NIST EC curve

Actual results:

Precertificate was issued

Expected results:

Request should have been rejected

Comment 1

[pekka.lahtiharju]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP mailing list, a Bugzilla bug, or internal self-audit), and the time and date.

In our internal post-lint checking on Wed 2021-10-27 we discovered that there were three precertificates that had problems in lint validation: "asn 1: structure error".

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

Mon 2021-10-25 Customer tried to use non-NIST EC curve in request but failed. However, three precertificates were created.
Wed 2021-10-27 We discovered the issue and revoked the three affected precertificates.
Wed 2021-10-27 We did initial analysis that there weren't any real certificates related to them and that there weren't any other similar ones. The problem was that customer had tried to use non-NIST-EC curve three times so that pre-certificate was created but CA refused to create the real certificate because CT stamps failed. Then he moved to RSA algorithm and got the certificate normally.
Wed 2021-10-28 We found the root cause of the problem to be in our CA configuration. There is a policy configuration to prevent non-NIST EC curves but the sentence was incorrectly put to configuration branch that was never used.
Wed 2021-10-28 We fixed the CA policy configurations so that this can't happen again.

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

We have fixed the CA policy configurations so that this can't happen again.

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

Three invalid precertificates from one use case. No affected real certificates.

5. In a case involving TLS server certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. When the incident being reported involves an SMIME certificate, if disclosure of personally identifiable information in the certificate may be contrary to applicable law, please provide at least the certificate serial number and SHA256 hash of the certificate. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

https://crt.sh/?id=5479539784
https://crt.sh/?id=5479539798
https://crt.sh/?id=5479539801

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

This was an error in our CA configuration which has lot of if-then-sentences. Non-NIST-curve-prevention sentence was incorrectly put under RSA key checking by human mistake. Same was correctly in Test system so we hadn't noticed this problem in our tests there.

Our primary NIST process today is checking certificate data after issuance because our CA software doesn't yet support pre-issuance checking. Our CA vendor has promised the pre-issuance checking to be available within few months. We hope that then we could prevent any lint issues even before pre-certificates.

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

Fix has already been done and all affected certificates were revoked on the same day when disclosed.

Comment 2

[Ryan Dickson]

Hi Pekka,

Our CA vendor has promised the pre-issuance checking to be available within few months.

Is there any timeline update regarding the new capability? Also, can you describe what specifically the updated software will be checking for and how?

Thanks,
Ryan

Comment 3

[pekka.lahtiharju]

Two weeks ago the vendor promised that this feature will be included in the next build that is available for testing in December 2021. If there are no problems we believe this could be in our production in February 2022.

The new feature enables us using pre-certificate before it will be sent to CT so that we can send it to external check. Our plan is to send it to zlint checking. If this external check gives error the pre-certificate won't be sent to CT at all and we can abort the whole enrollment.

Comment 4

[pekka.lahtiharju]

Update: We have now got the new CA software that is enabling pre-lint testing. It is installed to our test environment and we test it there about one month and then we move it to production if everything works as expected.

Comment 5

[pekka.lahtiharju]

Telia has now implemented the pre-linting solution in comment 3 and this kind of pre-certificate problem is no more possible for Telia. We can stop the enrollment before pre-certificate is created if zlint is giving any error. We are ready to close this bug.

Comment 6

[Ryan Sleevi]

Hi Pekka,

Can you more fully expand on your linting solution? It sounds like you just rely on ZLint, which does not comprehensively cover the requirements for CAs.

For example, can you share your process for evaluating the requirements, evaluating what the linter provides, and what steps you're taking to ensure coverage?

I ask, because it's easy to imagine an issue in the future being "ZLint didn't have a lint for this" (as we've seen some other CAs report), and so understanding what steps the CA is taking to ensure things are comprehensive seems valuable.

Comment 7

[pekka.lahtiharju]

Hi Ryan,

Few years ago when we started manual linting we asked if it is enough to use only one lint (zlint) and Mozilla answered that it is enough. After that we have used only zlint in our manual lint verification and now we implemented only it to our automatic pre-linting-script. If this is not the recommended method anymore could you give us new recommendation which lints or other systems should be used to properly auto-check the certificates.

In the current implementation we give the certificate to zlint and read the response so that if it is "fatal", "error" or "warning" we abort the enrolment before any pre-certificate is published. Alarm is sent to CA personnel to investigate the problem more closely and then necessary actions to fix the root cause are started. Zlint version we use is updated quarterly or sooner if some essential improvement is published. Then we also run batch run to all our older certificates to verify if some zlint improvement has revealed some findings from older certificates. Our CA software includes some quality checking features especially related to keys that are also utilized.

Lint is just one step in our verification process. The biggest work is to follow Mozilla and CABF discussions and Mozilla bugs so that we can implement all discussed ideas and requirements to our CA implementation. Two persons are allocated to follow and perhaps participate into those discussions. They bring their non-urgent findings into our weekly meetings and into our internal email/chat where we decide what improvements are needed. In urgent cases they escalate the issue according to our documented process. We also joined to CABF last Autumn to help Telia to get information about new plans as early as possible.

Comment 8

[Mathew Hodson]

(In reply to pekka.lahtiharju from comment #7)

Few years ago when we started manual linting we asked if it is enough to use only one lint (zlint) and Mozilla answered that it is enough. After that we have used only zlint in our manual lint verification and now we implemented only it to our automatic pre-linting-script. If this is not the recommended method anymore could you give us new recommendation which lints or other systems should be used to properly auto-check the certificates.

I read Ryan's question as asking Telia to provide details on how it ensures that it is meeting all requirements. Your answer doesn't address that. What is Telia doing to ensure that it meets requirements that ZLint doesn't cover?

Comment 9

[pekka.lahtiharju]

What is Telia doing to ensure that it meets requirements that ZLint doesn't cover

It is not simple to answer how we ensure that all hundreds of requirements are met. But I try it here. Telia certificate verification process has multiple elements like described below. Based on this issue we have made some minor improvements to in our existing operations and processes. These improvements further reinforce already extensive controls below to be more proactive to prevent accidental or otherwise incorrect certificate requests to filed in to our CA.

REQUIREMENT VALIDATION PROCESS
We are continuously monitoring the development, information and new requirements in different compliance frameworks and alike. Whenever new / changed requirement emerge we'll take them to our weekly change management agenda at our Telia CA Security Board. This board is an instrument in monitoring, identifying and deciding how and when we address the new / changed requirements. Identified change task is assigned to a responsible person, who is then accountable to make sure that changes are implemented as required (better and transparent assignment of responsibility). Said accountable shall report the progress to the Board on weekly basis, in the aforesaid meetings. The Board shall verify the change and make sure that adequate testing with supporting evidence is available before the Board accepts the change to put in production (proactive verification). Actual technical verification is done on two levels:

A) RA verification
Our self-made RA system restricts our customers to put any illegal requests to our system. It has hundreds of rules to prevent all kinds of problems before they go to CA software. In addition we have dozens of automatic tests that verify that all known illegal requests are rejected at RA system. Each time we get information about new kind of illegal request we add those to our automatic tests. If necessary we add new verification tests to RA system. After this issue we have fixed EC curve verification on RA level and added those to automatic testing. If the request bypass tests on RA system it will be checked again at CA system like in step B).

B) CA verification
There are certificate profiles in our CA software. Profiles define rules, parameters and extensions that each certificate type will have.
In the profile we can define which extensions come from CSR and which are generated by CA and what format is used. These rules have lot of conditions for example related to keys, data format, validity times and CT. E.g. illegal EC curve will be rejected also on this level. Now pre-zlint-check is a small but useful additional step on this level. Previously we used zlint only after enrolment.

CONCLUSION
We believe that our way of working will guarantee that also all non-zlint-requirements are fulfilled. Note! There haven't been many audit issues recently because our extensive verifications on two levels. If you have any doubts or improvement ideas I'm ready do listen or give more detailed answers.

Comment 10

[Ben Wilson]

Are there any additional questions or remediation items to address? If not, then I think this matter can be closed.