1830536 - e-commerce monitoring gmbh: certificate issued with two pre-certificates

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Daniel Zens]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:108.0) Gecko/20100101 Firefox/108.0

Comment 1

[Daniel Zens]

A. BASIC INCIDENT DESCRIPTION

• Issuance of one leaf certificate for which there are existing two pre-certificates
• For cross reference, this is a consequential error of “SCT in precertificate” as reported in https://bugzilla.mozilla.org/show_bug.cgi?id=1815534

B. GENERAL PRELIMINARY REMARK

GLOBALTRUST (e-commerce monitoring gmbh) takes every incident very seriously and cooperates closely with its partners and supervisory bodies in all cases.
We continuously analyze our operations with regard to potential deviations from legal, technical or other requirements and agreements, like CA/Browserforum-Requirements, RFC, ETSI or eIDAS-Regulation.

We improve compliance management on an ongoing basis, aiming to reduce the error rate to an absolute minimum of acceptable rest risk.

GLOBALTRUST also deem it as a higher goal to contribute with the „lessons learned“ to the benefit of the entire community. Therefore we are looking also beyond this singular case to apply a holistic approach and to describe the situation in the overall structure of a constantly changing system of processes, as transparently as possible.

We have now thoroughly examined the case and you will find below an incident report that has been prepared to the best of our knowledge and belief, and which we believe should meet the expectations of the community and the corresponding requirements. We look forward to your constructive feedback.
C. FIRST NOTICE OF INCIDENT

How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP mailing list, a Bugzilla bug, or internal self-audit), and the time and date.

2023-02-07 Our internal control system showed two pre-certificates for one leaf certificate. This happened because the first pre-cert issued did not have enough ct-log timestamp entries and as a result a follow-up pre-cert was obtained. The technical check-systems used, Cablint, x509lint, zlint, gave neither an error message nor a warning.

2023-02-07 Andrew Ayer reported this point at https://bugzilla.mozilla.org/show_bug.cgi?id=1815534

2023-02-08 We answered Andrew Ayer's request in detail and explained that the issuance of the "duplicate" pre-certificate was caused by insufficient entries in the ct-log-timestamps. At no time (before or after) was it possible for us to issue a leaf certificate with the same serial number.

2023-02-08 In addition to the answer in the bug, programmatic measures were also taken to ensure that no pre-certificates with the same serial number can be issued.

2023-02-13 Since - according to detailed analyzes on our side - it was never possible to issue two leaf certificates with the same serial number. According to RFC6962, a pre-certificate is not a valid certificate. From our point of view, this was the end of the incident

2023-02-16 We took note of this issue by the below contributions of Rob Stradling and Aaron Gable in [Bug 1815534] e-commerce monitoring GmbH: SCT in precertificate:
• https://bugzilla.mozilla.org/show_bug.cgi?id=1815534#c6
• https://bugzilla.mozilla.org/show_bug.cgi?id=1815534#c12
The passage quoted by the discussion participants (BR 4.9.1.1) did not appear to be clearly applicable to this case, so that no further activities were carried out.

D. TIMELINE

A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
PRELIMINARY REMARK: SCT IN PRECERTIFICATE
For cross reference, this is the timeline as of Bug 1815534
https://bugzilla.mozilla.org/show_bug.cgi?id=1815534

2023-02-02 14:04 UTC: After retrieving the SCTs using the precertificate, we found that the number and composition of the SCTs did not meet our specifications (2 instead of 3 SCTs and no SCTs from a Google ct-log-server). A detailed analysis revealed that some Google ct-log-servers changed the format of their urls at the turn of the year. This unused pre-certificate has also been revoked.

2023-02-07 Our software was converted to the new url format and the entire module was subjected to a comprehensive revision. No server end-entity certificates were issued during this conversion and revision process.

2023-02-07 10:23 UTC: An SCT query was made for "10:45:67:49:88:c4:db:00:76:89:b9" (in Bug 1815534 precertificate and response #1), which was insufficient according to our specifications (only one SCT instead of three). Since this SCT was suitable, it was stored internally.

2023-02-07 10:38 UTC: Shortly thereafter, another SCT query was made, the response of which met our specifications (Bug 1815534 precertificate and response #2). These SCT values were stored internally.

2023-02-07 10:50 UTC: Based on the correct SCT entries from response #2, the end-entity certificate of authenticity was issued (in Bug 1815534 certificate and response #3). The end-entity certificate of authenticity contains only SCTs from response #2

2023-02-07 A subsequent review was carried out, primarily to avoid future disruptions due to unexpected changes in the url format and to speed up the issuing process of server end-entity certificates.

TIMELINE

2023-02-07 Our internal control system showed us that there are two pre-certificates for one leaf certificate.

2023-02-08 We checked extensively whether our system could prevent issuing two leaf certificates with the same serial number. This turned out to be impossible. This is partly because the serial number is generated using a random generator with a minimum length of 20 hexadecimal digits, which also includes the time stamp of the time of issue in addition to the random number. At no time (before or after) was it possible for us to issue a leaf certificate with the same serial number.

2023-02-08 In addition programmatic measures were also taken to ensure that no pre-certificates with the same serial number can be issued.

2023-02-13 Since - according to detailed analyzes on our side - it was never possible to issue two leaf certificates with the same serial number. According to RFC6962, a pre-certificate is not a valid certificate. From our point of view, this was the end of the incident.

2023-02-16 Contributions to [Bug 1815534] e-commerce monitoring GmbH: SCT in precertificate indicating non-compliance with RFC 5280 Section 4.1.2.2, Mozilla Root Store Policy 5.2 and CA/Browser Forum Baseline Requirements Section 7.1.2.4 in the context of Mozilla Root Store Policy 5.4.

2023-02-16 Every time after the publication of contributions in this matter, we analyzed whether further measures are useful or possible. This led to a misleading interpretation of the importance of pre-certificates in relation to leaf certificates.

2023-03-30 To clarify that neither a leaf certificate nor any pre-certificate exists for the given serial number, all certificates in question were revoked.

2023-04-02 Another post https://bugzilla.mozilla.org/show_bug.cgi?id=1815534 repeats the existing position. From our point of view, however, no further explanations are necessary.

E. REACTIONS

Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

The underlying cause had already been resolved as described below at the time of determining the incident, so there was no necessity to stop future issuance.

F. DOCUMENTATION OF INVOLVED CERTIFICATES

In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

Involved certificates: 1 (+2 Pre-Certificates)

In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

Precertificate with serial number 10:45:67:49:88:c4:db:00:76:89:b9:
https://crt.sh/?sha256=6f1fd13b3c72fd064e01b4fa810d44a06bde36f44876179a45bf752ae988d6cf
Precertificate with serial number 10:45:67:49:88:c4:db:00:76:89:b9
https://crt.sh/?sha256=c967a3109b953f8c14ffda5901db5f76d1bc0b6f84a3b6163ce14cd16d12086a
Leaf certificate with serial number 10:45:67:49:88:c4:db:00:76:89:b9 https://crt.sh/?sha256=5ee21e3bb051f280dec1339e83f60969919837da84a554007553edfb77c74a5f
G. DESCRIPTION CAUSE OF ERROR

Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

This happened because the first pre-cert issued did not have enough ct-log timestamp entries and as a result a follow-up pre-cert was obtained. The technical check-systems used, Cablint, x509lint, zlint, gave neither an error message nor a warning.

Work out how the bug or problem was introduced. For a code bug, were the code review processes sufficient? Does your code have automated tests, and if so, why did they not catch this case?

In extensive and detailed analyses, we were able to establish that there are no checking and monitoring routines whatsoever for the issue of multiple pre-certificates being issued in the Cablint, x509lint, zlint programs.

As a result, the issuance program for pre-certificates was significantly expanded:

• more comprehensive error messages when ct-log servers are not available
• Additional verification of the serial number for pre-certificates
• no issuance of pre-certificates, not even if the certificate data is formally correct, but other (internal) requirements are not met.

Additional (unscheduled) review of the program code with regard to the applicable policies.

Work out why the problem was not detected earlier. Were these certificates missed by your self-audits? Or is the code or process you use for such audits insufficiently frequent or rigorous?

We have not linted and audited this aspect. Please see immediately below.

If the problem is lack of compliance to an RFC, Baseline Requirement or Mozilla Policy requirement: were you aware of this requirement? If not, why not? If so, was an attempt made to meet it? If not, why not? If so, why was that attempt flawed? Do any processes need updating for making sure your CA complies with the latest version of the various requirements placed upon it?

The underlying requirement “no duplicate serial numbers” was known. GLOBALTRUST has for several years maintained successful measures to ensure uniqueness of serial numbers.
However, certificate transparency has brought a new dimension as described in the present report – the fact that also an assumed-to-exist-certificate is in scope by virtue of Mozilla Root Store Policy 5.4. This had not been properly taken into account in our interpretation and measures, respectively.

Scan your corpus of certificates to look for others with the same issue. It does not look good for a CA to claim they have revoked all affected certificates and resolved the issue, and then for a researcher to discover another set of certificates with the same or a similar problem.

No similar cases were found.

Examine whether there are potential related problems which you can also remediate at the same time. For example, if the problem was bad data in a particular field, consider improving the validation of all fields in the certificate prior to issuance. You should be proactively looking for ways, such as pre-issuance lint testing, to harden your issuance pipeline against further problems.

As a result, the issuance program for pre-certificates was significantly expanded:

• more comprehensive error messages when ct-log servers are not available
• Additional verification of the serial number for pre-certificates
• no issuance of pre-certificates, not even if the certificate data is formally correct, but other (internal) requirements are not met.

If, as happens in a regrettably large number of cases, a problem report was sent to your CA but action in accordance with BR section 9.4.5 was not taken within 24 hours, investigate what happened to that report and whether your report handling processes are adequate.

No problem report was sent, but our internal control system showed us that there are two pre-certificates for a leaf certificate. We reacted to this immediately, in particular by taking measures to prevent double-issued pre-certificates within 24 hours.

H. STEPS TO RECTIFY THE ERROR AND AVOID SIMILAR ERRORS

List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

All possible measures have been taken on our part to prevent "double" serial numbers. There are therefore no further steps open.

Comment 2

[Ryan Dickson]

Hi Daniel,

It seems that, in part, this incident was caused because “the first pre-cert issued did not have enough ct-log timestamp entries and as a result a follow-up pre-cert was obtained.” It was later identified that “A detailed analysis revealed that some Google ct-log-servers changed the format of their urls at the turn of the year.”

Please describe the specific steps GLOBALTRUST has taken to ensure that future changes by CT log operators are detected in a timely manner and raised to the attention of members of the GLOBALTRUST team before they can become a contributing factor to mis-issuance.

Also, your opening comment includes:

GLOBALTRUST also deem it as a higher goal to contribute with the “lessons learned “to the benefit of the entire community. Therefore we are looking also beyond this singular case to apply a holistic approach and to describe the situation in the overall structure of a constantly changing system of processes, as transparently as possible.

Can you share any specific lessons learned resulting from this incident? For example, can you share any lessons learned with members of the community such that we can reduce the impact of unexpected changes to the CT ecosystem in the future? Or, perhaps, offer ways of improving agility in response to unexpected events in the CT ecosystem (e.g., log disqualification)?

-Ryan

Comment 3

[hans.zeger]

Hi all,
(In reply to Ryan Dickson from comment #2)

Please describe the specific steps GLOBALTRUST has taken to ensure that future changes by CT log operators are detected in a timely manner and raised to the attention of members of the GLOBALTRUST team before they can become a contributing factor to mis-issuance.

Now we have - as part of our global operational monitoring - an automated check of the relevant ctlog files (e.g. https://opensource.apple.com/source/security_certificates/security_certificates-55093.40.3/certificate_transparency/log_list.json.auto.html) implemented. Among other things, this check shows us which ctlog servers - relevant to us - have changed their name (or no longer exist). In this way we can react proactively to changed situations - before issuing pre-certificates.

Can you share any specific lessons learned resulting from this incident? For example, can you share any lessons learned with members of the community such that we can reduce the impact of unexpected changes to the CT ecosystem in the future? Or, perhaps, offer ways of improving agility in response to unexpected events in the CT ecosystem (e.g., log disqualification)?

Lesson learned in general: Don't trust any external service or programs that you have not developed entirely yourself. With this in mind, we have provided additional acceptance tests and installation checks for installations of any type (including browser certificate check routings such as **lint).

Regards Hans

Hi Daniel,

It seems that, in part, this incident was caused because “the first pre-cert issued did not have enough ct-log timestamp entries and as a result a follow-up pre-cert was obtained.” It was later identified that “A detailed analysis revealed that some Google ct-log-servers changed the format of their urls at the turn of the year.”

Please describe the specific steps GLOBALTRUST has taken to ensure that future changes by CT log operators are detected in a timely manner and raised to the attention of members of the GLOBALTRUST team before they can become a contributing factor to mis-issuance.

Also, your opening comment includes:

GLOBALTRUST also deem it as a higher goal to contribute with the “lessons learned “to the benefit of the entire community. Therefore we are looking also beyond this singular case to apply a holistic approach and to describe the situation in the overall structure of a constantly changing system of processes, as transparently as possible.

Can you share any specific lessons learned resulting from this incident? For example, can you share any lessons learned with members of the community such that we can reduce the impact of unexpected changes to the CT ecosystem in the future? Or, perhaps, offer ways of improving agility in response to unexpected events in the CT ecosystem (e.g., log disqualification)?

-Ryan

Comment 4

[Ben Wilson]

Should this be "[uncategorized]" in the whiteboard under [ca-compliance], or should it be "[ov-misissuance]", or something else?

Comment 5

[Ben Wilson]

Daniel/Hans,
Please provide a status on this bug.
Thanks, Ben

Comment 6

[Daniel Zens]

In https://bugzilla.mozilla.org/show_bug.cgi?id=1815534#c17 we have indicated a number of improvements including a more closely monitoring of the list of usable ct log servers, and the impossibility of different certificates ahring the same issuer and serial number (even though one of them is "only" a presumed-to-exist one). In our view the underlying problem has been remediated. There are no further changes planned at the moment.

Comment 7

[Ben Wilson]

I will close this on Wed. 11-Oct-2023, unless there are additional questions or concerns to be answered.