1567588 - D-TRUST: incorrectly formatted businessCategory entry

Enrico Entschew

Assignee

Description

•

6 years ago

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0

Steps to reproduce:

This is an initial incident report.

How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

2019-07-19, 02:23 UTC: Corey Bonnell brought via email (Problem Reporting Mechanism) to our attention that D-TRUST issued 81 EV certificates with an incorrectly formatted businessCategory entry.

A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

2019-07-19, 02:23 UTC: Initial report by Corey Bonnell
2019-07-19, 05:05 UTC: Start investigating the error
2019-07-19, 07:10 UTC: Shut down of application processing system
2019-07-19, 11:50 UTC: Intermediate result: 89 affected EV certificates
2019-07-19, 14:00 UTC: Start of customer communication process

Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

On 2019-07-12 due to the incident report https://bugzilla.mozilla.org/show_bug.cgi?id=1563772 we already have shut down the affected application processing website. Since than new applications are no longer accepted using this channel.

A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

Problem: Incorrect syntax of string in businessCategory
Number of affected certificates: 89 (intermediate result)
Issuing date of first certificate: 2017-07-27
Issuing date of last certificate: 2019-07-18

The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

To be handed later.

Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

According to current knowledge, due to an error in the application processing system for retail certificates and the X.509 control incorrect strings were made in the businessCategory subject and not recognized.

List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Since 2019-07-12 no certificates can be applied via the affected application processing website. Effective today (2019-07-19) no certificates can be produced via the affected application processing system. Investigations are still ongoing. We will work to have a response to the community by EOD, July 22th.

BugBot [:suhaib / :marco/ :calixte]

Updated

•

6 years ago

Status: UNCONFIRMED → ASSIGNED

Ever confirmed: true

Wayne Thayer

Updated

•

6 years ago

Assignee: wthayer → enrico.entschew

Type: defect → task

Wayne Thayer

Updated

•

6 years ago

Whiteboard: [ca-compliance]

Enrico Entschew

Assignee

Comment 1

•

6 years ago

List of all affected certificates according to Nr. 5 of the initial incident report.

Next response to the community by EOD, July 23th.

Enrico Entschew

Assignee

Comment 2

•

6 years ago

Update:
On 2019-07-21 we informed our Conformity Assessment Body (TÜV-IT) about this incident.

Right now replacements for all affected certificates are being produced via an unaffected application processing system. With the delivery of the replacement certificates customers will be informed that we plan to revoke the affected certificates no later than 2019-07-26.

The affected application processing system for retail certificates processed manual applications of customers only. Those customers do not run automated systems to apply for and receive certificates. A first lesson learned from this incident is that D-TRUST will migrate all customers who wish to do so to an automated system for applying and receiving certificates. A manual application for retail certificates will no longer be possible.
Next response to the community by EOD, July 25th.

Wayne Thayer

Comment 3

•

6 years ago

Enrico: thank you for the regular updates. I have no questions at this time.

Enrico Entschew

Assignee

Comment 4

•

6 years ago

Update:
75 certificates: replacement certificates produced
2 certificates: customer does not want a replacement certificate
12 certificates: pending revalidation

Next response to the community by EOD, July 26th.

Enrico Entschew

Assignee

Comment 5

•

6 years ago

Update:
Today we have revoked 42 certificates.

The remaining certificates couldn’t be revoked. There are two reasons for that.

For some certificates the revalidation is still pending.
For other certificates the customers are those who currently only have manual processes for applying, receiving and integrating certificates in their infrastructure. In addition, these customers operate in the finance, insurance, health care and government sector, stating that they operate critical infrastructures.

We are in daily contact with these customers to highlight the urgency of the replacement of these certificates as soon as possible.
We plan to revoke all remaining certificates 2019-08-09 at the latest.

Although we informed our customers at the beginning of this year that revocation of certificates is possible and necessary at any time. We now realize that this is not always understood by our customers. We are working on extended measures to ensure that this necessity is perceived more clearly by our customers.

Next response to the community by EOD, August 9th latest.

Ryan Sleevi

Comment 6

•

6 years ago

Enrico: Comment #5 does not meet the required information disclosure of https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation . I appreciate that you've enumerated "reasons", but as demonstrated by past CA incidents, this is not an acceptable level of detail. In particular, the following requirement exists, and has been practiced by other CAs, and is expected of all CAs missing the revocation deadlines:

When revocation is delayed at the request of specific Subscribers, the rationale must be provided on a per-Subscriber basis.

Of the remaining unrevoked certificates, please aggregate on a per-Subscriber basis, and provide details for each Subscriber, including the expected timeframes on a per-Subscriber basis.

With respect to Comment #0, the following explanation:

According to current knowledge, due to an error in the application processing system for retail certificates and the X.509 control incorrect strings were made in the businessCategory subject and not recognized.

Is ambiguous and difficult to understand the underlying root cause. Please provide an extended answer to

Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Note that such explanation should provide sufficient detail to understand both how your system currently works, as well as understanding why it failed to be detected. I would highlight bugs such as Bug 1550645 or Bug 1556948 as examples of significant technical details of the system that can demonstrate why and how things happened, or Bug 1551374 which carefully examined how things missed detection and the class of errors involved, as well as pre-existing mitigations.

Flags: needinfo?(enrico.entschew)

Enrico Entschew

Assignee

Comment 7

•

6 years ago

Update:
89 affected certificates
87 affected certificates: replacement certificates produced
64 affected certificates: revoked
2 affected certificates: customer does not want a replacement certificate
23 affected certificates: remaining for revocation

Ryan: Due to the current holiday period, many organizations have informed us that the availability of the technical staff is quite limited. This also affects their service providers. As already mentioned in comment #2, the affected system at D-TRUST was designed for retail customers who in most cases handle their certificates manually.

Please find attached a list of all certificates which were still valid on 2019-07-26.
22 of the certificates in the list have been revoked during the last 7 days.

The requested description of our system architecture and a more in-depth explanation why this error came to be and was not detected will be provided on Monday, 2019-08-05.

Entity
Affected Certificate: https://crt.sh/?id=1285711632
Replacement Pre-Certificate: https://crt.sh/?id=1696052364
current status: Certificate produced and delivered,
Reason for delay: Timely exchange by the customer not possible, request for postponement due to high risk (governmental infrastructure)
Scheduled revocation: 2019-08-09 at latest
Entity
Affected Certificate: https://crt.sh/?id=974083255
Replacement Pre-Certificate: https://crt.sh/?id=1698563641
current status: Certificate produced and delivered
Reason for delay: Timely exchange by the customer not possible, request for postponement due to high risk (health care)
Scheduled revocation: 2019-08-09 at latest
Entity
Affected Pre-Certificate: https://crt.sh/?id=1683728319
Replacement Pre-Certificate: https://crt.sh/?id=1699294492
Current Status: Certificate revoked
Revocation date: 2019-08-02 10:00:39 UTC
Reason for delay: Timely exchange by the customer not possible, request for postponement due to critical business impact

Affected Certificate: https://crt.sh/?id=1683751273
Replacement Pre-Certificate: https://crt.sh/?id=1699310038
Current Status: Certificate revoked
Revocation date: 2019-08-02 10:00:41 UTC
Reason for delay: Timely exchange by the customer not possible, request for postponement due to critical business impact

Entity
Affected Certificate: https://crt.sh/?id=757983557
Replacement Pre-Certificate: https://crt.sh/?id=1698666187
current status: Certificate produced and delivered
Reason for delay: Timely exchange by the customer not possible, request for postponement due to high risk (governmental infrastructure)
Scheduled revocation: 2019-09-05 at latest
Entity
Affected Pre-Certificate: https://crt.sh/?id=967530364
Replacement Pre-Certificate: https://crt.sh/?id=1721741178
current status: Certificate produced and delivered
Reason for delay: Re-Validation, timely exchange by the customer not possible, request for postponement due to high risk (governmental infrastructure)
Scheduled revocation: 2019-08-30 at latest

Affected Certificate: https://crt.sh/?id=920184019
Replacement Pre-Certificate: https://crt.sh/?id=1721776850
current status: Certificate produced and delivered
Reason for delay: Re-Validation, timely exchange by the customer not possible, request for postponement due to high risk (governmental infrastructure)
Scheduled revocation: 2019-08-30 at latest

Affected Pre-Certificate: https://crt.sh/?id=887849333
Replacement Pre-Certificate: https://crt.sh/?id=1721794413
current status: Certificate produced and delivered
Reason for delay: Re-Validation, timely exchange by the customer not possible, request for postponement due to high risk (governmental infrastructure)
Scheduled revocation: 2019-08-30 at latest

Affected Pre-Certificate: https://crt.sh/?id=887850071
Replacement Pre-Certificate: https://crt.sh/?id=1721810354
current status: Certificate produced and delivered
Reason for delay: Re-Validation, timely exchange by the customer not possible, request for postponement due to high risk (governmental infrastructure)
Scheduled revocation: 2019-08-30 at latest

Entity
Affected Certificate: https://crt.sh/?id=1618584614
Replacement Leaf Certificate: https://crt.sh/?id=1707449271
Current Status: Certificate revoked
Revocation date: 2019-07-31 13:29:00 UTC
Reason for delay: Timely exchange by the customer not possible, request for postponement due to high risk (governmental infrastructure)
Entity
Affected Pre-Certificate: https://crt.sh/?id=1595005572
Replacement Pre-Certificate: https://crt.sh/?id=1698729963
Current Status: Certificate revoked
Revocation date: 2019-08-02 12:12:42 UTC
Reason for delay: Timely exchange by the customer not possible, request for postponement, request for postponement due to high risk (governmental infrastructure)

Affected Certificate: https://crt.sh/?id=1241878230
Replacement Certificate: https://crt.sh/?id=1730624663
Current Status: Certificate revoked
Revocation date: 2019-08-02 10:01:47 UTC
Reason for delay: Timely exchange by the customer not possible, request for postponement, request for postponement due to high risk (governmental infrastructure)

Affected Certificate: https://crt.sh/?id=1232089373
Replacement Pre-Certificate: https://crt.sh/?id=1706282589
Current Status: Certificate revoked
Revocation date: 2019-08-02 12:11:38 UTC
Reason for delay: Timely exchange by the customer not possible, request for postponement, request for postponement due to high risk (governmental infrastructure)

Entity
Affected Pre-Certificate: https://crt.sh/?id=1366387762
Replacement Leaf Certificate: https://crt.sh/?id=1716562621
Current Status: Certificate revoked
Revocation date: 2019-07-31 13:26:56 UTC
Reason for delay: Timely exchange by the customer not possible, request for postponement due to critical business impact
Entity
Affected Certificate: https://crt.sh/?id=1402114088
Replacement Pre-Certificate: https://crt.sh/?id=1698949794
current status: Certificate produced and delivered
Reason for delay: Timely exchange by the customer not possible, request for postponement
Scheduled revocation: 2019-08-05 at latest

Affected Certificate: https://crt.sh/?id=1385059634
Replacement Pre-Certificate: https://crt.sh/?id=1698956839
current status: Certificate produced and delivered
Reason for delay: Timely exchange by the customer not possible, request for postponement
Scheduled revocation: 2019-08-05 at latest

Entity
Affected Certificate: https://crt.sh/?id=1578305245
Replacement Pre-Certificate: https://crt.sh/?id=1696080581
current status: Certificate produced and delivered
Reason for delay: Timely exchange by the customer not possible, request for postponement, request for postponement due to high risk (governmental infrastructure)
Scheduled revocation: 2019-09-06 at latest
Entity
Affected Certificate: https://crt.sh/?id=1597896254
Replacement Pre-Certificate: https://crt.sh/?id=1721824405
current status: Certificate produced and delivered
Reason for delay: Re-Validation, timely exchange by the customer not possible, request for postponement due to business impact
Scheduled revocation: 2019-08-09 at latest
Entity
Affected Certificate: https://crt.sh/?id=669094410
Replacement Pre-Certificate: https://crt.sh/?id=1699055166
current status: Certificate revoked.
Revocation date: 2019-08-02 10:01:49 UTC
Reason for delay: Timely exchange by the customer not possible, request for postponement due to business impact
Entity
Affected Certificate: https://crt.sh/?id=1510043408
Replacement Pre-Certificate: https://crt.sh/?id=1721983554
current status: Certificate revoked
Revocation date: 2019-08-02 04:55:33 UTC
Reason for delay: Re-Validation, timely exchange by the customer not possible, request for postponement, request for postponement due to high risk (public transport)

Affected Certificate: https://crt.sh/?id=1427442043
Replacement Pre-Certificate: https://crt.sh/?id=1721838220
current status: Certificate revoked
Revocation date: 2019-08-02 10:01:46 UTC
Reason for delay: Re-Validation, timely exchange by the customer not possible, request for postponement, request for postponement due to high risk (public transport)

Entity
Affected Certificate: https://crt.sh/?id=1573362043
Replacement Pre-Certificate: https://crt.sh/?id=1721854915
current status: Certificate produced and delivered
Reason for delay: Re-Validation, timely exchange by the customer not possible, request for postponement due to critical business impact
Scheduled revocation: 2019-08-09 at latest

Affected Certificate: https://crt.sh/?id=1305491609
Replacement Pre-Certificate: https://crt.sh/?id=1721872367
current status: Certificate produced and delivered
Reason for delay: Re-Validation, timely exchange by the customer not possible, request for postponement due to critical business impact
Scheduled revocation: 2019-08-09 at latest

Entity
Affected Certificate: https://crt.sh/?id=243519269
Replacement Leaf Certificate: https://crt.sh/?id=1716600863
Current Status: Certificate revoked
Revocation date: 2019-07-31 13:16:56 UTC
Reason for delay: Timely exchange by the customer not possible, request for postponement, request for postponement due to high risk (energy provider)
Entity
Affected Certificate: https://crt.sh/?id=1081324999
Replacement Pre-Certificate: https://crt.sh/?id=1706344955
current status: Certificate produced and delivered
Reason for delay: Timely exchange by the customer not possible, request for postponement, request for postponement due to high risk (health care insurance)
Scheduled revocation: 2019-08-09 at latest
Entity
Affected Pre-Certificate: https://crt.sh/?id=482036659
Replacement Pre-Certificate: https://crt.sh/?id=1721961647
current status: Certificate produced and delivered
Reason for delay: Re-Validation, timely exchange by the customer not possible, request for postponement, request for postponement due to high risk (health care insurance)
Scheduled revocation: 2019-08-09 at latest

Affected Pre-Certificate: https://crt.sh/?id=482021993
Replacement Pre-Certificate: https://crt.sh/?id=1722004110
current status: Certificate produced and delivered
Reason for delay: Re-Validation, timely exchange by the customer not possible, request for postponement, request for postponement due to high risk (health care insurance)
Scheduled revocation: 2019-08-09 at latest

Affected Pre-Certificate: https://crt.sh/?id=482036659
Replacement Pre-Certificate: https://crt.sh/?id=1721961647
current status: Certificate produced and delivered
Reason for delay: Re-Validation, timely exchange by the customer not possible, request for postponement, request for postponement due to high risk (health care insurance)
Scheduled revocation: 2019-08-09 at latest

Entity
Affected Certificate: https://crt.sh/?id=1153416688
Replacement Pre-Certificate: https://crt.sh/?id=1702061829
current status: Certificate revoked
Revocation date: 2019-08-02 04:56:40 UTC
Reason for delay: Timely exchange by the customer not possible, request for postponement due to critical business impact
Entity
Affected Certificate: https://crt.sh/?id=1132386978
Replacement Leaf Certificate: https://crt.sh/?id=1713536455
current status: Certificate revoked
Revocation date: 2019-08-02 04:56:38 UTC
Reason for delay: Timely exchange by the customer not possible, request for postponement due to critical business impact (finance)

Affected Certificate: https://crt.sh/?id=606600793
Replacement Pre-Certificate: https://crt.sh/?id=1701998221
current status: Certificate produced and delivered
Reason for delay: Timely exchange by the customer not possible, request for postponement due to critical business impact (finance)
Scheduled revocation: 2019-08-09 at latest

Entity
Affected Certificate: https://crt.sh/?id=1424842248
Replacement Leaf Certificate: https://crt.sh/?id=1713507586
current status: Certificate revoked
Revocation date: 2019-08-02 04:56:37 UTC
Reason for delay: Timely exchange by the customer not possible, request for postponement, request for postponement due to high risk (governmental infrastructure)
Entity
Affected Certificate: https://crt.sh/?id=626729077
Replacement Pre-Certificate: https://crt.sh/?id=1699521511
current status: Certificate produced and delivered
Reason for delay: Timely exchange by the customer not possible, request for postponement due to high risk (European PKI Infrastructure)
Scheduled revocation: 2019-08-09 at latest

Affected Certificate: https://crt.sh/?id=626931569
Replacement Pre-Certificate: https://crt.sh/?id=1699513671
current status: Certificate produced and delivered
Reason for delay: Timely exchange by the customer not possible, request for postponement due to high risk (European PKI Infrastructure)
Scheduled revocation: 2019-08-09 at latest

Affected Certificate: https://crt.sh/?id=624466451
Replacement Pre-Certificate: https://crt.sh/?id=1699502457
current status: Certificate produced and delivered
Reason for delay: Timely exchange by the customer not possible, request for postponement due to high risk (European PKI Infrastructure)
Scheduled revocation: 2019-08-09 at latest

Entity
Affected Certificate: https://crt.sh/?id=291949597
Replacement Pre-Certificate: https://crt.sh/?id=1705320996
current status: Certificate produced and delivered
Reason for delay: Timely exchange by the customer not possible, request for postponement due to critical business impact (finance)
Scheduled revocation: 2019-08-09 at latest
Entity
Affected Certificate: https://crt.sh/?id=1503760073
Replacement Certificate: https://crt.sh/?id=1730617879
current status: Certificate produced and delivered
Reason for delay: Timely exchange by the customer not possible, request for postponement due to high risk (Cloud Infrastructure)
Scheduled revocation: 2019-09-27 at latest

Affected Certificate: https://crt.sh/?id=1319976422
Replacement Certificate: https://crt.sh/?id=1730456199
current status: Certificate produced and delivered
Reason for delay: Timely exchange by the customer not possible, request for postponement due to high risk (Cloud Infrastructure)
Scheduled revocation: 2019-08-30 at latest

Entity
Affected Certificate: https://crt.sh/?id=1443586641
Replacement Leaf Certificate: https://crt.sh/?id=1710483708
Current Status: Certificate revoked
Revocation date: 2019-07-31 12:58:09 UTC
Reason for delay: Timely exchange by the customer not possible, request for postponement, request for postponement due to high risk (governmental infrastructure)
Entity
Affected Pre-Certificate: https://crt.sh/?id=1699343819
Replacement Certificate: https://crt.sh/?id=1730553850
Current Status: Certificate revoked
Revocation date: 2019-07-31 13:25:53 UTC
Reason for delay: Timely exchange by the customer not possible, request for postponement due to critical business impact
Entity
Affected Pre-Certificate: https://crt.sh/?id=1636252210
Replacement Pre-Certificate: https://crt.sh/?id=1699322452
current status: Certificate produced and delivered
Reason for delay: Timely exchange by the customer not possible, request for postponement, request for postponement due to high risk (governmental infrastructure)
Scheduled revocation: 2019-08-09 at latest
Entity
Affected Certificate: https://crt.sh/?id=1113944711
Replacement Pre-Certificate: https://crt.sh/?id=1698934566
Current Status: Certificate revoked
Revocation date: 2019-07-29 09:37:20 UTC
Reason for delay: Timely exchange by the customer not possible, request for postponement due to critical business impact
Entity
Affected Certificate: https://crt.sh/?id=568900293
Replacement Certificate: https://crt.sh/?id=1730636595
Current Status: Certificate revoked
Revocation date: 2019-07-30 07:44:44 UTC
Reason for delay: Timely exchange by the customer not possible, request for postponement due to critical business impact
Entity
Affected Certificate: https://crt.sh/?id=1548876874
Replacement Pre-Certificate: https://crt.sh/?id=1698984421
Current Status: Certificate revoked
Revocation date: 2019-07-29 09:38:23 UTC
Reason for delay: Timely exchange by the customer not possible, request for postponement due to critical business impact

Affected Certificate: https://crt.sh/?id=1042837384
Replacement Pre-Certificate: https://crt.sh/?id=1699003871
Current Status: Certificate revoked
Revocation date: 2019-07-29 09:36:16 UTC
Reason for delay: Timely exchange by the customer not possible, request for postponement due to critical business impact

Entity
Affected Certificate: https://crt.sh/?id=1283870923
Replacement Leaf Certificate: https://crt.sh/?id=1712053457
Current Status: Certificate revoked
Revocation date: 2019-07-31 12:55:03 UTC
Reason for delay: Timely exchange by the customer not possible, request for postponement due to critical business impact
Entity
Affected Certificate: https://crt.sh/?id=252197995
Replacement Leaf Certificate: https://crt.sh/?id=1727381751
Current Status: Certificate revoked
Revocation date: 2019-07-30 12:25:35 UTC
Reason for delay: Timely exchange by the customer not possible, request for postponement, request for postponement due to high risk (energy provider)

Flags: needinfo?(enrico.entschew)

Enrico Entschew

Assignee

Comment 8

•

6 years ago

Attached file 20190805-CA-Architecture.pdf — Details

This is to clarify the architecture of certificate production at D-TRUST, to show the reasons this error occurred and how we ensure it is not repeated in the future.

The diagram shows a system architecture overview of D-TRUST CA. System 1 (on the left) is the legacy application processing system for PTC retail certificates and System 2 (on the right) the current automated application processing system for PTC enterprise certificates. The affected legacy application processing system for PTC retail certificates has been shut down on 2019-07-19, 07:10 UTC and will remain shut down for good.

Until 2019-07-19 requests for PTC retail certificates were handled in system 1. All affected certificates were processed through this processing system. It allowed the entry of semantically incorrect data in the field businessCategory. The field check was not effective in this special use case. This was determined to be the root cause of the failure.

Since 2019-07-19, all PTC certificate applications are processed through our automated processing system (System 2). The logic of this newer application processing system effectively prevents this type of error.

Next response to the community by EOD, August 9th latest.

Enrico Entschew

Assignee

Comment 9

•

6 years ago

This is a pre-final incident report.

How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

2019-07-19, 02:23 UTC: Corey Bonnell brought via email (Problem Reporting Mechanism) to our attention that D-TRUST had issued 81 EV certificates with an incorrectly formatted businessCategory entry.

A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

2019-07-18, 16:00 UTC: Management decision to terminally shut down the application processing system for PTC TLS retail certificates according to https://bugzilla.mozilla.org/show_bug.cgi?id=1563772
2019-07-19, 02:23 UTC: Initial report by Corey Bonnell
2019-07-19, 05:05 UTC: Start investigating the error / Start Incident
2019-07-19, 07:10 UTC: Shut down of application processing system
2019-07-19, 11:50 UTC: Intermediate result: 89 affected EV certificates. This was also the final tally.
2019-07-19, 14:00 UTC: Start of customer communication process
2019-07-22, 07:00 UTC: Start of thorough analysis according to internal problem management procedures
2019-07-22, 14:30 UTC: Start issuing replacement certificates
2019-07-22, 15:10 UTC: Informing Conformity Assessment Body about the issue
2019-07-26, 14:30 UTC: Start revocation of affected EV certificates
2019-08-09, 11:00 UTC: Intermediate report of thorough analysis according to internal problem management procedures
2019-08-09: Pre-final incident report

2019-08-09: current status of certificate replacement
Affected certificates: 89
Certificates to be replaced: 87
Replacement certificates: 87
Revoked certificates: 80

Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

On 2019-07-12 due to the incident report https://bugzilla.mozilla.org/show_bug.cgi?id=1563772 we already had shut down the affected application processing website. Since then new applications are no longer accepted using this channel. Since 2019-07-19 no more certificates were produced through the affected application processing system for PTC TLS retail certificates, due to terminal shut-down of this system.

A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

Problem: Incorrect syntax of string in businessCategory
Number of affected certificates: 89
Issuing date of first certificate: 2017-07-27
Issuing date of last certificate: 2019-07-18

The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Please see Comment 1

Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Please see Comment 8

List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Since 2019-07-12 no certificates can be obtained for via the affected application processing website for PTC TLS retail certificates. Effective since 2019-07-19 no certificates can be produced via the affected application processing system for retail certificates.
The currently used automated application processing system for enterprise customers has been developed and improved in accordance with new development and testing procedures we set up at the end of 2017. These procedures were designed to prevent such issues more effectively.
We expect to revoke at least 4 of the remaining 7 certificates within August. Next response to the community (including update on the revocation plan) by EOD, August 30th latest.

Wayne Thayer

Updated

•

6 years ago

Whiteboard: [ca-compliance] → [ca-compliance] Next Update - 01-September 2019

Kim Nguyen

Comment 10

•

6 years ago

Today, August 20th 2019, the last affected certificate remaining has been revoked.
Therefore our pre-final incident report from August, 9th 2019 (as seen in Comment 9) with this addendum now turns into the final incident report.

Final revocation status:
89 affected certificates
87 affected certificates: replacement certificates produced
89 affected certificates: revoked
2 affected certificates: customer does not want a replacement certificate
0 affected certificate: remaining for revocation

Kim Nguyen, CEO D-Trust

Kim Nguyen

Comment 11

•

6 years ago

Sorry, today is the 30th of August 2019. Kim Nguyen

Ryan Sleevi

Comment 12

•

6 years ago

It took approximately six weeks to replace 89 certificates. The Baseline Requirements require CAs do so in five days, thus this is a significant and egregious breach of the BRs.

Have I missed a substantive plan to address this non-compliance and ensure that there are no future delays in revocation? That is, in handling this incident, D-TRUST intentionally allowed another incident to happen, and I don’t see any report, on this bug or as a new bug, regarding this. Please ensure a report is on file.

Is D-TRUST also committing to the public that it will never delay revocation again, beyond the time permitted?

Flags: needinfo?(enrico.entschew)

Kim Nguyen

Comment 13

•

6 years ago

Ryan, we clearly see your point and of course understand the requirement of revoking certificates within the timeframe given by the Baseline Requirements.

In fact within the remediation of this bug we analyzed our TLS business for retail customers and came to the conclusion that from our experience these are typically not able to replace certificates in the time frame mandated by the Baseline Requirements. This inability on the customer site due to lack of competence, resources or even insufficient contracts with IT suppliers results in a threat to the business of our customers resulting from a required short-term revocation of their TLS certificate.

We therefore made the major strategic decision to discontinue this retail business and to concentrate on business performed via our managed PKI platform which offers the ability to retrieve certificates in an automated manner.

As you suggested, we will prepare a new bug report regarding the delay of revocation next week.

Kim Nguyen, CEO D-Trust

Ryan Sleevi

Comment 14

•

6 years ago

Just making sure to track this: Bug 1580525 has been filed for the request in Comment #12

Ryan Sleevi

Comment 15

•

5 years ago

Wayne: With Bug 1580525 tracking the underlying root cause for delaying revocation, and keeping this focused on understanding the root cause for the businessCategory divergence in Comment #0, do you have any further questions, or should this be closed?

Flags: needinfo?(enrico.entschew) → needinfo?(wthayer)

Wayne Thayer

Comment 16

•

5 years ago

It appears that all questions have been answered and remediation is complete.

Status: ASSIGNED → RESOLVED

Closed: 5 years ago

Flags: needinfo?(wthayer)

Resolution: --- → FIXED

Whiteboard: [ca-compliance] Next Update - 01-September 2019 → [ca-compliance]

Nobody; OK to take it and work on it

Updated

•

3 years ago

Product: NSS → CA Program

David Lawrence [:dkl]

Updated

•

2 years ago

Whiteboard: [ca-compliance] → [ca-compliance] [ev-misissuance]