Closed Bug 1861069 Opened 1 year ago Closed 5 months ago

D-Trust: Issuance of 15 DV certificates containing ‘serialNumber’ field within subject

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: enrico.entschew, Assigned: enrico.entschew)

Details

(Whiteboard: [ca-compliance] [dv-misissuance])

Incident Report

This is a preliminary report.

Summary

After the September 15th, 2023 D-Trust issued 15 DV certificates containing the certificate field ‘serialNumber’ in the subject field. The value filled into the ‘serialNumber’ field was an internal reference number to the received certificate request. It is commonly used for internal purposes by D-Trust.

D-Trust stopped the production of DV certificates and is analysing the situation.

Assignee: nobody → enrico.entschew
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] [dv-misissuance]
Summary: Issuance of 15 DV certificates containing ‘serialNumber’ field within subject → D-Trust: Issuance of 15 DV certificates containing ‘serialNumber’ field within subject

Incident Report

Summary

This is an update to the preliminary report.

Issuance of 17 DV certificates containing ‘serialNumber’ field within subject

Impact

After the September 15, 2023 D-Trust issued 17 DV certificates containing the certificate field ‘serialNumber’ in the subject field. The value filled into the ‘serialNumber’ field was an internal reference number to the received certificate request. It is commonly used for internal purposes by D-Trust.

After discovering the issue, D-Trust stopped the production of DV certificates, analysed the situation, corrected the DV profile and restarted the production. 17 certificates were affected. 15 certificates needed to be revoked by the October 30, 2023. 2 certificates were already revoked by the customer before the incident was discovered.

All customers got informed about the situation. The revocation of 14 DV certificates took place in time. Unfortunately, one certificate was not revoked on time. A new ticket has been opened for this. The analysis of the reasons why the certificate was not revoked will start as soon as possible.

Timeline

All times are UTC.

2020-05-26:

  • 11:32 Start of operation of the DV certificate profile

2023-02-06:

  • 07:30 Check of existing certificate profiles against Ballot SC62

2023-09-15: Entry into force of the provisions from Ballot SC62

2023-10-24:

  • 09:30 Preparation of new DV certificate profiles for rollover SubCA D-TRUST BR CA 2-23-2 2023 fulfilling the requirements of BR 2.0.0
  • 12:15 Review with existing DV certificates and identifying a deviation
  • 12:30 Informing compliance team to further investigate the issue
  • 13:30 Start of internal analysis
  • 13:47 Precautionary stop of DV certificate issuance

2023-10-25:

  • 07:00 Manual check performed by compliance team based on DV certificates (unfortunately all of them issued before September 15th, 2023). As these certificates were issued before BR 2.0.0 came into force, there was no misissuance. Linting of all issued and valid DV certificates. No errors appeared.
  • 13:10 Manual check of DV certificates issued after September 15th, 2023. Discovery that misissuance has occurred.
  • 14:00 Decision to revoke affected certificates within 5 days
  • 14:25 Opening the bug at bugzilla

2023-10-26:

  • 08:30 Informing Customers
  • 08:30 Correction of product configuration and restart of production

2023-10-27:

  • 13:00 Informing Conformity Assessment Body about the issue

2023-10-30:

  • Until 13:00 Revocation of 14 remaining unrevoked deficient DV certificates
  • 21:10 Discovery that 1 deficient DV certificate remained unrevoked
  • 22:37 New incident report on the delay of the revocation opened (https://bugzilla.mozilla.org/show_bug.cgi?id=1862082)

Root Cause Analysis

D-Trust has been using the certificate field ‘serialNumber’ in the subject section to put in a value that is an internal reference number to the received certificate request when it was allowed by the standards. This number is commonly used for internal purposes like providing support to the customer.

Due to a misinterpretation of the newly issued Baseline Requirements 2.0.0 we came to the conclusion that no change in the certificate profile for DV certificates regarding the certificate field ‘serialNumber’ in the subject section was needed. That’s why the certificate field ‘serialNumber’ in the subject section was not deleted in time. 17 DV certificates were issued by the affected issuing CA after September 15, 2023.

Furthermore, the linter in use didn’t show any warning or error during the testing and production process. We are using the most updated version of the linter.

Lessons Learned

What went well

  • Creation of a new certificate profile from scratch, not based on existing profiles – That helped to identify the existing error/ misinterpretation.
  • Stop of production process, even if the team was not sure about the error at the time of error dedection – That prevented the issuance of more deficient DV certificates.

What didn't go well

  • Misinterpretation of the Baseline Requirements – The confusion arose because in the relevant chapter of the Baseline Requirements for the DV certificate, ‘all other attributes’ are mentioned. The ‘serialNumber’ field has been grouped under this category, what makes it difficult to find. For all other certificate types, the permissible and non-permissible attributes are listed in detail. In addition, the term ‘serialNumber’ is used as subject serial number and certificate serial number.
  • Trust in the linters in use as technical instruments – That shows that the human interpretation is still needed.
  • Customers ask for more time to exchange their DV certificates – We need to stress more that TLS certificates can be revoked between 24 hours and 5 days. Customers should be aware and prepared for that.
    On the other hand, it also shows that the burden of exchanging the certificate in complex infrastructures within 5 days is still too high and too costly. That’s of course only if the TLS certificates need to be revoked but are not creating any harm to certificate consumers or contain misleading information.
  • One certificate remained unrevoked. – A new ticket is opened. A thorough root cause analysis will be conducted to prevent such a circumstance in the future.

Where we got lucky

  • Our customers still ask for TLS certificates containing identity information. That’s why there is low demand for DV certificates from the customer side.

Action Items

Action Item Kind Due Date
Start the discussion about possible change request for Baseline Requirements to address the misleading handling of the certificate field ‘serialNumber’ Prevent 2024-04-01
Change request to update Z-Lint to detect prohibited certificate fields in the subject section of DV certificates Prevent 2024-02-01
Starting the discussion about a third revocation period for TLS certificates need to be revoked but are not creating any harm to certificate consumers or contain misleading information Prevent 2024-02-01

Appendix

Details of affected certificates

List of all affected certificates

Certificate that has not been revoked:
https://crt.sh/?sha256=57E5E699D847DA98F58AED2098CD1237925A6FEC03871B69B920BDFB425C92B2

Certificates that were revoked in time:
https://crt.sh/?sha256=0AF11626A995FB89FEF1F24FCB078C52453B03FC2F96AF9969BDF0023F94C97A
https://crt.sh/?sha256=E10AA8AAB7BCB507255D333C5506DFBA4B6EB7AE76CE8D0CAE242CA4B78F5136
https://crt.sh/?sha256=77521CDB939D398762C0ADAF5CD44C6D849DB340913206188D13043DA071372F
https://crt.sh/?sha256=990BC32A404859B91A1DA780411DB2CED743E803CA0A99D480989E55D184392A
https://crt.sh/?sha256=F0B45B55E1A9C428A079528F24DB813A5F1CBE3D4510CECA1A2B056D950DF649
https://crt.sh/?sha256=E62397EDB7B3F7A4F0B49505B0BCCB3EDC5C4FE99B394440CDA2FAD1B0A9162C
https://crt.sh/?sha256=BB1B4488967C121F6D2BF797981F8E673FDC16937ECFAA691878D3EE9FB79651
https://crt.sh/?sha256=606E8CBF8F3C4951C2734A7B7037480BEA350326A90A0DDE66FC2C5F4664E1E4
https://crt.sh/?sha256=ED2547DB70DC9D32EAFE7076DC79A88C4719DDA3048B63ABCF139CEED66F7AEC
https://crt.sh/?sha256=8E979E409F12C1644080DF56691ECE115451B90C49228C0E28870F9A914BB0A6
https://crt.sh/?sha256=475C4378D90A991A23EC2FE7123C29089645FF67FBCA71AF4BB4F65DBB9C041A
https://crt.sh/?sha256=E84968C63C6B16864E7BD96F798BD0331E095D938C68F44209A4242B274D3887
https://crt.sh/?sha256=96504F06A268242BCD1E74AF7776CB4E6AD276BCB4B68E63D84907127D25569D
https://crt.sh/?sha256=4C31DA1E85DE14BF04B08138101441274DA8F77914384B6DC098CF103B5E61DF

Certificates that were revoked by the customer before the incident was discovered
https://crt.sh/?sha256=59F94221B3666CE179D635526F3F154EC3260A54A82D5637E85A03CEE0CA73B0
https://crt.sh/?sha256=AFC5C61979D525C13B073C0192D327D18876E9426CF57937BB58B1EB9C943257

The affected certificate (https://crt.sh/?sha256=57E5E699D847DA98F58AED2098CD1237925A6FEC03871B69B920BDFB425C92B2) was successfully revoked today at 08:09 UTC.

Incident Report

Summary

This is the final report.

After the September 15, 2023 D-Trust issued 17 DV certificates containing the certificate field ‘serialNumber’ in the subject field. The value filled into the ‘serialNumber’ field was an internal reference number to the received certificate request. It is commonly used for internal purposes by D-Trust.

After discovering the issue, D-Trust stopped the production of DV certificates, analysed the situation, corrected the DV profile and restarted the production. 17 certificates were affected. 15 certificates needed to be revoked by the October 30, 2023. 2 certificates were already revoked by the customer before the incident was discovered.

All customers got informed about the situation. The revocation of 14 DV certificates took place in time. 6 DV certificates were revoked by the customers and 8 DV certificates were revoked by D-Trust. Unfortunately, one certificate was not revoked on time. A new ticket has been opened for this. The analysis of the reasons why the certificate was not revoked can be found here: https://bugzilla.mozilla.org/show_bug.cgi?id=1862082.

Impact

After the September 15, 2023 D-Trust issued 17 DV certificates containing the certificate field ‘serialNumber’ in the subject field. The error had to be corrected and all affected certificates had to be revoked.

Timeline

2020-05-26:

  • 11:32 Start of operation of the DV certificate profile

2023-02-06:

  • 07:30 Check of existing certificate profiles against Ballot SC62

2023-09-15: Entry into force of the provisions from Ballot SC62

2023-10-24:

  • 09:30 Preparation of new DV certificate profiles for rollover SubCA D-TRUST BR CA 2-23-2 2023 fulfilling the requirements of BR 2.0.0
  • 12:15 Review with existing DV certificates and identifying a deviation
  • 12:30 Informing compliance team to further investigate the issue
  • 13:30 Start of internal analysis
  • 13:47 Precautionary stop of DV certificate issuance

2023-10-25:

  • 07:00 Manual check performed by compliance team based on DV certificates (unfortunately all of them issued before September 15th, 2023). As these certificates were issued before BR 2.0.0 came into force, there was no misissuance. Linting of all issued and valid DV certificates. No errors appeared.
  • 13:10 Manual check of DV certificates issued after September 15th, 2023. Discovery that misissuance has occurred.
  • 14:00 Decision to revoke affected certificates within 5 days
  • 14:25 Opening the bug at bugzilla

2023-10-26:

  • 08:30 Informing Customers
  • 08:30 Correction of product configuration and restart of production

2023-10-27:

  • 13:00 Informing Conformity Assessment Body about the issue

2023-10-30:

  • Until 13:00 Revocation of 14 remaining unrevoked deficient DV certificates
  • 21:10 Discovery that 1 deficient DV certificate remained unrevoked
  • 22:37 New incident report on the delay of the revocation opened (https://bugzilla.mozilla.org/show_bug.cgi?id=1862082)

2023-10-31:

  • 08:09 Revocation of remaining affected certificate

2023-11-03:

  • 09:00 End of investigation of root cause

Root Cause Analysis

D-Trust has been using the certificate field ‘serialNumber’ in the subject section to put in a value that is an internal reference number to the received certificate request when it was allowed by the standards. This number is commonly used for internal purposes like providing support to the customer.

Due to a misinterpretation of the newly issued Baseline Requirements 2.0.0 we came to the conclusion that no change in the certificate profile for DV certificates regarding the certificate field ‘serialNumber’ in the subject section was needed. That’s why the certificate field ‘serialNumber’ in the subject section was not deleted in time. 17 DV certificates were issued by the affected issuing CA after September 15, 2023.

Furthermore, the linter in use didn’t show any warning or error during the testing and production process. We are using the most updated version of the linter.

Lessons Learned

What went well

  • Creation of a new certificate profile from scratch, not based on existing profiles – That helped to identify the existing error/ misinterpretation.
  • Stop of production process, even if the team was not sure about the error at the time of error dedection – That prevented the issuance of more deficient DV certificates.

What didn't go well

  • Misinterpretation of the Baseline Requirements – The confusion arose because in the relevant chapter of the Baseline Requirements for the DV certificate, ‘all other attributes’ are mentioned. The ‘serialNumber’ field has been grouped under this category, what makes it difficult to find. For all other certificate types, the permissible and non-permissible attributes are listed in detail. In addition, the term ‘serialNumber’ is used as subject serial number and certificate serial number.
  • Trust in the linters in use as technical instruments – That shows that the human interpretation is still needed.
  • Customers ask for more time to exchange their DV certificates – We need to stress more that TLS certificates can be revoked between 24 hours and 5 days. Customers should be aware and prepared for that.
  • On the other hand, it also shows that the burden of exchanging the certificate in complex infrastructures within 5 days is still too high and too costly. That’s of course only if the TLS certificates need to be revoked but are not creating any harm to certificate consumers or contain misleading information.
  • One certificate remained unrevoked. – A new ticket is opened. A thorough root cause analysis will be conducted to prevent such a circumstance in the future.

Where we got lucky

  • Our customers still ask for TLS certificates containing identity information. That’s why there is low demand for DV certificates from the customer side.

Action Items

Action Item Kind Due Date
Change request to update Z-Lint to detect prohibited certificate fields in the subject section of DV certificates Prevent 2024-02-01
Starting the discussion about a third revocation period for TLS certificates need to be revoked but are not creating any harm to certificate consumers or contain misleading information Prevent 2024-02-01

Appendix

Details of affected certificates

List of all affected certificates

Certificate that was not been revoked in time:
https://crt.sh/?sha256=57E5E699D847DA98F58AED2098CD1237925A6FEC03871B69B920BDFB425C92B2

Certificates that were revoked in time:
https://crt.sh/?sha256=0AF11626A995FB89FEF1F24FCB078C52453B03FC2F96AF9969BDF0023F94C97A
https://crt.sh/?sha256=E10AA8AAB7BCB507255D333C5506DFBA4B6EB7AE76CE8D0CAE242CA4B78F5136
https://crt.sh/?sha256=77521CDB939D398762C0ADAF5CD44C6D849DB340913206188D13043DA071372F
https://crt.sh/?sha256=990BC32A404859B91A1DA780411DB2CED743E803CA0A99D480989E55D184392A
https://crt.sh/?sha256=F0B45B55E1A9C428A079528F24DB813A5F1CBE3D4510CECA1A2B056D950DF649
https://crt.sh/?sha256=E62397EDB7B3F7A4F0B49505B0BCCB3EDC5C4FE99B394440CDA2FAD1B0A9162C
https://crt.sh/?sha256=BB1B4488967C121F6D2BF797981F8E673FDC16937ECFAA691878D3EE9FB79651
https://crt.sh/?sha256=606E8CBF8F3C4951C2734A7B7037480BEA350326A90A0DDE66FC2C5F4664E1E4
https://crt.sh/?sha256=ED2547DB70DC9D32EAFE7076DC79A88C4719DDA3048B63ABCF139CEED66F7AEC
https://crt.sh/?sha256=8E979E409F12C1644080DF56691ECE115451B90C49228C0E28870F9A914BB0A6
https://crt.sh/?sha256=475C4378D90A991A23EC2FE7123C29089645FF67FBCA71AF4BB4F65DBB9C041A
https://crt.sh/?sha256=E84968C63C6B16864E7BD96F798BD0331E095D938C68F44209A4242B274D3887
https://crt.sh/?sha256=96504F06A268242BCD1E74AF7776CB4E6AD276BCB4B68E63D84907127D25569D
https://crt.sh/?sha256=4C31DA1E85DE14BF04B08138101441274DA8F77914384B6DC098CF103B5E61DF

Certificates that were revoked by the customer before the incident was discovered
https://crt.sh/?sha256=59F94221B3666CE179D635526F3F154EC3260A54A82D5637E85A03CEE0CA73B0
https://crt.sh/?sha256=AFC5C61979D525C13B073C0192D327D18876E9426CF57937BB58B1EB9C943257

Quick Update: There is nothing new to report.

Quick Update: There is nothing new to report.

Quick Update: There is nothing new to report.

Quick Update: There is nothing new to report.

Whiteboard: [ca-compliance] [dv-misissuance] → [ca-compliance] [dv-misissuance] Next update 2024-02-01

Quick Update:
We are working on the ZLint update and will submit the corresponding change request to the project this month (in February).

We are preparing a presentation on the subject of a third revocation period for TLS certificates need to be revoked but are not creating any harm to certificate consumers or contain misleading information. Our plan is to present and discuss it during the 61st F2F of the CA/Browser Forum.

(In reply to Enrico Entschew from comment #8)

We are preparing a presentation on the subject of a third revocation period for TLS certificates need to be revoked but are not creating any harm to certificate consumers or contain misleading information. Our plan is to present and discuss it during the 61st F2F of the CA/Browser Forum.

As this topic was also presented on and discussed at the last CA/BF F2F, I hope that this presentation will incorporate and address the numerous concerns and points made there such that there can be forward movement on the topic.

Whiteboard: [ca-compliance] [dv-misissuance] Next update 2024-02-01 → [ca-compliance] [dv-misissuance]

(In reply to Clint Wilson from comment #9)

(In reply to Enrico Entschew from comment #8)

We are preparing a presentation on the subject of a third revocation period for TLS certificates need to be revoked but are not creating any harm to certificate consumers or contain misleading information. Our plan is to present and discuss it during the 61st F2F of the CA/Browser Forum.

As this topic was also presented on and discussed at the last CA/BF F2F, I hope that this presentation will incorporate and address the numerous concerns and points made there such that there can be forward movement on the topic.

Hi Clint, We will take these points into account and build on them in our presentation.
Thanks, Enrico

Hi Enrico,

Any update in regards to the ZLint action item?

As a reminder, CCADB.org describes:

Open incident reports should be updated:
- on or before the “Next update” date in the “Whiteboard” field of the bug;
- weekly, if a “Next update” date is not recorded; or
- when Action Items are changed, completed, or delayed.

Thanks,
Ryan

Flags: needinfo?(enrico.entschew)

Hi Ryan,

I am sorry for the delayed update of the status report.

We have made progress. Together with a partner, a change request was submitted to ZLint. This can be found here: https://github.com/zmap/zlint/pull/810

The change request has also been accepted in the meantime, so this change will appear in the next release.

Thanks,
Enrico

Flags: needinfo?(enrico.entschew)

Quick Update: There is nothing new to report.

Do you have a pre-production environment? I'm trying to understand if this could've been picked up in an environment before rolling out to production.

Yes, we do have a pre-production environment. However, it would not have changed the outcome because we use the same linter in our pre-production environment as in our production environment. The linter didn’t catch the error.

Quick update:
The topic of a third revocation period for TLS certificates that need to be revoked but are not creating any harm to certificate consumers or contain misleading information was brought up at the last CA/B Forum F2F and is still ongoing.

(In reply to Enrico Entschew from comment #16)

Quick update:
The topic of a third revocation period for TLS certificates that need to be revoked but are not creating any harm to certificate consumers or contain misleading information was brought up at the last CA/B Forum F2F and is still ongoing.

Please help me understand - how does a CA get to decide 'not creating harm'? And surely 'containing misleading information' is harmful?
There has been a rash of incidents lately where CAs simply ignore the rules around revocation, with no re-percussions.
Will adding a longer period for revoke help? Entrust is still trying (badly) to revoke and reissue the certificates they mis-issued. Would a third recvocation period help, or is it just another way for weak CAs to make excuses for themselves?

(In reply to JR Moir from comment #17)

(In reply to Enrico Entschew from comment #16)

Quick update:
The topic of a third revocation period for TLS certificates that need to be revoked but are not creating any harm to certificate consumers or contain misleading information was brought up at the last CA/B Forum F2F and is still ongoing.

Please help me understand - how does a CA get to decide 'not creating harm'? And surely 'containing misleading information' is harmful?
There has been a rash of incidents lately where CAs simply ignore the rules around revocation, with no re-percussions.
Will adding a longer period for revoke help? Entrust is still trying (badly) to revoke and reissue the certificates they mis-issued. Would a third recvocation period help, or is it just another way for weak CAs to make excuses for themselves?
I'd like to point out that the second revocation period came into effect in 2019, but had been pushed since at least 2017. The intent then was to create a distinction between a key compromise/security-impacting event and other non-compliance categories.

Speaking as someone from a different regulated field I find this slow erosion of standards unacceptable. As certificate lifetime goes down, why is the margin for response times increasing?

This isn't quite the place discussion but I'd prefer seeing an annual auditable event of 5-10% of certificates in a random batch being revoked and reissued (if they do not use short-term certificates and thus have an active CRL) as an assurance to both parties that they can both comply. No one is above a key compromise event, and I don't see why standards are getting relaxed.

Anecdotally from reviewing all the open issues recently I haven't seen a case where a change from 5 days to 15 days would materially impact what is reported. The CAs who unfortunately push past 5 days are either:

  • due to a few outlier subscribers which they are upfront about with highly detailed breakdowns, or
  • CAs who do not believe in adhering to the Baseline Requirements and have fought for weeks if they even consider revocation a requirement

Issues happen, but issues aren't interpreted in a vacuum. What matters is how you react to them and how open you are about each stage of your investigations.

My last post was probably too short, so there is a lot of room for misunderstanding here. I know this is a very difficult discussion. First of all, I can't speak for other CAs. My point is essentially to bring up a point that many of our clients have asked us about.

We, as D-Trust, had unfortunately issued certificates that had to be revoked due to violations of the Baseline Requirements, despite the utmost care. The revocation had to be done within 5 calendar days.

We are in relatively close contact with our customers and inform them about current topics in the CA/Browser Forum and the industry at regular intervals. Nevertheless, there is a lack of understanding on our customers’ side regarding the tight revocation period. The main point of criticism is that, due to our clerical errors, the customer is subject to a very strict deadline, which can tie up extensive resources on their side at very short notice . Applying for, issuing and delivering new certificates is often not the problem at all, but rather the internal distribution, for example. With more time to work with, this could run more smoothly.

To give you one example: We realize that it is very difficult for a customer to understand that the order of subject attributes leads to short-term revocation of the certificate . The certificate information is not misleading, as it has all been verified and provided by the customer and checked by the CA. I also do not see any security risk. In such cases, I can well imagine that this would ease the burden on the customer without compromising security. I hope that I was able to make my intention clearer.

I would like to emphasize that we are only talking about revocation due to clerical errors (not security-related revocation reasons). In the event of key compromise or other cases that already require immediate revocation, no compromises are to be made.

Quick Update: There is nothing new to report.

There is nothing new to report. I kindly ask you to check whether this incident can be closed.

Flags: needinfo?(bwilson)

Are there any comments, questions, or suggestions from the community? If not, I'll plan to close this sometime next week (May 27-31).

Status: ASSIGNED → RESOLVED
Closed: 5 months ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.