Closed Bug 1907667 Opened 1 year ago Closed 1 year ago

Disig: Two certificates with same serial number

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: peter.miskovic, Assigned: peter.miskovic)

Details

(Whiteboard: [ca-compliance] [ov-misissuance])

This is a preliminary report.

Summary

Disig was notified that it had issued two certificates with the same serial number but with different extension orders. This was documented by the existence of two different pre-certificates [https://crt.sh/?id=13680974784} and [https://crt.sh/?id=13680993322}.
In our CA system, we actually register only one certificate with a given serial number, which does not contain "poison extension" and has four CT log records written in it. According to the content, it is a certificate corresponding to the pre-certificate [https://crt.sh/?id=13680993322}. The existence of two pre-certificates indicates that there could be two different certificates with the same serial number, which would mean a violation of the requirement for the uniqueness of the serial number of the issued certificate in accordance with BR part 7.1.2.9.
This is a preliminary report. We will begin an investigation as soon as possible and update the full incident report as soon as possible. Till end of investigation we preventively stop issuing TLS certificate.

Impact

Timeline

All times are UTC.

YYYY-MM-DD:

  • HH:MM Example

Root Cause Analysis

Lessons Learned

What went well

What didn't go well

Where we got lucky

Action Items

Action Item Kind Due Date
Example Prevent 2038-01-19

Appendix

Details of affected certificates

https://crt.sh/?sha256=[sha256 fingerprint of the certificate]

Based on Incident Reporting Template v. 2.0

(In reply to Peter Miskovic from comment #0)

Disig was notified that it had issued two certificates with the same serial number but with different extension orders. This was documented by the existence of two different pre-certificates [https://crt.sh/?id=13680974784} and [https://crt.sh/?id=13680993322}.
In our CA system, we actually register only one certificate with a given serial number, which does not contain "poison extension" and has four CT log records written in it. According to the content, it is a certificate corresponding to the pre-certificate [https://crt.sh/?id=13680993322}.

You seem to be missing a link. The sentence indicates that you were trying to show two precertificates. You actually linked to one precertificate and one leaf certificate. The second and third links are the same.

Assignee: nobody → peter.miskovic
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] [ov-misissuance]

(In reply to Mathew Hodson from comment #1)

(In reply to Peter Miskovic from comment #0)

Disig was notified that it had issued two certificates with the same serial number but with different extension orders. This was documented by the existence of two different pre-certificates [https://crt.sh/?id=13680974784} and [https://crt.sh/?id=13680993322}.
In our CA system, we actually register only one certificate with a given serial number, which does not contain "poison extension" and has four CT log records written in it. According to the content, it is a certificate corresponding to the pre-certificate [https://crt.sh/?id=13680993322}.

You seem to be missing a link. The sentence indicates that you were trying to show two precertificates. You actually linked to one precertificate and one leaf certificate. The second and third links are the same.

Hi Mathew,

you are right. At the time I responded to the notification, two certificate files were attached to the notification email. I assume they are both pre-certificates so I looked for them through crt.sh and found two entries that I assumed belonged to the mentioned pre-certificates. Today, after analysis, I know that the first is a pre-certificate and the second is a certificate issued on the basis of this pre-certificate, but with shuffled extensions. More information will be provided in the incident report, which we would like to publish here today.

The Final Incident Report

Incident Report

Summary

This is the final report.
Disig was notified that it had issued two certificates with the same serial number but with different extension orders. This was documented by the existence of one pre-certificates [https://crt.sh/?id=13680974784] and one certificate [https://crt.sh/?id=13680993322] and the fact that pre-certificate and certificate with the same serial number have different order of certificate extension which are however in both pre-certificate and certificate identical.
In our CA system, we register only one issued certificate with a given serial number, which does not contain "poison extension" and has four CT log records written in it.
Based on the above, the notifier assumes that there is also a pre-certificate for the issued certificate [https://crt.sh/?id=13680993322] and also a certificate for the pre-certificate sent to the CT log [https://crt.sh/?id=13680974784] and thereby there was a violation of the requirement for the uniqueness of the serial number of the issued certificate in accordance with BR part 7.1.2.9.

Impact

Pre-certificate [https://crt.sh/?SHA256=3DF2ABF935E37A1D1A72581872255D67578AF66ECF4C8D5BBB725A8C366B1CD1] (1) and a corresponding leaf certificate [https://crt.sh/?SHA256=425DD228F0D66E6375AA5F8D361DD493FDBD312AF6960C249C80CEF738E4626C] (2) were issued at July 10, 2024
with different order of certificate extension present.

After further investigation we found other 2 affected TLS pre-certificates and certificate with the same issue:
Pre-certificate [https://crt.sh/?SHA256=8F3FF00320F391844FB75F96BFABF973A10528CC39495BA8E494C2E8982BDDC0]
Certificate [https://crt.sh/?SHA256=73872A347ADF120BD4D3F61B860F6335EA134B84C243469712FCBE8AA1EC2CA4]

Pre-certificate [https://crt.sh/?SHA256=D123AAD2FAC19A6E1F70BA27A3988F2F8E6BFA47B4887C5AB0DED538B1B703EE]
Certificate [https://crt.sh/?SHA256=0C0389D7FABCA25D93E1E1E87C39AE4AB27E49AC0C5AF2B9071B30CF41155CC5]

It is important to note that there have been no intend to issue two different TLS certificates with the same serial number and the different order of certificate extension.
In the reality are issued only three TLS certificates without "poisoning extension" which can be used as a publicly trusted TLS certificate by our customers.

Disig has decided preventively to stop the issuance of TLS certificates until a root cause has been found.

Timeline

All times are UTC.

2024-07-01:
Baseline Requirements for the Issuance and Management of Publicly‐Trusted Certificates Version 2.0.5 has become effective.

2024-07-12:
12:08 Disig was notified via e-mail that has issued two certificates with the serial number 0c1f909d0770fcbe9c00000000000007db.
It was based on the assumption that since there is a pre-certificate (1) and a certificate (2), in which the certificate extensions are identical, but listed in a different order.

19:30 The issuance of the TLS certificates was stopped.

19:30 After being alerted a preliminary investigation was started by checking the CA database to see if there were actually two issued TLC certificates with the same serial number. The result was negative.

20:38 Despite the negative result of checking if there are two issued TLS certificates with the same serial number, Disig created a bug for this incident on bugzilla.mozilla.org [https://bugzilla.mozilla.org/show_bug.cgi?id=1907667]

20:46 Disig informed notifier via e-mail about the creation of the new bug.

2024-07-15
05:00 An analysis was made of the reasons for issuing a TLS pre-certificate and a final certificate with different order of certificate extension, as the intention was to issue only one final certificate with a unique serial number, with the same content of extensions in the certificate, which eventually happened.
The result of our analysis is in the "Root Cause Analysis" part.

05:30 We asked our development team to investigate what was the reason that issued pre-certificate and corresponding final certificate have different extension order.

06:00 Our development team found that there is inccorect configuration error in our new CA signing software and immidiately stred to fix the issue.

11:43 We informed our customer for which we issued "problems" TLS certificate that we decided to revoke them in despite that we are not 100% sure that there exists two TLS certificate without "poisoning extension" with the same serial number.

12:04 Our development team announced us that they fixed the problem with the Bouncy Castle library and prepared a new release of our CA signing software. New release of CA signing software was sucesfully tested it on our testing CA.

13:11 The new release of CA Signing software was implemented to the production.

Root Cause Analysis

The investigation showed that TLS certificates in question were issued after an update to our own software certificate issuing tool, where the original version of the software written in C was replaced by a new version written in .NET.
The old version used openSSL to create the pre-certificate and the resulting TLS certificate, while the new version used BouncyCastle libraries.
In the case of the latter, when using it, we discovered that due to incorrect configuration, when removing the "poision extension" from the precertificate and creating the final certificate, together with the addition of a CT log record, the order of the extensions was changed, which we did not expect.

Lessons Learned

We learned that when we using external software library, it is necessary precisely test it, because of possible unexpected behaviour.

What went well

We were able quickly respond to incident and fix the problem.

What didn't go well

A thorough analysis of the issued pre-certificate and final certificate was not performed in terms of the order of extensions.

Where we got lucky

There have been no disruptions to our customers' websites or online services due to the potential existence of multiple TLS certificates from the same issuer with the same serial number, where such existence was ultimately not confirmed.

Action Items

Action Item Kind Due Date
Revoke and reissue all affected TLS certificates Mitigate 2024-07-19
DevOps and DB teams must must test the new release of our CA signing software more thoroughly and also check for unexpected behavior. Detect 2024-07-15
Implement a new version of our CA signing software where the extension order for pre-certificate and final certificate is fixed Prevent 2024-07-16
Implement in the new release of CA signing software checking that pre-certificate and corresponding final certificate has the same extensions order Prevent 2024-07-16

Appendix

Details of affected certificates

[https://crt.sh/?SHA256=3DF2ABF935E37A1D1A72581872255D67578AF66ECF4C8D5BBB725A8C366B1CD1]
[https://crt.sh/?SHA256=425DD228F0D66E6375AA5F8D361DD493FDBD312AF6960C249C80CEF738E4626C]
[https://crt.sh/?SHA256=8F3FF00320F391844FB75F96BFABF973A10528CC39495BA8E494C2E8982BDDC0]
[https://crt.sh/?SHA256=73872A347ADF120BD4D3F61B860F6335EA134B84C243469712FCBE8AA1EC2CA4]
[https://crt.sh/?SHA256=D123AAD2FAC19A6E1F70BA27A3988F2F8E6BFA47B4887C5AB0DED538B1B703EE]
[https://crt.sh/?SHA256=0C0389D7FABCA25D93E1E1E87C39AE4AB27E49AC0C5AF2B9071B30CF41155CC5]

A new version of our CA signing software was deployed to production yesterday afternoon at 13:11 GMT, where the extension order of the pre-certificate and final certificate is fixed, as well as checking the comparison of the contents of the pre-certificate and the final certificate regarding the order of extensions.

TLS certificate issuance was restarted today 2024-06-17 at 8:30 GMT and we successfully reissued two affected TLS certificates:

[https://crt.sh/?sha256=09DAEEE0CB5D6783DB665DC3F08A114A0F2A20F165D7944EF257F9152855CD06]
[https://crt.sh/?sha256=B74302FC6B368C6DF4544C31B2A28C98C874C667F954B547C1F7E6276FF2C5A3]
[https://crt.sh/?sha256=53C73CB9BCE49E909B687E066D787B7012A70E1F37E15CAC04BCE1917FE2F486]
[https://crt.sh/?sha256=D127DBDF5F390F47027C19947CB8823258119E379843C0136BA52D07CC933580]

Of the six problematic certificates, two were revoked today:
[https://crt.sh/?SHA256=D123AAD2FAC19A6E1F70BA27A3988F2F8E6BFA47B4887C5AB0DED538B1B703EE]
[https://crt.sh/?SHA256=0C0389D7FABCA25D93E1E1E87C39AE4AB27E49AC0C5AF2B9071B30CF41155CC5]

We have no new information in this bug. From our point of view, we consider the subject of this bug to be resolved and expect its closure.

Hi Peter,

  1. The timeline should be improved to describe when software certificate issuance tooling was updated (from C to .NET) to rely on Bouncy Castle (from OpenSSL). The “Root Cause Analysis" Section indicates this is the source of the issue, and therefore should be considered in the timeline.

  2. Can you describe how the issue avoided detection until it was discovered/identifed? (from the CCADB Incident Reporting Guidelines).

  3. Can you share more detail regarding the “Action Items” list? For example, in the case of “DevOps and DB teams must must test the new release of our CA signing software more thoroughly and also check for unexpected behavior." can you offer more specificity regarding what is being checked, how, and a what frequency?

  4. Can you share more on the specific approach Disig used to identify affected pre/final certificates?

  5. While looking into the reported issue, I was curious to better understand how Disig is establishing certificate serial numbers. This Sheet contains data related to pre/final issued by “CA Disig R2I2 Certification Service”. Studying the serial numbers included in this list, all appear to:

  • be 34-hex characters long
  • begin with “0c"
  • include a 16 hex character block of random values
  • include a block of 13 0’s (“0000000000000")
  • conclude with an incrementing pattern (i.e,. “07DD", “07DC", “07DB", etc.)

Is that understanding correct? Can you share more on this approach?

Thanks,
Ryan

Flags: needinfo?(peter.miskovic)

The Final Incident Report - update

Timeline

All times are UTC.

2024-07-01:
Baseline Requirements for the Issuance and Management of Publicly‐Trusted Certificates Version 2.0.5 has become effective.

2024-07-09:
12:22 Disig deployed a new version of the signature component (.NET) for signing of issued TLS certificates in the production environment

2024-07-10:
12:19 Disig issued a TLS pre-certificate and TLS certificate with the serial number 0c1f909d0770fcbe9c00000000000007db

2024-07-12:
12:08 Disig was notified via e-mail that has issued two certificates with the serial number 0c1f909d0770fcbe9c00000000000007db.
It was based on the assumption that since there is a pre-certificate (1) and a certificate (2), in which the certificate extensions are identical, but listed in a different order.

19:30 The issuance of the TLS certificates was stopped.

19:30 After being alerted a preliminary investigation was started by checking the CA database to see if there were actually two issued TLC certificates with the same serial number. The result was negative.

20:38 Despite the negative result of checking if there are two issued TLS certificates with the same serial number, Disig created a bug for this incident on bugzilla.mozilla.org [https://bugzilla.mozilla.org/show_bug.cgi?id=1907667]

20:46 Disig informed notifier via e-mail about the creation of the new bug.

2024-07-15
05:00 An analysis was made of the reasons for issuing a TLS pre-certificate and a final certificate with different order of certificate extension, as the intention was to issue only one final certificate with a unique serial number, with the same content of extensions in the certificate, which eventually happened.
The result of our analysis is in the "Root Cause Analysis" part.

05:30 We asked our development team to investigate what was the reason that issued pre-certificate and corresponding final certificate have different extension order.

06:00 Our development team found that there is inccorect configuration error in our new CA signing software and immidiately stred to fix the issue.

11:43 We informed our customer for which we issued "problems" TLS certificate that we decided to revoke them in despite that we are not 100% sure that there exists two TLS certificate without "poisoning extension" with the same serial number.

12:04 Our development team announced us that they fixed the problem with the Bouncy Castle library and prepared a new release of our CA signing software. New release of CA signing software was sucesfully tested it on our testing CA.

13:11 The new release of CA Signing software was implemented to the production.

Flags: needinfo?(peter.miskovic)

(In reply to Ryan Dickson from comment #7)

Hi Peter,

  1. The timeline should be improved to describe when software certificate issuance tooling was updated (from C to .NET) to rely on Bouncy Castle (from OpenSSL). The “Root Cause Analysis" Section indicates this is the source of the issue, and therefore should be considered in the timeline.

We added the deployment time of the new version of our signature component to the Final Incident Report timeline update.

  1. Can you describe how the issue avoided detection until it was discovered/identifed? (from the CCADB Incident Reporting Guidelines).

In our CA system, we did not control the content of the pre-certificate and the resulting certificate, regarding the exact order of extensions, since this problem did not occur during the entire time of its use when using the signature component written in C language. We had no idea that using .NET with the configuration we set up could cause the order to change.

  1. Can you share more detail regarding the “Action Items” list? For example, in the case of “DevOps and DB teams must must test the new release of our CA signing software more thoroughly and also check for unexpected behavior." can you offer more specificity regarding what is being checked, how, and a what frequency?

We anticipate that in our test environment, for any change in the signing component that affects the issuance of TLS certificates, we will now closely monitor the content of the issued cert compared to the pre-certificate as well as the content of all items in the certificate to avoid similar issues that occurred in the past.

  1. Can you share more on the specific approach Disig used to identify affected pre/final certificates?

After detecting a difference in the order of extensions in the pre-certificate and the certificate, a detailed check of their contents was incorporated into our CA software to ensure that the order of extensions was the same. Of course, this is only an additional check, but the current configuration setting for pre-certificate creation guarantees us that the problem with the order of extensions will not happen again. This is also evidenced by the newly issued TLS certificates, where extension are already in order from the point of view of the pre-certificate and the certificate.

  1. While looking into the reported issue, I was curious to better understand how Disig is establishing certificate serial numbers. This Sheet contains data related to pre/final issued by “CA Disig R2I2 Certification Service”. Studying the serial numbers included in this list, all appear to:
  • be 34-hex characters long
  • begin with “0c"
  • include a 16 hex character block of random values
  • include a block of 13 0’s (“0000000000000")
  • conclude with an incrementing pattern (i.e,. “07DD", “07DC", “07DB", etc.)

Is that understanding correct? Can you share more on this approach?

Yes,you understanding is correct.
The first two characters are a special indication of the serial number, then 16 hex characters block of random values comes from BR TLS requirement section 7.1.2.7 and rest is our former incremental patter we used before BR TLS requirements (version 1.3.7, effective from 30‐Sep‐2016)
stated that serial number must contained at least 64 bits of outputs from a CSPRNG.
According the RFC 5280 (4.1.2.2) the serial number "Given the uniqueness requirements above, serial numbers can be expected to contain long integers. Certificate users MUST be able to handle serialNumber values up to 20 octets.".

Thanks,
Ryan

RFC 5280 is superseded by the Baseline Requirements with a max length of up to 64 as per 7.1.4.2, Table 78.

Hi Wayne,
BR 7.1.4.2 concerning subject.serialNumber (OID 2.5.4.5) but the bug is about certificate serialNumber (see BR 7.1.2.7 Subscriber (Server) Certificate Profile)

We have no new information in this bug.

We have no new information in this bug. From our point of view, if there is no more questions, we consider the subject of this bug to be resolved and expect its closure.

I will close this on or about Friday, 9-August-2024, unless there are still questions or issues to address.

Flags: needinfo?(bwilson)
Flags: needinfo?(bwilson)
Summary: Disig has issued two certificates with the serial number 0c1f909d0770fcbe9c00000000000007db → Disig: Two certificates with same serial number

We have no new information in this bug. There were no more question after August 8, 2024 so we expect that case will be closed.

If there are no additional questions or issues to discuss, then I will close this on or about Friday, 16-August-2024.

Flags: needinfo?(bwilson)
Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.