Closed Bug 1888104 Opened 8 months ago Closed 5 months ago

Disig: TLS certificate with basicConstraints not marked as critical

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jozef.nigut, Assigned: jozef.nigut)

Details

(Whiteboard: [ca-compliance] [ov-misissuance])

Disig has been notified about a leaf certificate (https://crt.sh/?id=10581171523) and a corresponding precertificate (https://crt.sh/?id=10581155700) with basicConstraints extension that is not marked as critical. This is a violation of TLS BR 7.1.2.7.6
This is a preliminary report. We are investigating and will update with a full incident report as soon as possible.

Impact

Timeline

All times are UTC.

YYYY-MM-DD:

  • HH:MM Example

Root Cause Analysis

Lessons Learned

What went well

What didn't go well

Where we got lucky

Action Items

Action Item Kind Due Date
Example Prevent 2038-01-19

Appendix

Details of affected certificates

https://crt.sh/?sha256=[sha256 fingerprint of the certificate]

Based on Incident Reporting Template v. 2.0

Assignee: nobody → jozef.nigut
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] [ov-misissuance]

Certificate (https://crt.sh/?id=10581171523 ) was revoked on March 29 2024. New certificate has been issued. (https://crt.sh/?id=12513149633)
We provide full incident report this week.

Incident Report

Issuance of SSL/TLS certificates with Non-critical Basic Constraints.

Summary

Disig has been notified about a leaf certificate [https://crt.sh/?id=10581171523] and a corresponding precertificate [https://crt.sh/?id=10581155700] with basicConstraints extension that is not marked as critical.
This is a violation of TLS BR 7.1.2.7.6

Impact

Leaf certificate (https://crt.sh/?id=10581171523) and a corresponding precertificate (https://crt.sh/?id=10581155700)TLS certificates was issued September 1, 2023 with basicConstraints extension present but without the critical flag set.
After further investigation we found other 7 affected TLS certificates with the same issue:
[https://crt.sh/?SHA256=04F0D8E4BE06FD35E242DFB972B32491F49A374704A0B9D1F65436869D16316C]
[https://crt.sh/?SHA256=B463E197B5DA8766D28F267493403B7CC76233277CCE00342907CEE29DB3BCBE ]
[https://crt.sh/?SHA256=425438002FFCFFFF7FCCF703D54D4621DA04EBD4BDDE9A275A6236B4B60700D4]
[https://crt.sh/?SHA256=28BE42B0EA69B8D6F500173E63FDC9030223246F96F923F3AF74D39297A8EB76]
[https://crt.sh/?SHA256=583E5D94F691742B6CA3567A3109B0B76622A93EC94D3AA8078A138EC75746EB]
[https://crt.sh/?SHA256=08E5F1FFF84FE0A0DCF7F8E47AFA23D769E51A9B1FE2519A5EB1CD23DD0AA801]
[https://crt.sh/?SHA256=15017C718FBE2AB465C5C01380D1D7984FDBF46D5ED6DD44865298D53042681C]

It is important to note that there have been no actual disruptions to subscribers services due to the presence of the basicConstraints extension without the critical flag set in affected TLS certificates.

Timeline

All times are UTC.

2023-09-15:
Baseline Requirements for the Issuance and Management of Publicly‐Trusted Certificates Version 2.0.0 has become effective.

2024-03-26:
15:40 Disig has been notified by Rob Stradling <rob@sectigo.com> that issued leaf certificate [https://crt.sh/?id=10581171523] and a corresponding precertificate [https://crt.sh/?id=10581155700] with basicConstraints extension that is not marked as critical, which is non-compliant with TLS BR section 7.1.2.7.6.

2024-03-27:
7:17 Certificates were verified with pkilint and zlint. A violation of requirements TLS BR 7.1.2.7.6 Subscriber Certificate Extensions was confirmed.

7:18 The issuance of the TLS certificates was stopped

7:25 After being alerted to this incorrect release an incident was created internally and we immediately began investigating its causes.

10:30 Disig created a bug for this incident at bugzilla.mozilla.org [https://bugzilla.mozilla.org/show_bug.cgi?id=1888104]

11:30 Disig contacted the certificate subscriber by e-mail with a request to send a new request for the issuance of a TLS certificate, because the content of the existing TLS certificate is inconsistent with the requirements of TLS BR section 7.1.2.7.6 and will have to be revoked.

12:16 Disig informed Rob Stradling <rob@sectigo.com> about the creation of the bug by e-mail.

12:50 An analysis of the reasons for issuing a TLS certificate in violation of the requirements of TLS BR 7.1.2.7.6 was performed.
The analysis revealed that the TLS certificate in question was issued from a profile that includes the "serverAuth" extension in the EKU as well as the "clientAuth" extension, which is a rare combination that we issue for end clients, and that in this profile, unlike others two used, we forgot to remove the "basicConstraints" extension after version 2.0.0 BR came into effect (15.9.2023), which could have been used until the above date and at the same time could have been marked as nonCritical.

14:37 Customer replied, and started process to request new certificate.

15:05 CA manager prepared a change request with a corrective action. It was discussed with members of DB and internal support team.

17:42 Policy Management Authority approved suggested corrective action - remove basicConstraints extension from the corresponding TLS certificate profile

2024-03-28
7:49 TLS certificate profile with the EKU containing "serverAuth and clientAuth" has been modified and the basicConstraints extension has been removed from it, since according to BR 7.1.2.7.6 it may or may not be present in the TLS certificate.

7:55 The issuance of the TLS certificates was allowed.

13:48 Disig received a new request from the subscriber to issue a new TLS certificate for FQDN "video.vszp.sk".

14:12 A new TLS certificate was issued for the FQDN "video.vszp.sk" without the basicConstraints extension [https://crt.sh/?id=12513149633].
Certificate was verified with pkilint and zlint. No issues have been found.

14:21 A new certificate was sent to the subscriber.

20:58 Subscriber informed us that the new TLS certificate was successfully installed.

2024-03-29
8:13 TLS leaf certificate [https://crt.sh/?id=10581171523] was revoked.

2024-04-02:
07:15 We identified another seven (7) TLS certificates with same issue - see section Impact

07:30 We informed customers and started process to change certificates.

2024-04-04
9:38 TLS leaf certificate [https://crt.sh/?id=11403191015] was revoked.
9:42 TLS leaf certificate [https://crt.sh/?id=11403273891] was revoked.

To date, another four of the other affected certificates have been reissued and two of them (see above) were revoked. For the two remaining reissued we are waiting for customer confirmation of their deployment and then the affected certificates will be revoked.

Root Cause Analysis

The reason for issuing a TLS certificate with the basicConstraints extension marked as non-critical was a human error when implementing changes in connection with the changed requirements for the content of items in the TLS certificate for the end user (TLS BR 7.1.2.7.6) in the existing profiles for issuing TLS certificates introduced in our CA system.
All profiles were modified correctly and after version 2.0.0 Baseline Requirements for the Issuance and Management of Publicly‐Trusted Certificates (15.9.2023) came into effect, the basicConstraints extension was removed from them, with the exception of the profile containing both EKU serverAuth and EKU clientAuth.
The first TLS certificates issued after September 15, 2023 were tested using available pkilint and zlint tools and no errors were recorded in the issued TLS certificates.
In this case, however, it was TLS certificates issued from profiles that contained only EKU serverAuth, which were modified correctly, and which make up the majority of TLS certificates issued by us.
Due to the fact that all the issued TLS certificates from the most used profiles did not show any errors, we assumed that all modifications in the individual profiles were made in the same way.

Lessons Learned

We learned from this case that more consistency is needed when implementing changes in profiles to avoid possible human errors, as was the case in this case.
All the more so as we currently do not have any automated check of the issued TLS certificate in our system, which would notify us of inconsistency with the TLS BR even before the TLS pre-certificate is issued.

What went well

We were able quickly respond to incident and coordinate with customer to change the certificate.

What didn't go well

Due to a misconfiguration of one profile in our certificate issuing system, the basicConstraints extension was left in that profile even after TLS BR version 2.0.0 went into effect, without marking it as critical, which change in requirements for TLS BR changed compared to the previous one version TLS BR 1.8.7.

Where we got lucky

There have been no real disruptions to our customers' websites or online services due to the presence of the basicConstraints extension.
From this point of view, it is only a failure to comply with a relatively strict requirement, which in our view has no impact on the declared goal
of TLS BR "The primary goal of these requirements is to enable efficient and secure electronic communication and at the same time to address users' concerns about the trustworthiness of Certificates."

Action Items

| Action Item | Kind | Due Date |

| Reissue the remaining seven (7) TLS certificates containing the basicConstraints extension without marking them as critical | Mitigate | 2024-04-07 |
| Revoke the remaining seven (7) TLS certificates containing the basicConstraints extension without marking them as critical | Mitigate | 2024-04-07 |
|Implement into the system of issuing TLS certificates the possibility of checking the pre-certificate with appropriate tools even before its finalization and sending to the CT log, in order to prevent the issuance of a TLS certificate that
does not comply with the current BR TLS requirements.| Prevent | 2024-06-30

Details of affected certificates

See section Impact

Based on Incident Reporting Template v. 2.0

We have no additional updates for this bug at this moment.

One thing to consider here is that it took Disig a bit to identify the other 7 impacted certificates. What can be done here to speed that process up?

Implement into the system of issuing TLS certificates the possibility of checking the pre-certificate with appropriate tools even before its finalization and sending to the CT log, in order to prevent the issuance of a TLS certificate that
does not comply with the current BR TLS requirements

This is great, but the due date for this is way too far out in the future. This should be an absolute priority for your CA right now as it’s a feature that should’ve been there to begin with.

Please note, it is expected for CAs to follow the incidents here, and the various mailing lists that CAs follow. The importance of automatic linting has been well known for quite some time. So you may need to add ongoing monitoring Bugzilla and other forums to an action item.

There is no reason this linting system shouldn’t be in place by the end of this month. I’d potentially even go as far as saying that certificate issuances should be halted until some linting is in place.

Thank you for your comments, to which we have the following responses:

Regarding the "bit for identifying 7 additional affected certificates" - initially, we focused on resolving the issue with the certificate that we were alerted to, in order to meet all the deadlines outlined in the BR. Since we only had two days left until the Easter holidays and needed to address the issue with issuing a new TLS certificate, as well as its revocation, we decided to postpone the inspection of the remaining issued TLS certificates to the first working day after the Easter break. This decision was also influenced by the fact that including the basicConstraints extension in the TLS certificate without marking it as critical would not cause any issues for other holders when using them and would not pose any security threat to the PKI ecosystem, if they will be using such TLS certificate for a few more days.

Regarding the "possibility of pre-certificate control" - in this case, it's a preventive measure aimed at the future. We have solved the issue of including basicConstraint in the TLS certificate - there is no longer a profile in our system that contains the basicConstraints extension, so an external tool to detect it is unnecessary. Similarly, we have solved the issue of control the subject order field in the TLS certificates, where external tool inspection is currently unnecessary. Introducing the use of an external tool in the form of a linting system is to ensure any new changes in the BR, so we won't have to find our own solution for every possible BR change in the future. From this perspective, the deadline of June 30, 2024, seems acceptable to us.

there is no longer a profile in our system that contains the basicConstraints extension, so an external tool to detect it is unnecessary

Your stance here:

  1. Flies against the standard set by other CAs.
  2. Ignores that regressions happen in software.

So no, your perspective is wrong. The point of linting is not to detect issues of compliance of new requirements in the BRs. BR changes are not guaranteed to come with linter updates.

Since monitoring and triaging bugs from other CAs is necessary for being a CA, can you please inform us what led your CA to make this call? I’m interested in understanding the data you have backing your claims here.

Your stance here raises serious concerns about your understanding of the compliance system of WebPKI.

Flags: needinfo?(jozef.nigut)

I guess we didn't understand each other.
My answer regarding ""possibility of pre-certificate control" was related to this part of your post "There is no reason this linting system shouldn't be in place by the end of this month. I'd potentially even go as far as saying that certificate issuances should be halted until some linting is in place."
I did not comment on the part "Please note, it is expected for CAs to follow the incidents here, and the various mailing lists that CAs follow." and "So you may need to add ongoing monitoring of Bugzilla and other forums to an action item.".

I agree with you that it is necessary to monitor the incidents published here and try to react to them earlier. We registered the problem with the basicConstraint extension on this forum, but we were convinced that it did not concern us, since we made the necessary changes in the TLS certificate profiles immediately after the 2.0.0 BR version became effective. Unfortunately, it happened that in one profile, due to our fault, this change was not made in the end and this led to the issue of the TLS certificates involved in this incident.

In the future, we need to be more consistent in this, and of course, with immediate effect, we are introducing the obligation of daily monitoring of new incidents on this site.

In accordance with the requirements of the Mozilla Root Store Policy, Version 2.9, we monitor discussions daily at https://groups.google.com/a/mozilla.org/g/dev-security-policy
and https://groups.google.com/a/ccadb.org/g/public.

Thanks for the response, but let me make sure I understand this:

Are you adding pre and post issuance linting in the form of zlint, pkilint, etc? If so, is there an action item associated with this?

Yes, we plan to use zlint. We have opened a new issue for this in our internal gitlab. Our intention is to implement an automatic check in order to detect any TLS inconsistency before issuing the certificate itself.
The introduction of the automatic check before the release is also linked to the deadline set by us, which I believe our developers will be able to shorten.
Currently, we are only able to perform a manual check of the issued TLS certificate using the available tools zlint, pkilint, X.509 lint.

Is it this?

Implement into the system of issuing TLS certificates the possibility of checking the pre-certificate with appropriate tools even before its finalization and sending to the CT log, in order to prevent the issuance of a TLS certificate that
does not comply with the current BR TLS requirements.| Prevent | 2024-06-30

Please note, once you've signed a pre-cert, even before you've sent it to CT still counts as "post-issuance" linting. As in, signing a precertificate (even if you never sent it to CT) with a production key still counts as issuing that certificate.

To effectively do pre-issuance linting, you generally sign with a non-publicly trusted certificate (this can just be generated at runtime), run the linting on that, and then have your HSMs sign the precert afterwards.

Thank you for your valuable comments. Within the framework of the proposed solution, we had exactly the same goal.

Thanks, but I still think that the deadline you of 2024-06-30 is way too far out. This should be an absolute priority for your CA right now.

What is the reason this can't be done within a month? This is not an unreasonable request. The community expects CAs to have agility when responding to new requirements.

NSRs already require CA's to patch severe CVEs within the CA systems within 24 hours of finding out about them, and I know first hand that some of those patches are significantly harder to do in 24 hours compared to setting up a preissuance linting system in a month.

So to understand the delay:

  1. What other work are you prioritizing over this for your CA?
  2. What in your systems slow you down so much that it takes you months to add pre-issuance linting.

I understand that you said that the engineers can potentially get this done faster. I'm hoping you can commit to an earlier date.

If another incident happens between now and then, that could've been prevented by having linting setup, the community will not be happy with that.

Personally, I do not think that the rapid introduction of a linting system within the framework of issuing TLS certificates needs to be our priority at the moment. I will explain the reasons.

Both of the BR violations we encountered (this Bug1888104 and the Bug 1889672) have been resolved in our system to prevent them from happening again. The basicConstraint extension is no longer present in currently issued TLS certificates, and the order of the subject of the certificate is strictly controlled by the CA system to conform to BR requirements.

The accelerated introduction of the lintink system due to possible changes in BR requirements is also not of great importance at the moment, since the requirements are relatively stable after the dramatic changes during the transition from version 1.8.7 to version 2.0.0 and there will certainly be no changes in them until 30.6.2024.

As a member of the Server Certificate Working Group within the CA/Browser forum, we know, that CA/Browser forum currently has no plans to change the BR requirements. As part of the discussion, there is only one proposal regarding "Compromised and weak keys" and it is planned to be implemented, if the proposal passes, only on November 15, 2024.
We are also not aware that there should be major changes in the requirements of individual root programs that are not included in BR by 30.6.2024.

There are no updates this week, and we continue preparing to implement the TLS pre-certificate control system before its issuance.

There are no updates this week.

We have no new information in this bug.

Flags: needinfo?(jozef.nigut)

We have no new information in this bug.

We have no new information in this bug.

We have successfully completed the integration of zlint into our CA system, so that we are able to detect possible deviations from BR when issuing TLS certificates. We have successfully verified the entire system in our test environment. We are currently waiting for its implementation in the CA production environment. We will inform about deployment in production.

We have no new information in this bug.

We have successfully implemented zlint into our production CA system. This was the last corrective action for this bug. This fixes the issue with the basicConstraints extension not being marked as critical in the issued TLS certificate.

We have no new information in this bug. From our point of view, we consider the subject of this bug to be resolved and expect its closure.

I will close this on Wed. 10-July-2024, unless there are additional questions or comments.

Flags: needinfo?(bwilson)
Status: ASSIGNED → RESOLVED
Closed: 5 months ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.