Closed Bug 1865080 Opened 1 year ago Closed 11 months ago

Asseco DS / Certum: TLS EV certificates with incorrect Subject attribute order

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: aleksandra.kurosz, Assigned: aleksandra.kurosz)

Details

(Whiteboard: [ca-compliance] [ev-misissuance])

Attachments

(2 files)

Affected certificates
7.68 KB, text/plain
Details
Affected certificates
7.68 KB, text/plain
Details

Incident Report

This is a preliminary report.

Summary

During the weekly Bugzilla review, Certum's compliance team verified that Certum issued a number of EV TLS certificates with incorrect relative ordering of entity attributes, similar to bug 1864204. The error has been fixed and certificates issued after this fix have a correct relative order of attributes. Full incident report will be provided no later than November 23th.

Assignee: nobody → aleksandra.kurosz
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] [ev-misissuance]
Attached file Affected certificates —
## Incident Report ### Summary Certum has issued 138 EV TLS certificates since September 15, 2023, with an incorrect relative order of Subject attributes as defined in BR section 7.1.4.2. ### Impact After September 15, 2023, a total of 138 EV TLS certificates were issued with the incorrect relative order of Subject attributes. Upon identifying this error on November 16, 2023, Certum halted the issuance of EV TLS certificates. Subsequently, when the Subject attribute order was corrected, the certificate issuance process was resumed. All certificates affected by this mis-issuance have been revoked. ### Timeline All times are UTC +1. 2023-04-11: - BR TLS 2.0.0 was published 2023-07: - Certum Compliance Team analysed all certificate profiles that Certum uses and verified if they comply with the upcoming changes in BR for TLS 2.0.0. and BR for S/MIME 1.0.1. standards. S/MIME profiles had been inconsistent and required some changes, but no problems with TLS certificates were identified. 2023-09-15: - BR TLS 2.0.0. came into effect 2023-11-16: - 8:45 A Compliance team member investigated the Buypass bug during the weekly Bugzilla review, decided to examine Certum TLS certificates and discovered that some EV TLS certificates issued after September 15, 2023, had incorrect order of Subject attributes. - 9:30 Certum Compliance Team was informed about the mis-issuance - 9:40 Certum stopped issuing EV TLS certificates - 10:40 Certum identified all certificates affected - 11:30 Certum implemented the fix to the system, after successful tests on test environment - 11:45 Certum restarted issuance of EV TLS certificates - 12:30 Certum started notifying Subscribers about the problem. On 2023-11-16 all affected Subscribers were notified about the incident and were provided with instructions to apply for replacement certificates and information that the affected certificates had to be revoked. - 14:00 Certum informed its Conformity Assessment Body about the incident - 15:02 A preliminary incident report was registered in Bugzilla 2023-11-21: - Certum revoked all affected certificates 2023-11-23: - Certum published a full incident report ### Root Cause Analysis During the analysis of certificate profile requirements mentioned in BR v2.0.0. Certum concentrated on the relative order of subject attributes basing on OV TLS certificate. The Subject attributes in the OV TLS Certificate Profile was correct according to BR v2.0.0. ### Lessons Learned #### What went well * Immediate actions were taken when the error was identified * All certificates involved in the incident were revoked in time * Weekly Bugzilla review helped us to identify the problem #### What didn't go well * The analysis of the BR v2.0.0. requirements obviously didn’t go well, the analysis did not cover all possible release scenarios, it focused only on OV certificates * Identifying the problem 2 months after the requirements came into effect * The linter that Certum was using (z-lint) did not focus on the order of Subject attributes as defined in BR 2.0.0. and did not report any issue #### Where we got lucky * The problem with attribute relative order appeared only in some of the EV TLS certificates with optional fields ### Action Items | Action Item | Kind | Due Date | | ----------- | ---- | -------- | | Include pkilint in the certificate issuance process | Prevent |2024-04-01 | | Update of the internal instructions for new ballot revision | Prevent |2023-12-31 | #### Details of affected certificates
Attached file Affected certificates —
## Incident Report ### Summary Certum has issued 138 EV TLS certificates since September 15, 2023, with an incorrect relative order of Subject attributes as defined in BR section 7.1.4.2. ### Impact After September 15, 2023, a total of 138 EV TLS certificates were issued with the incorrect relative order of Subject attributes. Upon identifying this error on November 16, 2023, Certum halted the issuance of EV TLS certificates. Subsequently, when the Subject attribute order was corrected, the certificate issuance process was resumed. All certificates affected by this mis-issuance have been revoked. ### Timeline All times are UTC +1. 2023-04-11: - BR TLS 2.0.0 was published 2023-07: - Certum Compliance Team analysed all certificate profiles that Certum uses and verified if they comply with the upcoming changes in BR for TLS 2.0.0. and BR for S/MIME 1.0.1. standards. S/MIME profiles had been inconsistent and required some changes, but no problems with TLS certificates were identified. 2023-09-15: - BR TLS 2.0.0. came into effect 2023-11-16: - 8:45 A Compliance team member investigated the Buypass bug during the weekly Bugzilla review, decided to examine Certum TLS certificates and discovered that some EV TLS certificates issued after September 15, 2023, had incorrect order of Subject attributes. - 9:30 Certum Compliance Team was informed about the mis-issuance - 9:40 Certum stopped issuing EV TLS certificates - 10:40 Certum identified all certificates affected - 11:30 Certum implemented the fix to the system, after successful tests on test environment - 11:45 Certum restarted issuance of EV TLS certificates - 12:30 Certum started notifying Subscribers about the problem. On 2023-11-16 all affected Subscribers were notified about the incident and were provided with instructions to apply for replacement certificates and information that the affected certificates had to be revoked. - 14:00 Certum informed its Conformity Assessment Body about the incident - 15:02 A preliminary incident report was registered in Bugzilla 2023-11-21: - Certum revoked all affected certificates 2023-11-23: - Certum published a full incident report ### Root Cause Analysis During the analysis of certificate profile requirements mentioned in BR v2.0.0. Certum concentrated on the relative order of subject attributes basing on OV TLS certificate. The Subject attributes in the OV TLS Certificate Profile was correct according to BR v2.0.0. ### Lessons Learned #### What went well * Immediate actions were taken when the error was identified * All certificates involved in the incident were revoked in time * Weekly Bugzilla review helped us to identify the problem #### What didn't go well * The analysis of the BR v2.0.0. requirements obviously didn’t go well, the analysis did not cover all possible release scenarios, it focused only on OV certificates * Identifying the problem 2 months after the requirements came into effect * The linter that Certum was using (z-lint) did not focus on the order of Subject attributes as defined in BR 2.0.0. and did not report any issue #### Where we got lucky * The problem with attribute relative order appeared only in some of the EV TLS certificates with optional fields ### Action Items | Action Item | Kind | Due Date | | ----------- | ---- | -------- | | Include pkilint in the certificate issuance process | Prevent |2024-04-01 | | Update of the internal instructions for new ballot revision | Prevent |2023-12-31 | #### Details of affected certificates

We have no additional updates for this bug.

We have no additional updates for this bug. If there are no questions are we fine to close this bug?

I will close this on next Wed., 13-Dec-2023.

Flags: needinfo?(bwilson)

All certificates involved in the incident were revoked in time

I opened three random certificates from the list of impacted certificates.

Assuming we start counting from 8:30 UTC (9:30 UTC+1) 2023-11-16

It seems like none of them were revoked in the 120 hour time frame?

If that is the case, then Asseco/Certum will need to file another bug to address the delayed revocation.

Our understanding was based on counting full days, and as per our records, the problem was acknowledged on 16.11.2023. We considered the full day of 21.11.2023 and subsequently revoked all certificates on that day, meeting the 5-day baseline requirement. The revocations were completed on 21.11.2023.

If there is a specific reference to 120 hours that we may have missed, we would appreciate clarification. As it stands, we believe we followed the correct procedure according to the information available.

If you think we need to report a new issue about this, we're okay with that. In that case, we would recommend adding the specific 120-hour requirement to the baseline instructions so that everyone understands it better in the future.

If there is a specific reference to 120 hours that we may have missed, we would appreciate clarification.

https://github.com/cabforum/servercert/blob/main/docs/BR.md#4911-reasons-for-revoking-a-subscriber-certificate

Given the ambiguity of the language in section 4.9.1.1 of the BRs and its susceptibility to an interpretation that "day" could mean a calendar day, I'm inclined to not require that an additional bug be filed.

Is there any precedent for that interpretation/ambiguity? I don't think I've ever come across that view into what "day" means.

Considering the BRs specifically specify hours for the 24 hour section in that same section, I don't think its reasonable to interpret the BRs as "calendar day".

The Baseline Requirements could have been written to say 120 hours, but they weren't. I suppose we ought to look at the definition of "day" and determine whether it runs from midnight to midnight and if so, then for what time zone?

Let's look at the sentence in the BRs:

With the exception of Short-lived Subscriber Certificates, the CA SHOULD revoke a certificate within 24 hours and MUST revoke a Certificate within 5 days and use the corresponding CRLReason (see Section 7.2.2) if one or more of the following occurs:

Why are we over-complicating this definition with timezones?

The same sentence specifies 24 hours as a SHOULD, and 5 days as a MUST. It is not only reasonable to read 5 days as 24*5, but I'd go as far as saying it is unreasonable to read 5 days and try to bring in timezones into it. Days here is a mark of duration, and not a mark of relativity to your jurisdiction.

If there is no precedent reading this line as "5 but also maybe more" days, then I strongly disagree with introducing that precedent here.

Here's some historical discussions that had happened over the definition of time: https://bugzilla.mozilla.org/show_bug.cgi?id=1715455

Looking at the BR's closely, there is this section that also defines a day as 86,400 seconds: https://github.com/cabforum/servercert/blob/main/docs/BR.md#632-certificate-operational-periods-and-key-pair-usage-periods

(In reply to Ben Wilson from comment #12)

The Baseline Requirements could have been written to say 120 hours, but they weren't. I suppose we ought to look at the definition of "day" and determine whether it runs from midnight to midnight and if so, then for what time zone?

Given the context, I'm struggling to see the sense in that interpretation. Why would the clock start ticking at midnight? That would mean the 24 hour clock starts at one point in time and the 5 day clock at another. Would it start the following midnight or the preceding midnight?

Interpreting it as an interval, it seems pretty clear: 5 * 24 86400 seconds. If the clock starts ticking at 1:23 AM UTC on Monday, the interval has lapsed at 1:23 AM UTC on Saturday. That's the only interpretation that doesn't give rise to ambiguity, fits with the context, and aligns with definitions given elsewhere in the BRs.

At the time of https://bugzilla.mozilla.org/show_bug.cgi?id=1619179#c3, it was clear that the 5-day revocation window ended at the same time of day as it began (03:00 UTC in that case). In other words, four years ago it was undisputed that the revocation window was 120 hours. The phrasing of this section of the BRs has not changed since that time.

Does Asseco/Certum have responses to the points raised in Comments 13-16?

Flags: needinfo?(bwilson) → needinfo?(aleksandra.kurosz)

You can also find some relevant discussions in the following CA/B Forum threads from 2021 related to ballot SC-52:

We understand presented arguments and agree that our interpretation of the 5 days was incorrect. However, we would like to draw attention to the fact that this term is not precise and may be misleading, because one entry in the BR refers to hours and the other to days, which may indicate that these dates should be treated differently.
We have created a bug for the delay in revocations: https://bugzilla.mozilla.org/show_bug.cgi?id=1871393

Flags: needinfo?(aleksandra.kurosz)

FWIW I agree with Ben's interpretation that the BRs are silent on the exact duration of 5 days. During the discussion for SC-52 (which didn't make progress as no consensus was reached), the SCWG made a clear decision to add specific duration (in seconds) for specific requirements, namely the "validity" and OCSP nextUpdate. The SCWG did not expand the 1'' specificity for any time interval (day, 5 days, 1 month, 3 months, 1 year, etc).

I agree that CAs should err towards the strictest interpretation of those time intervals but the current language could be interpreted to mean 5 calendar days instead of 5x24 hours or 432000 seconds.

Because Bug #1871393 has been opened, I'll close this matter.

Flags: needinfo?(bwilson)

During the discussion of Ballot SC-52, Tim Hollebeek said:

As a practical matter, though, if you don’t measure 24 hours or five days to revoke a certificate down to the second, you’re already in trouble, as plenty of incidents have already been filed for missing revocation deadlines by fairly small time periods

Additionally, discussions of Ballot SC-52 during CABF meetings make it clear that participants considered the ballot to be a clarification, not a change, albeit a clarification that many CAs might not be expecting:

Clint: I tend to agree with you Tim just having this out of the way and clarified seems worthwhile.

Most of the objections to precise measurements of time are regarding scheduling recurring tasks, like audit reports.

Dimitris: The problem is that the tools that are out there whenever you need to schedule something, and that was exactly the reason why I switched from a specific number of days in Network security requirements to months was preciously to avoid that. For example, you calculate that you need to execute a script once a month at the first day of every month, sometimes it will be 31 days sometimes 30 and that is ok, but if you want to be precise to the days it creates difficulties in many different systems.

These are already handled by building some extra days into the relevant deadlines: validation documents being valid for 398 days instead of 365, audit reports not being due until 90 days after the audit period ends, etc.

I agree that the BRs need to be clearer here so that incidents like this don't reoccur. But claiming that the BRs actually allow validation documents to be used 398.5 days after they're obtained, or that random values can be used for 30.5 days, or that CRLs can be re-issued every 4.5 days, or that revocations to happen 5.5 days after they're requested, (i.e. claiming that this is not in fact an incident) appears ahistorical.

To provide greater clarity in the BRs, I have created https://github.com/cabforum/servercert/compare/90a98dc7c1131eaab01af411968aa7330d315b9b...c3e928e73caed8c8489ab5406127aad661b8a63e, and will ask for endorsers to make it a full ballot.

We will need to make things a bit more specific because this "catch-all" definition that extends to NetSec was the problematic point during SC-52. Specifying the period of a day to 86400 seconds and extend that to all the time measurements of "day" (1 day, 2 days,.... 5 days, 30 days, 398 days), is ok. We must be careful how to interpret weeks, months, years and not to link the accuracy of the "day" to those other cases (week, month, year).

I prefer to continue this conversation at the SCWG but you can count me in as a supporter of such a ballot.

We have no additional updates for this bug.

Unless there are any objections, I will close this item on Wed. 3-Jan-2024.

Status: ASSIGNED → RESOLVED
Closed: 11 months ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: