Closed Bug 1896108 Opened 2 years ago Closed 2 years ago

Telia: Certificates Issued with lower case value in subject:countryName

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: antti.backman, Assigned: antti.backman)

Details

(Whiteboard: [ca-compliance] [ov-misissuance])

Initial Incident Report

This is the initial incident report, full detailed report shall be provided by Telia CA by 2024-05-17 13:00 UTC.

Summary

Telia CA mississued thirteen (13) TLS certificates with subject:countryNamein lower case letters. This is a violation of CA/Browser Forum Baseline Requirements for the Issuance and Management of Publicly‐Trusted TLS Server Certificates (TLS BR) section 7.1.2.7.4 Organization Validated for subject:countryName attribute format. The said requirement requires that the attribute is represented in the certificate Subject by two-letter ISO 3166-1 alpha-2 code represented by two (2) LATIN CAPITAL letters LETTER A through LATIN CAPITAL LETTER Z.

Telia CA wish to thank you the reporter for bringing this issue to our attention, remedy and as an opportunity to further strengthen our processes and practices to prevent certificate misissuance in the future.

Impact

Thirteen (13) end entity certificates were misissued with subject:countryNamein lower case. The affected certificates shall be revoked by Telia CA and subscribers are instructed to renew the certificates with immediate effect.

Telia CA estimates that it is unlikely possible to revoke all affected certificates within the 5 days time frame as required by TLS BR in section 4.9.1.1 / §12, whithout causing undue harm to the subscribers. If the deadline would be missed, based on the minor issue on the certificates, Telia CA estimates the risk of not being able to revoke in 5 days lower than possible issues to the subscribers if Telia CA would forcefully revoke the certificates.

If the 5 days deadline (May 14th 2024 00:17 UTC) would be exceeded Telia CA will post additional incident report of possible delayed revocation. Telia CA works to the best possible effort with the affected certificate holders / subscribers to meet the deadline to successfully revoke all certificates within the required 5 days time frame by May 14th 2024 00:17 UTC.

As the data in the subject:countryNameis properly validated and indicating correct Country of the certificate Subject, Telia CA considers that the existing certificates may be trusted by the respective certificate consumers as long as certificate holders / subscribers have been able to renew the certificates to minimize the impact to the services protected by the certificates.

Timeline

All times are UTC.

Date and time Description
Jan 22 14:28:16 2024 First certificate with subject:countryName malformed was issued
Jan 23 07:35:28 2024 Second certificate with subject:countryName malformed was issued
Jan 23 07:37:09 2024 Third certificate with subject:countryName malformed was issued
Jan 23 07:57:21 2024 Fourth certificate with subject:countryName malformed was issued
Jan 23 09:13:35 2024 Fifth certificate with subject:countryName malformed was issued
Jan 23 09:18:34 2024 Sixth certificate with subject:countryName malformed was issued
Jan 23 10:10:46 2024 Seventh certificate with subject:countryName malformed was issued
Jan 23 11:00:27 2024 Eight certificate with subject:countryName malformed was issued
Jan 23 13:23:44 2024 Ninth certificate with subject:countryName malformed was issued
Jan 23 13:27:11 2024 Tenth certificate with subject:countryName malformed was issued
Jan 23 13:28:11 2024 Eleventh certificate with subject:countryName malformed was issued
Jan 24 08:07:17 2024 Twelvth certificate with subject:countryName malformed was issued
Jan 24 09:00:41 2024 Thirteenth certificate with subject:countryName malformed was issued
May 09 00:17 2024 Email report from Ryan Daurne (the email was signed by -Wayne) received with list of problematic certificates (SHA-256) to Telia's problem reporting email ca-problems@telia.fi. <br />B2604755F06B9EA8B033E3B5A07EE2338789C9959210DEB3C97EF77F7EA81619 <br />4F58CF4B9E3D1C218A7A7E92654865C4683D81AA80E4DFF70D6AAEDDD07BF055 <br />C358AE8CA1E02D28EF8E8843AC7B9526BE195294A491879A4F9BAD104DA9E225 <br />4CDFDBE5736E179AF00D16CCF1D4679C3C856B0E303B765AB9903CC52425602B <br />EC5CB482143D7A57F8B16289525F7AC71BC143D693A48F1AB22F8C8F6650ADB2 <br />1E8B460171834832A47A15B9736E59CEB81F7A665C52FB39697830C17377686C <br />9F665A0E37D8AAEE3E23E0796DCCBE9A29B4BD4C7DDA5357F79300557759FD09 <br />6C6DCD1E38AC443BCDFE58111AACCFF1A9BB45EAE025CA10618DB14C387AC5CE <br />D84B1D54143A5A15D05CBB178382EEA187ED6128609A0C689CE886FE09CB035D <br />734338F9C43CA2810543557D5ED711FBF70C22CAF7120D831867289195724B67
May 09 17:11 2024 Telia CA Security Manager responded to the problem report via Telia CA's support email (cainfo@telia.fi) to acknowledge the reception of the problem report.
May 10 03:50 2024 Telia CA's Security Manager initiated internal investigation process and started to collect evidence data and prepare incident reports.
May 10 04:58 2024 Telia CA Security Manager called upon CA PKI Security Board supplemental meeting to initally address the incident and verify the incident as violation of requirements as indicated in this incident report.
May 10 06:30 2024 Telia CA Security Board conviened to address the matter and assigned Telia CA Security Manager and designated Telia CA Administrators for further investigation of the issue in order to produce and compile full incident report as required.
May 10 07:40 2024 Externally reported ten (10) Certificate holders / subscribers were informed by Telia CA Security Manager of the details and the need to renew the certificates concerned with immediate effect as the existing certificates shall be revoked. Telia CA will revoke the existing misissued certificates immediately after replacing certificates have been issued by the certificate holders / subscribers.
May 10 08:25 2024 Complete review of all issued certificates completed to verify if additional misissued certificates of similar kind would be existing by designated Telia CA Administrator.<br />Completed review confirmed that the certificates listed in this incident report were the only ones with malformed subject:countryName<br />Additional three (3) certificates with the same issue found and included in this report.
May 10 08:43 2024 Additional three (3) certificate holders / subscribers informed after internal review of all certificates found those additional certificates with the same issue.
May 10 08:53:47 2024 Certificate https://crt.sh/?id=11840646747 revoked after renewed by certificate holder with a new one.
May 10:12:13 2024 Certificate https://crt.sh/?id=11841121388 revoked after renewed by certificate holder with a new one.
May 10 13:36 2024 Telia CA Security Manager created the intial incident report for the issue.

Root Cause Analysis

Detailed root cause analysis will provided with the full report. Based on initial investigation Telia CA has clear understanding of the Root Cause and full details will be provided with the final report.

Based on Telia CA's initial investigation, Telia CA would like to notify the community of the fact that PKI linters used by Telia CA do not indicate an issue with lower case in subject:countryName. Telia CA is using zLint and pkilint for certificate data linting both real-time during the issuance process prior the the final certificate is issued and on daily basis to lint all issued certificates for the given day.

With above observation, Telia CA acknowledges our responsiblity as a CA to be solely responsible for the contents of the data in the certificates regardless if technical means are used to aid in conformance verification during the certificate issuance process, thus the above observation is not to imply that referenced tools would be at fault in this case. Merely Telia CA wishes to inform the authors of the tools and the PKI community, if it would be possible to consider futher enhancement to detect this type of issue and help the users of the linting tools going forward.

Lessons Learned

What went well

What didn't go well

Where we got lucky

Action Items

Action Item Kind Due Date
Inform affected certificate holders / subscribers to renew their certificate and report back to Telia CA when new certificates are installed. Action required 2024-05-10, COMPLETED
Perform full review of the issued certificates, if additional certificates would be found with the same issue. Investigate 2024-05-10, COMPLETED
Certificate https://crt.sh/?id=11840646747 revoked after renewed by certificate holder with a new one. Action (revocation) 2024-05-10 08:53:47 UTC
Certificate https://crt.sh/?id=11841121388 revoked after renewed by certificate holder with a new one. Action (revocation) 2024-05-10 10:12:13 UTC
Submit initial incident report to the Mozilla Bugzilla Action 2024-05-10
Revoke misissued certificates after certificate holders / subscribers have been able to renew the certificates Action After new certificates have been issued successfully by the certificate holders / subscribers.
Submit full incident report to the Mozilla Bugzilla Action 2024-05-17

Appendix

Details of affected certificates

Certificates on the list marked with x found in Telia CA's full review of certificates.

  1. https://crt.sh/?id=11832169804 (Not Before: Jan 22 14:28:16 2024 GMT)
  2. https://crt.sh/?id=11839864048 (Not Before: Jan 23 07:35:28 2024 GMT x)
  3. https://crt.sh/?id=11839878201 (Not Before: Jan 23 07:37:09 2024 GMT)
  4. https://crt.sh/?id=11840035891 (Not Before: Jan 23 07:57:21 2024 GMT)
  5. https://crt.sh/?id=11840646747 (Not Before: Jan 23 09:13:35 2024 GMT)
  6. https://crt.sh/?id=11840688066 (Not Before: Jan 23 09:18:34 2024 GMT)
  7. https://crt.sh/?id=11841121388 (Not Before: Jan 23 10:10:46 2024 GMT x)
  8. https://crt.sh/?id=11841544979 (Not Before: Jan 23 11:00:27 2024 GMT x)
  9. https://crt.sh/?id=11842684205 (Not Before: Jan 23 13:23:44 2024 GMT)
  10. https://crt.sh/?id=11842698882 (Not Before: Jan 23 13:27:11 2024 GMT)
  11. https://crt.sh/?id=11842711133 (Not Before: Jan 23 13:28:11 2024 GMT)
  12. https://crt.sh/?id=11851026130 (Not Before: Jan 24 08:07:17 2024 GMT)
  13. https://crt.sh/?id=11851399778 (Not Before: Jan 24 09:00:41 2024 GMT)
Whiteboard: Next update by 2024-05-17 13:00 UTC

The said requirement requires that the attribute is represented in the certificate Subject by two-letter ISO 3166-1 alpha-2 code represented by two (2) LATIN CAPITAL letters LETTER A through LATIN CAPITAL LETTER Z

Can you provide a reference for this requirement? There are requirements in ISO 3166-1 for the assignment of values, but the document is seemingly silent on the accepted representations of such values.

Hi Corey,

Thank you for your question,

There's no such requirement in the RFC 5280 nor TLS BR, but in the ISO 3166-1 standard it is said to the letter as reported by us and as you have referenced in your comment. So as TLS BR requires subject:countryName to be ISO 3166-1 alpha-2, our interpretation is that consequently the country code should be represented in the subject:countryName in capital letters. Obviously we're willing to hear from the community if we would have made incorrect interpretation of the issue.

(In reply to Corey Bonnell from comment #1)

The said requirement requires that the attribute is represented in the certificate Subject by two-letter ISO 3166-1 alpha-2 code represented by two (2) LATIN CAPITAL letters LETTER A through LATIN CAPITAL LETTER Z

Can you provide a reference for this requirement? There are requirements in ISO 3166-1 for the assignment of values, but the document is seemingly silent on the accepted representations of such values.

While I can't say where Telia's precise interpretation comes from I'll quote ISO 3166-1 Chapter 5.1:

5.1 Code element formation
Code element shall consist only of the following characters (cf. ISO/IEC 10646):
— LATIN CAPITAL LETTER A through LATIN CAPITAL LETTER Z
— DIGIT ZERO through DIGIT 9
— HYPHEN-MINUS
Code elements alpha-2 and alpha-3 are formed with LETTERS, num-3 is formed with DIGITS.
Code elements alpha-4 are formed with LETTERS.
Code elements for subdivisions are formed with up to 6 characters: alpha-2, HYPHEN-MINUS, and 1 up to 3 LETTERS or DIGITS.

This incident is one of the can of worms I mentioned in #1894560. There are a lot of standards built into the baseline requirements that have precise interpretations on casing.

Thank you for investigating further and finding an additional 3 certificates. Also unless explicitly agreed I would recommend not including researchers' names in reports, especially when it has no relevance, however I understand this is a common practice in this community.

Hi Wayne

Your reference is the same I was referring to in my response to Corey's question.

And duly noted and sorry for explicitly citing names on the report. We'll revert from disclosing such individual details in the future.

Thanks for the prompt replies, Antti and Wayne.

My understanding is that the cited requirement in section 5.1 is to ensure that a standard, canonical representation is applied for the assignment of alpha-2 country codes. Section 5 ("Principles for assigning code elements") and its subsections do not establish any restrictions on the allowed forms in which such country codes can be expressed, but rather only on the assignment of such codes.

Given the lack of any restriction in ISO 3166-1 on the allowed forms of such country codes and the lack of any specific encoding requirements in X.520, RFC 5280, or the BRs in regard to the countryName attribute, I do not believe that using non-canonical casing is a clear violation of the relevant requirements. If such a restriction on the BR profile is desired, the BRs should be amended to explicitly specify the use of uppercase letters when encoding ISO 3166-1 alpha-2 country codes.

Hi Antti, Wayne, and Corey

(with personal capability)

I would like to provide different perspective for ISO 3166-1.
There are many version of ISO3166-1. In my hand, I only have two variant of them, but let me provide information out of that.

ISO3166-1:2020(newest one), “Section 5.1Code Element Formation” state that code element formation shall be consist of only Capital letter, Digit, and HYPHEN-MINUS, as referred by Wayne.
On the other hand, Section 5.1 of ISO3166-1:2006 (which was not available as is, so I checked the Japanese translation, JIS X 0304:2011, which I had on hand) does not mention such usage on 5.1, so I interpret this as not necessarily prohibiting the use of uppercase Country Codes.
Although both the 2020 and 2006 editions have descriptions in Section 5.3 on how to "Construct" a Country Code, but it is only construction of Code. Only the 2020 edition mentions using capital letters when forming it.

Based on the above as the background, and

  • The current BR references 3166-1, but doesn’t specify which version does that.
  • From BR v.1.0 (Effective Date of 1 July 2012), references to 3166-1 have continued to exist and unchanged.
  • I believe we did not have discussion on CABForum about change of version of 3166-1

I believe there might be some ambiguity of which 3166-1 we should refer.

I agree there is ambiguity especially given the timeframe of ISO 3166-1 being updated. I was concerned at additional elements being added in :2020, but did not have :2006 on-hand so thank you for clarifying.

This is why during the discussion in #1894560 I said a can of worms opens up on how you interpret the ISO/RFC/etc standards in the baseline requirements. Most of the time it's a vague reference that this other standard exists, but with no specificity on which version is applied and as noted in that bug we're at the stage of divergence from the standards as-written entirely.

The other headache is that the following comment puts the subject of conformity of the fields in general as an element per the Chrome Root Program:

The above interpretation (the values are case sensitive) appears both widely and reliably interpreted by other EV-issuers.

Even if there is ambiguity in the standard as originally written, we have the issue of it becoming a strict format by proxy of enough CAs following a specific format prior to any ballots. There is a reason I looked at country code's solely, realized the extent of the issues that could appear and stopped further research. It is not an interpretation I am comfortable with either, but these issues will get the questions put to the community far more effectively than a hypothetical comment.

I am feeling referring ISO is very tricky and I believe I need to discuss on CAB forum how to deal with that, and I am happy to do so.
There is some challenge like... the number of CA who had bought 2020 is limited, there is some restriction on using that document even I purchased... but I believe we can deal with that.

we have the issue of it becoming a strict format by proxy of enough CAs following a specific format prior to any ballots.
We need evidence, and justification to revoke certificate, to explain customers.
Revoking certificate prior to any ballots (or discussion) is potentially cause much more trouble for people, and I feel that is not right way to go.

sorry, I missed line-break on last paragraph.

we have the issue of it becoming a strict format by proxy of enough CAs following a specific format prior to any ballots.

We need evidence, and justification to revoke certificate, to explain customers.
Revoking certificate prior to any ballots (or discussion) is potentially cause much more trouble for people, and I feel that is not right way to go.

I agree with Tadahiko's concerns about explaining this to customers, and I don't see it explicitly mentioned in BR section 7.1.2.7.4, which only says, "The two‐letter ISO 3166‐1 country code for the country associated with the Subject." But having country codes in all caps also appears to be a universal practice (if not an ISO requirement), so even if versions of 3166-1 don't expressly say "uppercase", etc., it appears that ISO only uses the codes as uppercase in all of its official ISO documentation and lists. So, I agree with Telia and its conclusion that country codes must be in uppercase.

Assignee: nobody → antti.backman
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: Next update by 2024-05-17 13:00 UTC → [ca-compliance] [ov-misissuance] Next update by 2024-05-17 13:00 UTC

(In reply to Wayne from comment #7)

The other headache is that the following comment puts the subject of conformity of the fields in general as an element per the Chrome Root Program:

The above interpretation (the values are case sensitive) appears both widely and reliably interpreted by other EV-issuers.

Even if there is ambiguity in the standard as originally written, we have the issue of it becoming a strict format by proxy of enough CAs following a specific format prior to any ballots. There is a reason I looked at country code's solely, realized the extent of the issues that could appear and stopped further research. It is not an interpretation I am comfortable with either, but these issues will get the questions put to the community far more effectively than a hypothetical comment.

I think this is important, but I would characterize the feedback from the other bug differently. Root programs have requirements that go beyond just the BRs. CAs are also responsible for following their stated practices and for validating all the information that they include in their certificates. Did Telia intend to issue these 13 certificates differently from all the other OV certificates that they issue? If their system behaved unexpectedly, that indicates an operational or procedural issue.

Even if issuance is determined to be a near miss in terms of strict BR compliance, I think the expectation should be that the CA provides an incident report for analysis of what caused the issue and action items to prevent similar issues.

Hi,
Thank you for the comments and discussions on this topic / incident so far.

Based on these comments, it could be something to be followed-up in the CA/B to further improve the language in the BRs?

To Matthew

Even if issuance is determined to be a near miss in terms of strict BR compliance, I think the expectation should be that the CA provides an incident report for analysis of what caused the issue and action items to prevent similar issues.

We will address this incident as we have indicated in the initial incident report and file in full incident report in due time.

Whiteboard: [ca-compliance] [ov-misissuance] Next update by 2024-05-17 13:00 UTC → [ca-compliance] [ov-misissuance] Next update 2024-05-17 13:00 UTC

This is to update the revocation of additional four (4) certificates reported on this incident:

  1. Certificate https://crt.sh/?id=11851026130 revoked after renewed by certficate holder with a new one. May 13 06:36:28 2024 UTC
  2. Certificate https://crt.sh/?id=11840035891 revoked after renewed by certificate holder with a new one. May 13 09:19:08 2024 UTC
  3. Certificate https://crt.sh/?id=11832169804 revoked after renewed by certificate holder with a new one. May 13 09:41:20 2024 UTC
  4. Certificate https://crt.sh/?id=11839864048 revoked after renewed by certificate holder with a new one. May 13 10:07:59 2024 UTC

Further this is to notice about delayed revocation incident for the remainder of the certificates affected by this incident. As anticipated in the initial incident report, all affected certificates could not be revoked in 5 days.

Telia continues to follow up this incident and report progress by Next Update.

Corey is right, this isn't a violation of any requirement that currently exists. pkilint is correct in not flagging this.

If people want countries to be consistently expressed in uppercase, I would support that ballot. However, the sort of logic being used to justify this as an error requires a lot of interpretation and guessing that really has no place in a compliance analysis.

In fact, if you type "Are country codes case sensitive?" into Google Search, it will tell you no, and cite this document:
https://docs.edgecast.com/pci-cdn/Content/Reference/Country_Codes.htm

Now, it's debatable that this answer is correct, but I think that's actually the correct assessment. If I were a lawyer, I could probably argue this one both ways. And in the case of ambiguity, we should fix it, not punish those who chose an interpretation that is retroactively classified as "wrong". Even if the CA itself classifies it that way. Because other CAs are affected by the precedent, too (we aren't, but there might be other CAs that are).

If I can't find this invisible requirement that nevertheless is so important that its worth a compliance incident, and Google and various documents on the internet contradict it, how is a non-native speaker at a small CA supposed to handle compliance with this non-requirement? Sometimes, we're too fond of our clever arguments to see the obvious truth. None of the documents we have currently provide any guidance about what's right here. The solution is to fix the requirements and then fix it going forward.

-Tim

(In reply to Tim Hollebeek from comment #14)

The logic used to support the uppercase interpretation is: reading the ISO document referenced in the baseline requirements. I sincerely hope Digicert do not perform compliance interpretation by searching google and reading how edgecast, a completely unrelated party, handle case sensitivity in their own products that do not have any conformity requirements. If a page exists on the edgecast site mentioning that a hour is defined as 100 minutes, we could all agree it would not impact how the BR is interpreted right?

If you have been following the discussion then you'll be aware there is a distinction between the 2006 and 2020 versions of ISO 3166-1. The sole reason this even came into review was due to #1894560 where at least one Root Program said that case-sensitivity mattered on a fixed-string object, overriding the X.520 specification itself which said it was case-insensitive.

I have no issue at all with the BRs being updated to reflect either interpretation. In fact the issue above and beyond the Country Name aspect is that the BRs themselves bring in these ISO/RFC standards but make no attempt to clarify which aspect they'll override on a whim. If we wish to overanalyse what the original writers' intentions were, then we must question why they even referenced the standards to begin with.

I entirely agree that the BR needs to be clearer on these aspects. For transparency this was what was sent to Telia:

I've noticed for 10 of your certificates the Country Name is listed as 'se' lowercase. This isn't following ISO 3166 Country Code formats and is an outlier for Certificate Issuers.

The same wording was used for SECOM. Any aspect of revocation is a decision the CA made by themselves. A request to post an incident was made in the CPR given this would undoubtedly be a topic of discussion within the community. However I would note that being an outlier has been grounds for a Root Program to consider a misinterpretation as non-compliance, thus the conformity checks.

On the topic of invisible requirements and 'various documents on the internet contradict it'... this is a fundamental misunderstanding of reading regulatory guidelines and contracts in general. The terms defined in that specific document and how they are used internally are all that matter, not what an unrelated party writes. To this effect the only authoritative parties involved would be the BR and the ISO standard, which is incorporated by reference.

If you have been following the discussion then you'll be aware there is a distinction between the 2006 and 2020 versions of ISO 3166-1.

While I agree that the 2020 version of ISO 3166-1 provided additional clarity on the assignment of alpha-2 country codes, both versions do not specify any requirements on how such codes must be expressed. Given this, I am at a loss how a change in verbiage in a sub-section of a section entitled "Principles for assigning code elements" can be construed as a universal normative restriction on how such codes are represented.

The fact of the matter is that none of the relevant documents (ISO 3166-1, X.509, RFC 5280, or the TLS BR) establish any normative requirement to use uppercase letters and thus lowercase/mixed case country codes are acceptable. To assert otherwise is to create an interpretation that is not based in the written requirements but rather on some abstract, subjective concept.

There was also an assertion raised that since the ISO documents themselves use uppercase, the only acceptable form of the countryName attribute value is uppercase. The list in ISO 3166-1 using the uppercase form is a natural consequence of the document being self-consistent with the assignment guidance in section 5. Looking beyond the world of X.509, it largely universal practice to represent domain names using lowercase. Many would it find it quite surprising that a ccTLD domain name expressed in lowercase does not actually contain a valid ISO 3166-1 alpha-2 country code, which this assertion seemingly is making.

If I can't find this invisible requirement that nevertheless is so important that its worth a compliance incident, and Google and various documents on the internet contradict it, how is a non-native speaker at a small CA supposed to handle compliance with this non-requirement? Sometimes, we're too fond of our clever arguments to see the obvious truth. None of the documents we have currently provide any guidance about what's right here. The solution is to fix the requirements and then fix it going forward.

Tim, I appreciate your comments about pragmatism regarding incidents, etc. What I believe is the main issue here isn't that this incident happened. It's more about "What can we learn from this incident so that it doesn't happen in the future?" For example, the ballot you suggested is a great idea and can be an action item that comes out of this incident.

The deeper concern is more that some CAs are using the "well its not a big deal" to justify not revoking the misissued certificates within 120 hour mark, citing hardship across many of their subscribers. I hope we agree that even if this was not a big deal, the expectation to revoke and correct the certificate issuance process is still important. One of the benefits we get from these "silly" revocations is effectively mini fire drills. These are useful to keep pressure on the subscribers who have not yet prioritized adopting a CLM system.

But having country codes in all caps also appears to be a universal practice (if not an ISO requirement), so even if versions of 3166-1 don't expressly say "uppercase", etc., it appears that ISO only uses the codes as uppercase in all of its official ISO documentation and lists. So, I agree with Telia and its conclusion that country codes must be in uppercase.

This is really what this comes down to. 1) This is a universal practice, and unfortunately potentially one of those "falsehood programmers believe about..." 2) These incidents have now set a precedent going forward withe country codes. I hope that a ballot makes its way to specify this in the BRs to remove confusion going into the future as well.

This is update to inform that we will deploy corrective fix to prevent this issue recurring in our systems on Monday the 20th of May.

Further we are finalizing the final report, but we shall do one final verification of the root cause analysis and therefore we shall update the final report on this incident at latest 2024-05-22 13:00 UTC.

Whiteboard: [ca-compliance] [ov-misissuance] Next update 2024-05-17 13:00 UTC → [ca-compliance] [ov-misissuance] Next update 2024-05-22 13:00 UTC

This is to update the full incident report.

Incident Report

Summary

Telia CA mississued thirteen (13) TLS certificates with subject:countryNamein lower case letters. This is a violation of CA/Browser Forum Baseline Requirements for the Issuance and Management of Publicly‐Trusted TLS Server Certificates (TLS BR) section 7.1.2.7.4 Organization Validated for subject:countryName attribute format. The said requirement requires that the attribute is represented in the certificate Subject by two-letter ISO 3166-1 alpha-2 code and ISO 3166-1 standard defines in 5.1 Code element formation:

Code element shall consist only of the following characters (cf. ISO/IEC 10646):
— LATIN CAPITAL LETTER A through LATIN CAPITAL LETTER Z
— DIGIT ZERO through DIGIT 9
— HYPHEN-MINUS
Code elements alpha-2 and alpha-3 are formed with LETTERS, num-3 is formed with DIGITS.

And subsequently in section 5.3 Construction of the alpha-2 code:

The alpha-2 code uses combinations, in upper case, of two letters of the 26-character alphabet in Latin
script, allowing for 26 × 26 = 676 combinations. The ISO 3166 country code uses combinations in the
range AB to QL, RA to WZ, and YA to ZY.

Telia CA wish to thank you the reporter for bringing this issue to our attention, remedy and as an opportunity to further strengthen our processes and practices to prevent certificate misissuance in the future.

Impact

Thirteen (13) end entity certificates were misissued with subject:countryNamein lowercase.

As the data in the subject:countryNameis properly validated and indicating correct Country of the certificate Subject, Telia CA considers that the existing certificates may be trusted by the respective certificate consumers as long as certificate holders / subscribers have been able to renew the certificates to minimize the impact to the services protected by the certificates.

Timeline

All times are UTC.

Date and time Description
Jan 22 13:55:19 2024 Subscriber started the certificate order wizard selecting wrong certificate type for the request and included certificate signing request (CSR) containing lowercase value in subject:countryName.
Jan 22 14:28:04 Certificate order handling was started by RA. Due to the defect in the subject:countryName verification against internal database (comparison was not case sensitive in automatic validation), RA order view did not indicate incorrect data in subject:countryName. RA person approved the order and the value in order subject:countryName was updated to the internal database to the organization's address record.
Jan 22 14:28:16 2024 First certificate with subject:countryName in lowercase value was issued: 00018D31927024AF567E1EBA64B02C57
Jan 23 07:35:28 2024 Second certificate with subject:countryName in lowercase value was issued:<br /> 00018D353EDAB3E94BC30085960ACEF2
Jan 23 07:37:09 2024 Third certificate with subject:countryName in lowercase value was issued:<br /> 00018D3540668D52AAB58BD97329B9DC
Jan 23 07:57:21 2024 Fourth certificate with subject:countryName in lowercase value was issued:<br /> 00018D3552E4F902A2F4AD180846580F
Jan 23 09:13:35 2024 Fifth certificate with subject:countryName in lowercase value was issued: <br />00018D3598B19290E7B099815410EE87
Jan 23 09:18:34 2024 Sixth certificate with subject:countryName in lowercase value was issued: <br />00018D359D3F06ACAB4AABDF493FDB0D
Jan 23 10:10:46 2024 Seventh certificate with subject:countryName in lowercase value was issued:<br />00018D35CD0B0C7B4AA7340C34F1450A
Jan 23 11:00:27 2024 Eight certificate with subject:countryName in lowercase value was issued:<br />00018D35FA886151201DDC0C313CFDA2
Jan 23 13:23:44 2024 Ninth certificate with subject:countryName in lowercase value was issued:<br />00018D367DEC00A0633CC4C2F629182E
Jan 23 13:27:11 2024 Tenth certificate with subject:countryName in lowercase value was issued:<br />00018D3680DD2DFA085FE2D84F8D1141
Jan 23 13:28:11 2024 Eleventh certificate with subject:countryName in lowercase value was issued:<br />00018D3681C9FE323D3E6800950ED504
Jan 24 08:07:17 2024 Twelveth certificate with subject:countryName in lowercase value was issued:<br />00018D3A825A95F9A26C57FE82D12292
Jan 24 09:00:41 2024 Thirteenth certificate with subject:countryName in lowercase value was issued:<br />00018D3AB33F04A969F7955973DAB521
Jan 26 12:42:41 The Fourteenth (14th) order reversed the issue, when subscriber provided company information that needed to be verified from the Reliable Data Source. Information from the Reliable Data Source returned subject:countryName in uppercase. Oder was processed by RA the same way as described in the above. Subsequent certificates for the organization were then issued by using subject:countryName ('SE') stored on the internal database
May 09 00:17 2024 Email report from pki researcher received with list of problematic certificates (SHA-256) to Telia's problem reporting email ca-problems@telia.fi. <br />B2604755F06B9EA8B033E3B5A07EE2338789C9959210DEB3C97EF77F7EA81619 <br />4F58CF4B9E3D1C218A7A7E92654865C4683D81AA80E4DFF70D6AAEDDD07BF055 <br />C358AE8CA1E02D28EF8E8843AC7B9526BE195294A491879A4F9BAD104DA9E225 <br />4CDFDBE5736E179AF00D16CCF1D4679C3C856B0E303B765AB9903CC52425602B <br />EC5CB482143D7A57F8B16289525F7AC71BC143D693A48F1AB22F8C8F6650ADB2 <br />1E8B460171834832A47A15B9736E59CEB81F7A665C52FB39697830C17377686C <br />9F665A0E37D8AAEE3E23E0796DCCBE9A29B4BD4C7DDA5357F79300557759FD09 <br />6C6DCD1E38AC443BCDFE58111AACCFF1A9BB45EAE025CA10618DB14C387AC5CE <br />D84B1D54143A5A15D05CBB178382EEA187ED6128609A0C689CE886FE09CB035D <br />734338F9C43CA2810543557D5ED711FBF70C22CAF7120D831867289195724B67
May 09 17:11 2024 Telia CA Security Manager responded to the problem report via Telia CA's support email (cainfo@telia.fi) to acknowledge the reception of the problem report.
May 10 03:50 2024 Telia CA's Security Manager initiated internal investigation process and started to collect evidence data and prepare incident reports.
May 10 04:58 2024 Telia CA Security Manager called upon CA PKI Security Board supplemental meeting to initally address the incident and verify the incident as violation of requirements as indicated in this incident report.
May 10 06:30 2024 Telia CA Security Board conviened to address the matter and assigned Telia CA Security Manager and designated Telia CA Administrators for further investigation of the issue in order to produce and compile full incident report as required.
May 10 07:40 2024 Externally reported ten (10) Certificate holders / subscribers were informed by Telia CA Security Manager of the details and the need to renew the certificates concerned with immediate effect as the existing certificates shall be revoked. Telia CA will revoke the existing misissued certificates immediately after replacing certificates have been issued by the certificate holders / subscribers.
May 10 08:25 2024 Complete review of all issued certificates completed to verify if additional misissued certificates of similar kind would be existing by designated Telia CA Administrator.<br />Completed review confirmed that the certificates listed in this incident report were the only ones with malformed subject:countryName<br />Additional three (3) certificates with the same issue found and included in this report.
May 10 08:43 2024 Additional three (3) certificate holders / subscribers informed after internal review of all certificates found those additional certificates with the same issue.
May 10 08:53:47 2024 Certificate https://crt.sh/?id=11840646747 revoked after renewed by certificate holder with a new one.
May 1010:12:13 2024 Certificate https://crt.sh/?id=11841121388 revoked after renewed by certificate holder with a new one.
May 10 13:36 2024 Telia CA Security Manager created the initial incident report for the issue.
May 13 06:36:28 2024 Certificate https://crt.sh/?id=11851026130 revoked after renewed by certficate holder with a new one.
May 13 09:19:08 2024 Certificate https://crt.sh/?id=11840035891 revoked after renewed by certificate holder with a new one.
May 13 09:41:20 2024 Certificate https://crt.sh/?id=11832169804 revoked after renewed by certificate holder with a new one.
May 13 10:07:59 2024 Certificate https://crt.sh/?id=11839864048 revoked after renewed by certificate holder with a new one.
May 14 04:48 2024 Delayed revocation incident report created related to this Bug with Bugzilla ID 1896553.
May 15 04:54:35 2024 Certificate https://crt.sh/?id=11842684205 revoked after renewed by certificate holder with a new one.
May 15 04:56:56 2024 Certificate https://crt.sh/?id=11842698882 revoked after renewed by certificate holder with a new one.
May 15 04:58:44 2024 Certificate https://crt.sh/?id=11842711133 revoked after renewed by certificate holder with a new one.
May 16 04:17:51 2024 Certificate https://crt.sh/?id=11840688066 revoked after renewed by certificate holder with a new one.
May 10 to May 21 2024 Thorough Root Cause analysis performed by Telia CA PKI team to identified underlying reasons for the incident.
May 20 13:30 Configuration fix to prevent subject:countryName in lower case deployed in production. Change acceptance criteria for subject:countryName to accept only uppercase characters in certificate order to ensure that all Certificates are always requested by using the uppercase values in subject:countryName.
May 21 08:30 2024 Telia Security Board reviewed, verified and confirmed the final version of the incident analysis and cleared the incident report to be submitted.
May 22 2024 This incident report submitted to Bugzilla ID 1896108.

Root Cause Analysis

When a subscriber requests a new organization-validated certificate from Telia CA using single certificate order function guided wizard-based order process is used to configure an OV-certificate order for Telia.

The subscriber provides the necessary certificate subject information through a guided process accompanied with Certificate Signing Request (CSR). The subject:organizationName (O), subject:localityName (L), and subject:countryName (C) values from the CSR are replaced with values validated and valid data from CA's internal database or from Reliable Data Source (if Subject data needs to be validated during the process) by the guided wizard process. If Reliable Data Source is used validated values will be stored on the internal database for subsequent usage for the subscriber.

In Telia CA Root Cause analysis, we identified that the wizard has defect which caused subject:countryName from the CSR to be used instead of the validated data in the internal database.

Following is the explanation of the defect symptoms and sequence of events that contributed to the certificates being issued with lower case subject:countryName:

Following conditions must be met to the identified defect to cause subject:countryName to be incorrectly taken from the CSR:

  • Subscriber selects Certificate type for order that doesn’t match the content of the CSR put on the order
  • CSR contains lowercase value in the subject:countryName
  • System did not indicate to RA operator in the order confirmation that the subject:countryName was in lowercase leading to RA operator to miss the wrong value in subject:countryName and RA approved the first certificate to be issued.

When Telia RA open pending order for approval**, **RA order approval view indicates certificate Subject values that haven’t yet been approved and requires to be validated by RA. The subject:countryName verification in this process was case insensitive so lowercase value in subject:countryName matched already validated and approved uppercase subject:countryName in the internal database and RA did not get an indication of the non-approved subject:countryName.

At the end of the order handling process, prior to certificate issuance, all validated, approved and changed values are stored in the internal database. The comparison of the changed values is case sensitive which could lead lowercase subject:countryName to be stored. The subsequent certificate orders and requests related to the organization use the newly approved subject:countryName when data is retrieved from internal database.

  1. Subscriber of Telia Company AB initiated the guided wizard-based order process for an Organization Validated Certificate:
  • The subscriber selected Certificate type that didn’t match the content of the Certificate Signing Request (CSR).
  • Using the wizard, the subscriber provided the company name for the wizard to fetch subject organization values from the Reliable Data Source
  • Subscriber pasted the CSR to the order, which contained a lowercase subject:countryName (‘se’).
  1. Subscriber chose to continue to the next step in the order process
  • The wizard compared order's certifcate type with the CSR content:
  • A warning message was displayed to the subscriber about Certificate type and CSR mismatch.
  • Due to the identified defect in the order wizard, the order wizard incorrectly re-filled the subject organization values, taking the subject:countryName from the CSR instead of the Reliable Data Source.
  1. The subscriber finalized the order, and it was sent to RA (Registration Authority) for approval with lowercase value ‘se’ in subject:countryName.
  2. The RA personnel began handling the order:
  • subject:countryName value comparison is case-insensitive, so no indication of a new value was given to the RA personnel.
  • The RA person finalized and approved the order. Order approval update took the subject:countryName value from the order and in this update process the comparison is case-sensitive, resulting system to update subscriber organization's order record to be update in the internal database with the new value in lowecase (‘se’).
  • Certificate with serial number 00018D31927024AF567E1EBA64B02C57 was issued with lowercase value in subject:countryName.
  1. Subsequent 12 certificates for Telia Company AB organization were issued with lower case subject:countryName value.
  2. Telia Company AB subscriber for the fourteenth (14th) certificate order initiated the guided wizard-based order process for an Organization Validated Certificate:
  • Subscriber selected Certificate type that matched the CSR content.
  • Subscriber inputted abbreviated name for the company name.
  • Because organization name was not exact match with the validated value stored in internal database, the wizard retrieved subject organization values (subject:countryName ‘SE’) from Reliable Data Source to verify subject data validity.
  • Organization information was filled with data from Reliable Data Source
  • Subscriber pasted the CSR.
  1. The subscriber finalized the order, which was sent for RA approval with subject:countryName ‘SE’.
  2. The RA personnel handled the order:
  • As explained in item four (4), the case-insensitive comparison provided no indication of change in subject:countryName value.
  • The RA person finalized the order process, storing value ‘SE’ in the internal database, the same update as explained in item four (4) reversed the data in the internal database to contain uppercase value in the organization's order record.
  1. Subsequent certificates for Telia Company AB organization were issued using valid subject:countryName.

Lessons Learned

What went well

  • Response to problem report and required actions to address the incident were initiated as expected from a CA.

What didn't go well

  • Safeguard to check data contents in subject:countryName did not recognize the lowercase value in the automation workflow as described by the Root Cause Analysis due to the explained defect.
  • Revocation of all certificates was not completed in five (5) days and subsequent delayed revocation incident needed to be submitted.

Where we got lucky

  • The material error in the certificates was still unambiguous.
  • The issue affected only one subscriber and for a short period of time.
  • Automation process for refreshing data from Reliable Data Source fixed the problem in relatively soon.
  • Only thirteen (13) certificates were issued.

Action Items

Action Item Kind Due Date
RA was immediately instructed to pay extra caution to verify correct case in subject:countryName when approving orders before preventive fix in place Action (prevent) 2024-05-10, 12:07 COMPLETED
The configuration of the wizard so that in subject:countryName only uppercase value is allowed. Action (prevent) 2024-05-20, COMPLETED
Submit full incident report to the Mozilla Bugzilla Action 2024-05-22, COMPLETED
Fix to the certificate configuration wizard so that it will always use validated and valid subject:countryName from the database or from Reliable Data Source Action (prevent) 2024-06-30
Improve the RA-approval view to ensure subject:countryName verification is case sensitive. Action (prevent) 2024-06-30
Enforce all API request to issuing CA system pass subject:countryName in uppercase. Action (prevent) 2024-08-30

Appendix

Details of affected certificates

Certificates on the list marked with x found in Telia CA's full review of certificates.

  1. https://crt.sh/?id=11832169804 (Not Before: Jan 22 14:28:16 2024 GMT)
  2. https://crt.sh/?id=11839864048 (Not Before: Jan 23 07:35:28 2024 GMT x)
  3. https://crt.sh/?id=11839878201 (Not Before: Jan 23 07:37:09 2024 GMT)
  4. https://crt.sh/?id=11840035891 (Not Before: Jan 23 07:57:21 2024 GMT)
  5. https://crt.sh/?id=11840646747 (Not Before: Jan 23 09:13:35 2024 GMT)
  6. https://crt.sh/?id=11840688066 (Not Before: Jan 23 09:18:34 2024 GMT)
  7. https://crt.sh/?id=11841121388 (Not Before: Jan 23 10:10:46 2024 GMT x)
  8. https://crt.sh/?id=11841544979 (Not Before: Jan 23 11:00:27 2024 GMT x)
  9. https://crt.sh/?id=11842684205 (Not Before: Jan 23 13:23:44 2024 GMT)
  10. https://crt.sh/?id=11842698882 (Not Before: Jan 23 13:27:11 2024 GMT)
  11. https://crt.sh/?id=11842711133 (Not Before: Jan 23 13:28:11 2024 GMT)
  12. https://crt.sh/?id=11851026130 (Not Before: Jan 24 08:07:17 2024 GMT)
  13. https://crt.sh/?id=11851399778 (Not Before: Jan 24 09:00:41 2024 GMT)

Next update by Next Update.

Whiteboard: [ca-compliance] [ov-misissuance] Next update 2024-05-22 13:00 UTC → [ca-compliance] [ov-misissuance] Next update 2024-05-29

(In reply to Antti Backman from comment #19)

| Fix to the certificate configuration wizard so that it will always use validated and valid subject:countryName from the database or from Reliable Data Source | Action (prevent) | 2024-06-30 |

I think this is the most important action item. There are many incidents filed here on Bugzilla caused by using customer input directly from the CSR. Have you checked if there are any other scenarios where data is taken directly from a CSR rather than from your verification source or from a limited set of predetermined values?

Hi Mathew,

We entirely agree with your observation. When we identified the issue root cause, we did full review of the whole issuing system components to identify if similar issue could occur elsewhere in the system.

That review showed that this specific function only is having the issue that we now will fix.

Whiteboard: [ca-compliance] [ov-misissuance] Next update 2024-05-29 → [ca-compliance] [ov-misissuance] Next update 2024-06-28

This is to update that plans and actions to complete the action items pending are set and expected to be completed as indicated.

Next update at Next Update.

This is to update that we have completed following action items on Friday the 14th of June.

Fix to the certificate configuration wizard so that it will always use validated and valid subject:countryName from the database or from Reliable Data Source

Improve the RA-approval view to ensure subject:countryName verification is case sensitive.

One action remains to be completed by end of August 2024.

Telia continues to follow this incident, next update by Next Update.

This is to update that the last pending action item progresses as planned and we see no issues to deploy the update in time as defined for the action item.

Telia continues to follow this incident, next update by Next Update.

Whiteboard: [ca-compliance] [ov-misissuance] Next update 2024-06-28 → [ca-compliance] [ov-misissuance] Next update 2024-08-05

Dear Antti, please allow Ben Wilson or someone from Mozilla to update the Whiteboard regarding Next updates. I believe you are the only CA that is currently setting these flags as if they are "automatically granted".

I agree with Dimitris. Even if the expected deployment date is in August, weekly updates should be provided to reaffirm that things are on schedule, and share progress (like “feature complete”, “manual test complete”, “running as shadow of production for validation”) ahead of final delivery and report of successful production deployment.

Thank you! I’m taking the liberty of changing update-due to today+7d and pinging Ben for his concurrence or overruling.

(It occurs to me now that this might have just been an 07/08 typo in the update too?)

Flags: needinfo?(benwilsonusa)
Whiteboard: [ca-compliance] [ov-misissuance] Next update 2024-08-05 → [ca-compliance] [ov-misissuance] Next update 2024-07-05
Flags: needinfo?(benwilsonusa) → needinfo?(bwilson)

Thanks. Yes, CAs should suggest "Next updates" and then we will adjust the Whiteboard accordingly, if appropriate.

Flags: needinfo?(bwilson)

Hi,

Thank you for pointing this out, my mistake.

We'll be providing updates of the progress and status of the last pending Action item on weekly basis.

Next update as set on this bug.

This is to update that the last pending action item progresses as planned and we see no issues to deploy the update in time as defined for the action item.

Telia continues to follow this incident and provide updates of the progress of pending action items on weekly basis.

This is to update that the last pending action item progresses as planned and we see no issues to deploy the update in time as defined for the action item.

Telia continues to follow this incident and provide updates of the progress of pending action items on weekly basis.

This is to update that the last pending action item progresses as planned and we see no issues to deploy the update in time as defined for the action item.

Telia continues to follow this incident and provide updates of the progress of pending action items on weekly basis.

This is to update that the last pending action item progresses as planned and we see no issues to deploy the update in time as defined for the action item.

Telia continues to follow this incident and provide updates of the progress of pending action items on weekly basis.

Whiteboard: [ca-compliance] [ov-misissuance] Next update 2024-07-05 → [ca-compliance] [ov-misissuance]

This is to update that the last pending action item progresses as planned and we see no issues to deploy the update in time as defined for the action item.

Telia continues to follow this incident and provide updates of the progress of pending action items on weekly basis.

This is to update that the last pending action item progresses as planned and we see no issues to deploy the update in time as defined for the action item.

Telia continues to follow this incident and provide updates of the progress of pending action items on weekly basis.

This is to update that the last pending action item's development has progressed to final QA stage, we see no issues to deploy the update in time as defined for the action item.

Telia continues to follow this incident and provide updates of the progress of pending action items on weekly basis.

This is to update that the last pending action item has passed the final QA stage and is cleared to be deployed, update will be deployed to production on week 35 as planned in our next service window.

Telia continues to follow this incident and provide updates of the progress of pending action items on weekly basis.

This is to update that the last pending action item has been successfully completed on Wednesday the 28th of August 2024 by deploying required software update to production.

Action Item Kind Due Date
Enforce all API request to issuing CA system pass subject:countryName in uppercase. Action (prevent) 2024-08-30

Telia continues to follow this incident and provide updates of the progress of pending action items on weekly basis.

Should this issue be closed now?

Flags: needinfo?(antti.backman)

Hi Ben,

We've completed all action items as updated, so from our perspective this issue could be closed.

Thanks

I'll intend on closing this on Wed. 4-Sept-2024, unless there are more questions or issues to discuss.

Flags: needinfo?(bwilson)
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Flags: needinfo?(antti.backman)
You need to log in before you can comment on or make changes to this bug.