Closed Bug 1859314 Opened 1 year ago Closed 10 months ago

Telia: TLS certificates issued in violation of TLS BR v2.0.1

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: antti.backman, Assigned: antti.backman)

Details

(Whiteboard: [ca-compliance] [ov-misissuance])

This is initial report for missisuance of three (3) TLS certificates by Telia CA. Full disclosure and report shall be updated on this incident no later than the 24th of October 2023 16:00 EET.

In daily automatic compliance review on Friday the 13th of October 2023 23:59 EET for all issued TLS certificates for the 13th of October, the compliance review report indicated the problematic organizations having UNVALIDATED organization name. The compliance report was reviewed by Telia CA Administrator responsible for the review of the report on Monday the 16th of October 2023 08:20 EET after the weekend. Problem indicated by the report was immediately investigated and identified to be typo in the issued certificate.

Identified certificates were found having minor typo in the organizationName Subject attribute. All of the problematic certificates have the same issue. Name in the certificates is Nordic Investement Bank when the correct name should be Nordic Investment Bank.

Telia CA has automated validation of organization names deployed in the validation practice, but in this specific case the registered organization name exceeded 64 characters and therefore system initiated manual validation of the organization name to meet requirement set fort in TLS BR v2.0.1 section 7.1.4.2 Subject Attribute Encoding for organizationName (OID 2.5.4.10).

The official registered name retrieved from CA's verified datasource is Nordiska Investeringsbanken, Pohjoismaiden Investointipankki NIB, Nordic Investment Bank for the subscriber.

For organization validated subscriber certificates it is allowed by the TLS BR v2.0.1 section 7.1.2.7.4 Organization Validated

The Subject’s name or DBA. The CA MAY include information in this field that differs slightly from the verified name, such as common variations or abbreviations, provided that the CA documents the difference and any abbreviations used are locally accepted abbreviations; e.g. if the official record shows “Company Name Incorporated”, the CA MAY use “Company Name Inc.” or “Company Name”.Manual validation was initiated by Telia Registration Authority and the said Telia CA Administrator was responsible for the verification of the validation data to confirm RA's validation practice and end result.

In the manual validation RA Officer and Telia CA Administrator determined that unambiguous name for the organization would be Nordic Investment Bank. Telia RA Officer defined the organization name in the CA system and Telia CA Administrator verified the name to be as agreed. In this manual verification phase both RA Officer and CA Administrator misread the verified name, which was eventually misspelled by RA as Nordic Investement Bank.

Three (3) problematic certificates were then issued as listed below in the certificate details to contain the incorrect and typoed organization name.

Telia CA Administrator responsible for daily compliance report review informed Telia CA Security and Compliance manager immediately after the detailed investigation of the reported issue was completed and verified as explained in this initial report.

Telia CA's Security Board convened at the 16th of October 11:30 EET to make final review of the incident and verify need for public incident reporting in accordance with TLS BR and Root program policy requirements. Telia CA Security and Compliance manager was designated responsible for public reporting and required internal resources were immediately assigned to collect the evidence for full disclosure report.

Immediate actions were initiated both internally and externally to inform affected organization and to collect details and chain of events to identify all the details why the certificates were in fact mississued.

Affected certificate subject / organization was notified at the 16th of October 11:44 EET that their respective certificates shall be revoked. Subject organization replaced the problematic certificates by Oct 16 11:07:04 2023 UTC. Subject organization informed Telia CA Oct 16th 15:22 EET that they had revoked the problematic certificates. Telia CA revoked immediately after being notified all problematic certificates as detailed below.

List of problematic certificates:

Full disclosure report will be submitted no later than the 24th of October 2023 16:00 EET.

Assignee: nobody → antti.backman
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] [ov-misissuance]
Whiteboard: [ca-compliance] [ov-misissuance] → [ca-compliance] [ov-misissuance] Next update at 24.10.2023

This is the full disclosure report for the incident.

Summary

The Summary section (as set forth in the markdown template below) should contain a short description of the nature of the issue. This provides just enough context for new readers to understand the details in the rest of the report.

In daily automatic compliance review on Friday the 13th of October 2023 23:59 EEST for all issued TLS certificates for the 13th of October, the compliance review report indicated the problematic organizations having UNVALIDATED organization name. The compliance report was reviewed by Telia CA Administrator responsible for the review of the report on Monday the 16th of October 2023 08:20 EEST after the weekend. Problem indicated by the report was immediately investigated and identified to be typo in the issued certificate's organizationName value.

Identified certificates were found having minor typo in the organizationName Subject attribute. All of the problematic certificates have the same issue. Name in the certificates is Nordic Investement Bank when the correct name should be Nordic Investment Bank.

Impact

The Impact section should contain a short description of the size and nature of the incident. For example: how many certificates, OCSP responses, or CRLs were affected; whether the affected objects share features (such as issuance time, signature algorithm, or validation type); and whether the CA Owner had to cease issuance during the incident.

As the number of certificates was small (3 pcs.) and more importantly the information in the certificate organizationName was unambiguous even with the typo, the impact is determined by Telia CA to be very low. Furthermore the issue was detected swiftly and as the detailed timeline depicts problematic certificates were revoked quickly after discovery of the issue and replaced with ones having correct organizationName.

For such low impact issue, Telia CA did not stop issuance.

Timeline

The Timeline section must include a detailed timeline of all events and actions leading up to and taken during and after the incident. The timeline must include not just the actual discovery of the incident and subsequent events, but also relevant events occuring beforehand (e.g. something changed or was introduced). All times should be in UTC (or local system time with the offset from UTC provided) and have at least minute-level granularity. In addition, it must indicate the following events:

  • All policy, process, and software changes that contributed to the Root Cause
  • The time at which the incident began
  • The time at which the CA Owner became aware of the incident
  • The time at which the incident ended
  • The times at which issuance ceased and resumed, if relevant

All times are EEST (UTC).
2023-10-03:

  • 10:44 (07:44)
    New applicant submitted their order for the Certificate Manager self service portal. Telia CA's order template includes, among other information of the applicant, organization information pre-filled by the customer. The order documentation contained the said misspelling of the company name (Nordic Investement Bank). In the prevalidation of the order the misspell was not notifed by the RA Officer.

  • 11:59 (08:59)
    Telia's RA Officer started to file in the new application and subsequently the creation of new account for the applicant. When RA files in new account for new applicant / customer the process begins in the RA system by input of company's verfied company identification number / business ID from the QIIS used by Telia CA. RA Officer inputs applicants information for the application documentation and the RA system verifies the inputted data with information retrieved from the QIIS. In this verification step RA system informed the RA Officer that company information could not be verified and the organization record could not be created in the RA system.

  • RA Officer called-in for support from the Telia CA Administrator on call that time by posting support request in the internal support system to register the RA support request to CA.

2023-10-04:

  • 13:20 (10:20)
    Telia CA Administrator on call started to investigate the support request and the issue described by RA Officer. The issue was soon detected to be caused by QIIS record having the organization name exceeding 64 characters, thus exceeding the maximum length allowed by TLS BR v2.0.1 section 7.1.4.2 Subject Attribute Encoding for organizationName (OID 2.5.4.10). Telia CA Administrator declined the organization name as per the said requirement for maximum length.
  • 13:46 (10:46)
    Telia CA Administrator informed RA Officer of the root cause for the issue and instructed RA Officer to initiate manual enrollment of the applicant to the CA System. Telia CA Administrator determined based on the information available from the QIIS that organization name providing unambiguous name for the certificate processing would be Nordic Investment Bank and instructed RA Officer to use that name for the organization record in the manual enrollment. The selected name was also in the applicant's order. At this step Telia CA Administrator did not recognize the typo in the applicant's order documentation.
  • 14:34 (11:34)
    After receiving the instructions, RA Officer began manual enrollment of the applicant organization into the CA system. Information inputted to the system was taken from the applicants order documentation. RA Officer either did not recognize the typo in the organization name provided by the applicant when compared the data from the QIIS to the inputted information from the applicant's order and saved the incorrect organization name to the system.

2023-10-05:

  • 12:19 (09:19)
    RA Officer finalized the self-service portal order and created customer's account and required details. Applicant was informed about the newly created agreement and account in Telia CA's Certificate Manager Portal. Applicant was prepared and ready for as Subscriber in in the portal.

2023-10-13:

2023-10-16:

  • 08:00 (05:00)
    The compliance verification report was validated by the CA Administrator responsible for the daily validation. An immediate investigation started about the reported problematic Organization name in the report.

  • 08:19 (05:19)
    Telia CA Administrator came aware about the typo in the organization name as the organization name could not be verified from approved QIIS. The name was immediately corrected in the CA system database to what is was determined to be unambiguous and meeting the TLS BR requirements for the maximum length.

  • 08:20 (05:20)
    Telia CA Security Manager was informed about the enrolled certificates with a wrong O-value.

  • 11:30 (08:30)
    Telia CA Security Manager called-in Telia CA Security Board for an emergency meeting to verify and document the issue in accordance with internal polices. Security Board verified the issue to be factual and designated Security Manager to initiate public incident disclosure procedure.

  • 11:44 (08:44)

    Affected certificate subject / organization was notified by the CA Administrator that their respective certificates shall be revoked due to the issue identified. Customer was informed about the misspelled organizationName in the certificates and requested to do new certificates as soon as possible and revoke old certificates himself or inform Telia CA administrator to do so.

  • 14:07 (11:07)

    Subject organization replaced the problematic certificates.

  • 15:22 (12:22)

    Subject organization informed Telia CA that they had replaced the problematic certificates and requested Telia CA to revoke problematic certificates.

  • 15:38 (12:38)

    Telia CA Administrator completed the revocation of all problematic certificates as detailed below.

Root Cause Analysis

The Root Cause Analysis section must contain a detailed analysis of the conditions which combined to give rise to the issue. It is unusual for an incident to have a single root cause; often there must be a confluence of several issues such as a software bug, insufficient checks, and a malformed request. Make sure that all contributing causes are identified and described, including noting when they first arose and how they avoided detection until they were discovered or identified.

When new applicant enrollment was transferred to manual process due to the long organization name exceeding TLS BR maximum length, the manual verification by dual operation did not identify the minor typo and the organization name was approved misspelled Nordic Investement Bank. RA Officer performed the enrollment according to internal instructions in accordance with certificate information violating requirements governing certificate data (e.g. TLS BR) in publicly trusted certificate issuance, but as the applicant's order documentation included the typo and the instructed operation is to copy the name as it is in the applicant's order to the system the issue was made possible.

Lessons Learned

What went well

a list of things that caused the incident to have less impact than it otherwise could have, such as early detection, rapid response, or good safety mechanisms. This section provides an opportunity for others to learn from the good practices of this CA Owner.

  • Telia CA's safeguards to ensure compliance (automated daily compliance verification routine) detected the issue and enabled swift response to the issue at earliest possible time.
  • Problematic certificates were revoked quickly and replaced with no undue harm for the subscriber.

What didn't go well

a list of things that caused the incident to have more impact than it otherwise would have, such as missing checks or unclear documentation. Each item here must have at least one corresponding Action Item below and should provide opportunities for others to ensure they make similar improvements if they haven’t already.

  • As explained in this report the manual verification of the organization name passed through the dual verification and incorrect / misspelled organization name was approved for the certificate issuance causing the reported problem.

Where we got lucky

a list of things that went well, but which cannot be relied upon, such as early detection by an external security researcher or limited impact simply due to a small number of requests. Items here should generally also have corresponding Action Items, so that the CA Owner doesn’t have to rely on luck in the future.

  • Being newly issued applicant / subscriber, there wasn't substantial number of certificates in fault. In case this subscriber would have been longer term customer, this validation issue could have resulted in larger number of missisued certificates.

Action Items

The Action Items section must contain a list of remediation items that will be undertaken to ensure that similar incidents do not reoccur in the future. Note that it is not sufficient for these action items to simply stop this incident, they must create additional protections to prevent future incidents. Each Action Item should state:

  • A short description of the action to be taken.
  • A classification of whether the action will help Prevent future incidents, Mitigate the impact of future incidents, or Detect future incidents. CA Owners are encouraged to propose action items in all three categories, with an emphasis on prevention and mitigation.
  • A date by which the action item will be complete.
Action Item Kind Due Date
Internal instructions to be changed that organization name SHALL be verified and copied from the approved QIIS instead from the applicant's provided documentation. Instruction improvement and training to the RA and CA personnel. Mitigate 26.10.2023

Appendix

The Appendix is for all supporting data: log files, graphs and charts, etc. In particular, in the case of incidents which directly impacted certificates, the Appendix must include a listing of the complete certificate details of all affected certificates. The recommended format is to ensure that all affected certificates are logged to CT, then to attach a text file where each line is of the form https://crt.sh/?sha256=[sha256 fingerprint of the certificate]. When the incident being reported involves an SMIME certificate, if disclosure of personally identifiable information in the certificate may be contrary to applicable law, please provide at least the certificate serial number and SHA256 hash of the certificate.

Details of affected certificates

Whiteboard: [ca-compliance] [ov-misissuance] Next update at 24.10.2023 → [ca-compliance] [ov-misissuance] Next update at 27.10.2023

This update is to report that the action item to mitigate disclosed incident has been completed fully as planned and in planned schedule.

Telia CA shall be following incident and further update as needed and defined by Next update.

Whiteboard: [ca-compliance] [ov-misissuance] Next update at 27.10.2023 → [ca-compliance] [ov-misissuance] Next update at 30.11.2023

This is to inform that Telia CA is following this incident periodically and prepared to responed on any comment, question or notification.

All action times has been fully completed, so nothing to report on this update.

Further updates as defined by Next Update.

Whiteboard: [ca-compliance] [ov-misissuance] Next update at 30.11.2023 → [ca-compliance] [ov-misissuance] Next update at 30.12.2023
Whiteboard: [ca-compliance] [ov-misissuance] Next update at 30.12.2023 → [ca-compliance] [ov-misissuance] Next update at 29.12.2023

Regarding the action item:

Internal instructions to be changed that organization name SHALL be verified and copied from the approved QIIS instead from the applicant's provided documentation. Instruction improvement and training to the RA and CA personnel.

This seems like something where a technical control would be more reasonable than a human-centric solution?

(In reply to amir from comment #4)

Regarding the action item:

Internal instructions to be changed that organization name SHALL be verified and copied from the approved QIIS instead from the applicant's provided documentation. Instruction improvement and training to the RA and CA personnel.

This seems like something where a technical control would be more reasonable than a human-centric solution?

Hi Amir and thank you for you suggestion.

In this particular case technical (or automated) control would be challenging to implement. As explained in the below quote of above description of the indicent:

Telia CA has automated validation of organization names deployed in the validation practice, but in this specific case the registered organization name exceeded 64 characters and therefore system initiated manual validation of the organization name to meet requirement set fort in TLS BR v2.0.1 section 7.1.4.2 Subject Attribute Encoding for organizationName (OID 2.5.4.10).

The official registered name retrieved from CA's verified datasource is Nordiska Investeringsbanken, Pohjoismaiden Investointipankki NIB, Nordic Investment Bank for the subscriber.

For organization validated subscriber certificates it is allowed by the TLS BR v2.0.1 section 7.1.2.7.4 Organization Validated

We do deploy automated solution whenever the retrieved value from the QIIS / QGIS meets the TLS BR requirements. But as in this case the name exceeded the maximun allowed lenght for the organizationName field. The registered official name contained the organization's name in three languages (swedish, finnish and english respectively), thus just by automatically cutting the registered official name would have led to a name that is not commonly known nor used by the organization.

As per TLS BR 7.1.2.7.4 Telia CA needed to verify that the name put into the certificate is in accordance of the requirement and in this type of case to determine appropriate value requires manual verficiation. To find technical control to this type of case would be preferred, but as the data in the registered name provided by QIIS / QGIS may be structured in any which way accepted by the QGIS there's no particular quaranteed structure that could be relied on to build technical control in this type of case.
Therefore manual verification was required in this case and will be required in similar cases in the future. The unfortunate thing obviously was that even the four (4) eyes principal passed the typoed value. The importance of accurate and correctly formed values have been stressed internally in the CA and by the trainings and updated documentation to RA officers to mitigate the possiblity of reoccurence of the issue in the future. Telia CA considers this to be sufficient.

Telia CA values all input to provide opportunities for improving any operational procedure to avoid mistakes, faults or errors to occur in certificate issuance.

We continue to monitor this incident and provide updates as indicated by the 'Next update'.

The unfortunate thing obviously was that even the four (4) eyes principal passed the typoed value.

This is generally what I have an issue with. What is going to make this training (per the action item) different from the other trainings to prevent this going into the future?

I'm not necessarily looking to design a solution for your systems here. There are a few avenues of technical improvements that can be made.

For example, using wdiff for comparing what's about to go into the organizationName field, and what QIIS is displaying:

wdiff qiis selected 
Nordic [-Investment-] {+Investement+} Bank [-With Extra Letters-]

Attaching a record of these diffs for the situations where QIIS / QGIS don't return an automatable result will help prevent these issues into the future. It also allows the reviewers to have better visibility on what they're reviewing.

The technical control here would be that a certificate wouldn't be issued if a diff report such as this didn't exist & that the organizationName MUST be what was used in the diff tool (issuance fails, if not).

I don't have a strong opinion on what you'll add to prevent this in the future. What I mentioned above is just an example. But I do think "more training" is not the solution here. Humans make errors, especially if they need to diff characters that look very similar to eachother.

Hi Amir,

Thank you for futher detailing your concerns and more importantly providing ideas for possible technical controls.

We'll take further consideration to evaluate possible implementation of techincal controls to mitigate (or prevent) such issues in the future.

Telia CA values all input to provide opportunities for improving any operational procedure to avoid mistakes, faults or errors to occur in certificate issuance.

We continue to monitor this incident and provide updates as indicated by the 'Next update'.

Telia CA continues to monitor this incident, next update by Next update.

Whiteboard: [ca-compliance] [ov-misissuance] Next update at 29.12.2023 → [ca-compliance] [ov-misissuance] Next update at 26.1.2024

Telia CA continues to monitor this incident, next update by Next update unless this incident is closed prior that.

As there has not been any further questions / comments on this incident since about a month ago and all planned tasks / actions have been completed, Telia CA is kindly requesting this incident to be closed.

Flags: needinfo?(bwilson)
Whiteboard: [ca-compliance] [ov-misissuance] Next update at 26.1.2024 → [ca-compliance] [ov-misissuance] Next update at 23.2.2024

I'll close this tomorrow - Friday, 26-Jan-2024.

Whiteboard: [ca-compliance] [ov-misissuance] Next update at 23.2.2024 → [ca-compliance] [ov-misissuance]
Status: ASSIGNED → RESOLVED
Closed: 10 months ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.