Open Bug 1886998 Opened 2 months ago Updated 25 days ago

Microsec: Late response to a certificate problem report

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
ASSIGNED

People

(Reporter: szoke.sandor, Assigned: szoke.sandor)

Details

(Whiteboard: [ca-compliance] [policy-failure])

Incident Report

Summary

It was reported by email to info@... , that Microsec misissued an EV certificate.
The problem is that the certificate does not contain the CPSuri link.
Microsec did not react in time, so a second email was sent to info@... and also to the Microsec's CCADB contact persons.
Due to the delay, 3 separate incident reports will be created as follows:

  • Bug #1 must focus on the certificate misissuance reported in the problem report.
  • Bug #2 must focus on the delayed revocation of the misissued certificates described in the problem report.
  • Bug #3 must focus on the failure to respond to a certificate problem report in a complete and/or timely manner.

.

Impact

The missing CPSuri information has no impact on the usability or security of the certificate, but it makes it more difficult for users to find the policy information.
The misissued certificate is:
https://crt.sh/?id=12302329269

Timeline

Processing email 1

2024-03-08

  • 03:02 UTC (04:02 CET)
    • Microsec received an email reporting a potentially misissued certificate.
      • the email was sent from a @gmail.com address
      • the email was sent to the general purpose email address info@... provided in CCADB
    • Microsec's OTRS system received the email and sent an automatic notification email with a registration number: Ticket#2024030884001045 within one minute
      • The received emails are automatically answered and moved to the "Inbox waiting list" in OTRS
  • 08:29 UTC (09:29 CET)
    • OTRS Ticket#2024030884001045 was moved from the "Inbox waiting list" to the "Standard waiting list"
      • An operator checks the incoming emails and classifies them by moving them into specific waiting boxes every workday morning
      • It was a potential problem, because there is no strict deadline for the processing of emails in this waiting list
      • Emails in this waiting list are typically processed and answered within 3 days in case of normal load

2024-03-18

  • 15:40 UTC (16:40 CET)
    • OTRS Ticket#2024030884001045 was assigned to the head of Customer Service Department

2024-03-19

  • 11:19 UTC (12:19 CET)
    • OTRS Ticket#2024030884001045 was closed as "resolved in another ticket"
  • 15:02 UTC (16:02 CET)
    • OTRS Ticket#2024030884001045 was linked to OTRS Ticket#2024031884006012

Processing email 2

2024-03-18

  • 13:06 UTC (14:06 CET)
    • Microsec received a second email reporting the same potentially misissued certificate.
      • the email was sent from a @google.com address
      • the email was sent not only to the general purpose email address info@... provided in CCADB, but was also sent directly to the contact persons of Microsec provided in CCADB.
    • Microsec's OTRS system received the email messages and sent an automatic notification email with a registration number: Ticket#2024031884006012
  • 13:38 UTC (14:38 CET)
    • OTRS Ticket#2024031884006012 was moved from the "Inbox waiting list" to the "IT support waiting list"
      • this prioritization was OK, because this waiting list is managed with much higher priority and typically processed within one day
  • 15:00 UTC (16:00 CET)
    • Sándor Szőke, the main contact person in CCADB, received the email in his personal mailbox and sent a manual notification email to the sender.
    • Microsec began the investigation of the issue.
  • 15:22 UTC (16:22 CET)
    • Microsec opened internal jira tickets for the management of the issue.
  • 15:30 UTC (16:30 CET)
    • Microsec studied the current CABF BR and CABF EVG requirements and verified that the presence of CPSuri is mandatory in the case of EV TLS certificates.
    • Microsec decided to modify the problematic EV TLS certificate profiles and issue a new version of its Certificate Profiles document.
    • Microsec realized that the problem starts with version 2023-08-29 of the certificate profiles. Each EV certificate issued after this date shall be considered as misissued and has to be replaced.
  • 16:15 UTC (17:15 CET)
    • Microsec modified all (7) EV TLS certificate profiles in the open working area.
  • 16:40 UTC (17:40 CET)
    • A new TAG was created in the SVN configuration management system for the new certificate profiles to be activated.
  • 16:45 UTC (17:45 CET)
    • The Customer was informed about the changes and the necessary measures. There was no problem with revocing the misissued certificate.
  • 17:03 UTC
  • 17:05 UTC
    • The misissued certificate was revoked.
  • 18:38 UTC (19:38 CET)
    • Microsec officially released the new version of the Certificate Profiles document.

2024-03-19

  • 07:01 UTC (08:01 CET)
    • OTRS Ticket#2024031884006012 was assigned to G.S. at IT Support Department
  • 07:05 UTC (08:05 CET)
    • incoming email was forwarded to the management
      • It was the correct action, so the formal OTRS based process reached the other process based on the personal email notifications
      • From this time there are several records in OTRS about the processing of the ticket, there is no importance to give more details here
  • 11:00 UTC (12:00 CET)
    • Microsec discussed the issue at its regular management meeting and developed a detailed action plan.
    • Microsec discovered that a total of 45 EV TLS certificates were misissued, including some test certificates. These certificates shall be replaced and revoked within 5 days in accordance with CABF requirements.
  • 18:10 UCT
    • Bug #1 incident report was opened in Bugzilla.

2024-03-20 -22

  • collecting the information for Bug #3 in progress

2024-03-22

  • Bug #3 incident report was opened in Bugzilla

Root Cause Analysis

The late response was caused by different thing happening simultaneuosly

  • the emails were sent to info@ email address

    • Microsec offers special email addresses for revocation requests and for High Priority Problem Reports.
      • emails received on these special addresses are processed within 24 hours
    • This email address is given in CCADB as a general contatc email. The purpose of this email address according to CCADB note:
      • "CA Email Alias 1 and 2 are used to reach more than one person in your organization to receive notifications in case the primary contact is out of the office or leaves the organization."
    • The input email did not match to any existing classification so it was forwarded to "Standard waiting list"
    • Peek load on the "Standard waiting list"
      • Microsec launched a big campaign to replace old signature cards on 2024-03-06, and this resulted a huge pick load in the incoming email traffic.
      • Usually each ticket in this list is processed in less than 3 workdays, but due to the mentioned project the processing delay temporarily increased to 8-10 days
      • First email was to be processed when the second email arrived
    • The second email was sent to the contact persons directly and was processed in two hours by one of the contact persons

  • The first email was sent from a standard @gmail.com address

    • there are special domain names which are managed with higher priority, but free email addresses are not among them
    • the second email was sent from @google com domain
      • it was forwarded to another waiting list and was processed very quickly

Lessons Learned

Our standard email address "info@ " is used for many purposes and sometimes we receive a huge amount of messages here.
We plan to set up a new email address dedicated to CCADB and Root Program messages. We plan also a dedicated OTRS waiting list for these emails which will be processed with high priority.

What went well

  • The second email was processed very quickly thanks to the followings
    • it was sent also directly to CCADB contact persons
    • it was sent from a domain managed with high priority

What didn't go well

  • The temporary heavy load caused delay on the processing of the standard incoming emails
  • the operator could not realize that this is an important email and should be managed with high priority

Where we got lucky

Action Items

Action Item Kind Due Date
Improving the OTRS processing rules Detect 2024-05-31
Setting up a dedicated email for CCADB Prevent 2024-05-31

Appendix

Details of affected certificates

There are 45 misissued certificates, the full list will be published in the following incident report:

Bug #1 already opened, see https://bugzilla.mozilla.org/show_bug.cgi?id=1886257

Based on Incident Reporting Template v. 2.0

Assignee: nobody → szoke.sandor
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] [policy-failure]
Component: CA Certificate Root Program → CA Certificate Compliance

Incident Status Report - 2024-04-05

2024-03-25

  • info@... emails were also temporarily forwarded to our CTO, as a quick fix

2024-04-03

  • First professional consultation on the development possibilities of our OTRS system.

Action Items

Action Item Kind Due Date Status
Improving the OTRS processing rules Detect & Prevent 2024-05-31 In progress
Setting up a dedicated email for CCADB Prevent 2024-05-31 Planned

Incident Status Report - 2024-04-19

2024-04-12

  • CCADB_contact@... email adress was created
  • incoming emails were also forwarded to our internal EHSZ mailing list, which contains the personal email addresses of the EHSZ management responsible for compliance issues.

Action Items

Action Item Kind Due Date Status
Improving the OTRS processing rules Detect & Prevent 2024-05-31 In progress
Setting up and configuring a dedicated email for CCADB contact group Prevent 2024-05-31 In progress
You need to log in before you can comment on or make changes to this bug.