1844799 - SHECA: Failure to Submit Annual CCADB Self Assessment

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Chris Clements]

Section 6. Annual Self Assessments of the Chrome Root Program policy requires CA Owners to submit an annual self assessment to the CCADB.

The initial annual self assessment for SHECA was required in June, 2023. As of July 21, 2023, SHECA has not submitted an initial annual self assessment in accordance with the Chrome Root Program policy and an incident report is now requested.

Comment 1

[jasmine.tang]

(In reply to Chris Clements from comment #0)

Section 6. Annual Self Assessments of the Chrome Root Program policy requires CA Owners to submit an annual self assessment to the CCADB.

The initial annual self assessment for SHECA was required in June, 2023. As of July 21, 2023, SHECA has not submitted an initial annual self assessment in accordance with the Chrome Root Program policy and an incident report is now requested.

On July 27th,SHECA was made aware of this problem via this Bugzilla.The Chrome Root Program's annual self-assessment policy requires CA owners to submit an annual self-assessment to CCADB. However, SHECA did not submit the initial annual self-assessment as required. We are currently doing our best to complete our initial annual self-assessment and we take our compliance issues very seriously and are undertaking an external dedicated audit of our business. Regarding this case, we will also conduct a comprehensive investigation internally and provide an incident report as soon as possible.

Comment 2

[jasmine.tang]

After internal coordination and communication, we completed the annual self-assessment on August 23 and submitted it to CCADB. Unfortunately, the internal investigation is still in progress, we hope to understand the complete incident as soon as possible, which will take some more time.

Comment 3

[jasmine.tang]

1、How your CA first became aware of the problem (e.g., via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP or CCADB public mailing list, a Bugzilla bug, or internal self-audit), and the time and date.

On July 27th, SHECA learned about the problem through Bugzilla.

2、A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a requirement became applicable, a document changed, a bug was introduced, or an audit was performed.

20230727 CST(UTC+8) SHECA learned about this issue through a Bugzilla notification.
20230728 09:30 CST(UTC+8) The investigation was initiated by the Information Security and Compliance Department.
20230824 09:44 CST(UTC+8) SHECA completed the annual self-assessment and submitted it to CCADB.

3、Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

SHECA has not stopped issuing certificates because this incident did not result in the issuance of certificates by mistake.

4、In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g., OCSP failures, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help measure the severity of each problem.

N/A

5、In a case involving TLS server certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. It is also recommended that you use this form in your list “https://crt.sh/?sha256=[sha256-hash]”, unless circumstances dictate otherwise. When the incident being reported involves an SMIME certificate, if disclosure of personally identifiable information in the certificate may be contrary to applicable law, please provide at least the certificate serial number and SHA256 hash of the certificate. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.
N/A

6、Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

We start by honestly acknowledging our concerns and we regret that we did not submit the self-assessment to CCADB within the allotted time.Here are the reasons for the bug's introduction and what to do so far:

Personnel changes: As mentioned in Bug 1838866, at the end of May we experienced a personnel change and we had to deal with the departure of a key colleague who was responsible for completing the CCADB self-assessment. This caused a disruption in the workflow as new colleagues needed time to adjust and understand the work.

Handover of tasks: Due to the urgency, we can only hand over the task to a colleague with relatively little experience. This colleague took longer to get used to the job, so it had an impact on our self-assessment work. It took the colleague longer to adapt and become familiar with the job, which also led to slower progress in the self-assessment.

Despite the many challenges we faced, we have been doing our best to keep the self-assessment work going and ensure their quality and accuracy, culminating in the completion of the CCADB self-assessment exercise at the end of August. To avoid the problem from happening again, we have taken the following measures:

1、Enhanced internal communication: We now pay more attention to communication within the team, especially on critical tasks. We will hold occasional meetings to ensure that all team members understand the status and deadlines of tasks.
2、Task backup: We have established a plan for backup personnel to ensure that tasks can be handed over to others in the event that the person in charge of a key task is unable to continue working.
3、Plan Ahead: We start planning tasks in advance to prevent last minute emergencies. This gives us more buffer time to deal with unexpected situations.

7、List of steps your CA is taking to resolve the situation and ensure that such a situation or incident will not be repeated in the future. The steps should include the action(s) for resolving the issue, the status of each action, and the date each action will be completed.

To address the current situation and ensure that similar incidents do not recur in the future, we have taken the following steps:

1、Staff Training Program: We have expanded our compliance incident team membership to improve our ability to deal with similar incidents in a timely manner. Expanding the Compliance Incident Team will also ensure parallel and timely investigation, resolution and reporting of compliance incidents. We will also provide occasional training for team members to ensure that multiple people understand key tasks. We plan to carry out the latest training at the end of September. We think that it is a better time to carry out relevant training on a quarterly basis.

2、Workflow documentation: We have made it clear on August 30th that we need to create detailed workflow documents for similar work tasks, including task assignments, work steps and deadlines, and a template has also been created for this process document so that everyone can follow the template to execute work process. At the same time, we support document sharing using the company's office software, so team members can view the current work progress in real time. And establish a culture of knowledge sharing to encourage the sharing of knowledge and experience among team members.

3、Backup personnel plan: In response to similar incidents, we plan to develop a backup personnel plan. In terms of critical tasks, at least one backup personnel can take over the work to ensure that tasks can be handed over to others smoothly when the person in charge of key tasks is unavailable. Also maintain some flexibility to ensure the team can adapt to staff turnover or changes. The program will take place in October, and next year we will also plan ahead for the annual self-assessment tasks, avoiding starting work at the last minute, which can provide buffer time for unexpected events.

Comment 4

[Antonis]

Hello Jasmine,

please find some comments / questions below:

(In reply to jasmine.tang from comment #3)

2、A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a requirement became applicable, a document changed, a bug was introduced, or an audit was performed.

20230727 CST(UTC+8) SHECA learned about this issue through a Bugzilla notification.
20230728 09:30 CST(UTC+8) The investigation was initiated by the Information Security and Compliance Department.
20230824 09:44 CST(UTC+8) SHECA completed the annual self-assessment and submitted it to CCADB.

Is this the complete timeline? You reference many things below that are not included here, so we can't get a complete picture of what happened when. It would be useful in assessing this report to know of the time frames, etc.

6、Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

We start by honestly acknowledging our concerns and we regret that we did not submit the self-assessment to CCADB within the allotted time.Here are the reasons for the bug's introduction and what to do so far:

Personnel changes: As mentioned in Bug 1838866, at the end of May we experienced a personnel change and we had to deal with the departure of a key colleague who was responsible for completing the CCADB self-assessment. This caused a disruption in the workflow as new colleagues needed time to adjust and understand the work.

Handover of tasks: Due to the urgency, we can only hand over the task to a colleague with relatively little experience. This colleague took longer to get used to the job, so it had an impact on our self-assessment work. It took the colleague longer to adapt and become familiar with the job, which also led to slower progress in the self-assessment.

Does SHECA currently adequately staff its CA operations? We've had issues with CAs in the past reporting that they relied in a single person, and due to a change in work, or vacation, etc. something was missed. One example is Bug 1708516.

Moreover, are all SHECA personnel adequately trained and capable of handling all tasks? Do they all go through their training refreshers and have the opportunity to address any questions they have? I am just wondering (due to the very simple timeline) what happened here: did the new colleague finish training and onboarding when they were given this task and responsibilities or not?

Despite the many challenges we faced, we have been doing our best to keep the self-assessment work going and ensure their quality and accuracy, culminating in the completion of the CCADB self-assessment exercise at the end of August. To avoid the problem from happening again, we have taken the following measures:

1、Enhanced internal communication: We now pay more attention to communication within the team, especially on critical tasks. We will hold occasional meetings to ensure that all team members understand the status and deadlines of tasks.
2、Task backup: We have established a plan for backup personnel to ensure that tasks can be handed over to others in the event that the person in charge of a key task is unable to continue working.
3、Plan Ahead: We start planning tasks in advance to prevent last minute emergencies. This gives us more buffer time to deal with unexpected situations.

Do you have a calendar that tracks all deadlines you have, short, medium, or long term? Perhaps it would be a good idea to start something like that, review it regularly, and set up some kind of alert for deadlines that are approaching fast.

3、Backup personnel plan: In response to similar incidents, we plan to develop a backup personnel plan. In terms of critical tasks, at least one backup personnel can take over the work to ensure that tasks can be handed over to others smoothly when the person in charge of key tasks is unavailable. Also maintain some flexibility to ensure the team can adapt to staff turnover or changes. The program will take place in October, and next year we will also plan ahead for the annual self-assessment tasks, avoiding starting work at the last minute, which can provide buffer time for unexpected events.

Do you have backup personnel currently for any task or position within the CA? Or will backup be introduced in October / Next Year for the first time? What do you define as a "critical task"?

Comment 5

[jasmine.tang]

(In reply to Antonis from comment #4)

2、A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a requirement became applicable, a document changed, a bug was introduced, or an audit was performed.

20230727 CST(UTC+8) SHECA learned about this issue through a Bugzilla notification.
20230728 09:30 CST(UTC+8) The investigation was initiated by the Information Security and Compliance Department.
20230824 09:44 CST(UTC+8) SHECA completed the annual self-assessment and submitted it to CCADB.

Is this the complete timeline? You reference many things below that are not included here, so we can't get a complete picture of what happened when. It would be useful in assessing this report to know of the time frames, etc.
20230727 CST(UTC+8) SHECA learned about this issue through a Bugzilla notification.
20230728 09:30 CST(UTC+8) The investigation was initiated by the Information Security and Compliance Department.
20230804 09:30 CST (UTC+8) The Information Security and Compliance Department checks whether there are other similar missed tasks and at the same time promotes the continuation of self-assessment work.
20230824 09:44 CST(UTC+8) SHECA completed the annual self-assessment and submitted it to CCADB.

6、Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

We start by honestly acknowledging our concerns and we regret that we did not submit the self-assessment to CCADB within the allotted time.Here are the reasons for the bug's introduction and what to do so far:

Personnel changes: As mentioned in Bug 1838866, at the end of May we experienced a personnel change and we had to deal with the departure of a key colleague who was responsible for completing the CCADB self-assessment. This caused a disruption in the workflow as new colleagues needed time to adjust and understand the work.

Handover of tasks: Due to the urgency, we can only hand over the task to a colleague with relatively little experience. This colleague took longer to get used to the job, so it had an impact on our self-assessment work. It took the colleague longer to adapt and become familiar with the job, which also led to slower progress in the self-assessment.

Does SHECA currently adequately staff its CA operations? We've had issues with CAs in the past reporting that they relied in a single person, and due to a change in work, or vacation, etc. something was missed. One example is Bug 1708516.
Yes, we believe we have a sufficient number of personnel in place to support our CA operations. In terms of timely response to browser and CAB forum requests, our compliance team has added two members to follow up on the matter to ensure that the problem can be handled in a timely manner. This team is well configured to meet our current work needs.

Moreover, are all SHECA personnel adequately trained and capable of handling all tasks? Do they all go through their training refreshers and have the opportunity to address any questions they have? I am just wondering (due to the very simple timeline) what happened here: did the new colleague finish training and onboarding when they were given this task and responsibilities or not?
Yes, we value the importance of training and task assignment, and all SHECA personnel have received training, including new employees. In addition to induction training, we will conduct refresher training from time to time to ensure that team members’ knowledge remains up-to-date. Team members also have the opportunity to address any issues they encounter on the job, and are encouraged to proactively raise and resolve issues. In response to the corresponding CAB forum requirements and browser requirements, we have expanded the relevant processing staff and provided them with sufficient onboarding training. We believe that they can complete this task independently in the next work.

Yes, new colleagues received onboarding training when they joined the company, which included learning about our company's CP&CPS, CCADB policy, root certificate program policy, EV guidelines, baseline requirements, etc. However, CCADB self-assessment is a completely new task for us, so new colleagues need some time to accept and handle new tasks. We also recognize that there is room for improvement. We continue to work hard to ensure that training and task assignments are seamless to meet the needs of the team and improve work efficiency.

Despite the many challenges we faced, we have been doing our best to keep the self-assessment work going and ensure their quality and accuracy, culminating in the completion of the CCADB self-assessment exercise at the end of August. To avoid the problem from happening again, we have taken the following measures:

1、Enhanced internal communication: We now pay more attention to communication within the team, especially on critical tasks. We will hold occasional meetings to ensure that all team members understand the status and deadlines of tasks.
2、Task backup: We have established a plan for backup personnel to ensure that tasks can be handed over to others in the event that the person in charge of a key task is unable to continue working.
3、Plan Ahead: We start planning tasks in advance to prevent last minute emergencies. This gives us more buffer time to deal with unexpected situations.

Do you have a calendar that tracks all deadlines you have, short, medium, or long term? Perhaps it would be a good idea to start something like that, review it regularly, and set up some kind of alert for deadlines that are approaching fast.
Yes, thanks for the advice. We have set up a calendar to track events, such as various tasks or surveys initiated by the CAB forum, browser or CCADB. This calendar is one of our key tools for staying organized and time-managed. Additionally, we will review this calendar regularly and set reminders and alerts for upcoming deadlines to ensure we complete tasks on time.

3、Backup personnel plan: In response to similar incidents, we plan to develop a backup personnel plan. In terms of critical tasks, at least one backup personnel can take over the work to ensure that tasks can be handed over to others smoothly when the person in charge of key tasks is unavailable. Also maintain some flexibility to ensure the team can adapt to staff turnover or changes. The program will take place in October, and next year we will also plan ahead for the annual self-assessment tasks, avoiding starting work at the last minute, which can provide buffer time for unexpected events.

Do you have backup personnel currently for any task or position within the CA? Or will backup be introduced in October / Next Year for the first time? What do you define as a "critical task"?
Yes, due to the sudden adjustment of the organizational structure within the group, we lost key personnel. This kind of thing has never happened before, so we lack experience in handling it. In light of our previous staffing shortfalls, we have taken action to first replenish our compliance team members while building a back-up staff to ensure we do not face similar issues again. At present, the staff has been fully supplemented, and other relevant training and backup mechanisms will be officially implemented in October. This backup mechanism will become our normal operation to ensure that we always have sufficient personnel resources to support the continued operation of CA.

We consider "critical task" to mean those tasks or responsibilities within our organization that are critical to the smooth conduct and success of a project or operation. These tasks are often characterized by high impact on the project outcome, time sensitivity, and requiring specific skills or knowledge. They are those tasks that, if ignored because the person responsible is unavailable, could cause significant disruption, delay, or negative consequences to a project or operation. In our context, examples of critical task might include activities involved in all normal CA operations, project management, key customer communications, system maintenance, and any activity critical to achieving our organizational goals. All work involving full life cycle management such as certificate issuance, renewal, and revocation are critical tasks. As part of our backup staffing plan, we will identify and prioritize these critical tasks to ensure that we have contingency plans in place to continue them seamlessly in the event that the primary personnel responsible for these tasks are unavailable.

Comment 6

[jasmine.tang]

The above information is our response to the question.Please let us know if there are any further questions or concerns about this bug.

Comment 7

[Ben Wilson]

I plan to close this item on Wed. 11-Oct-2023, unless there are additional questions or concerns expressed.