Open Bug 1893546 Opened 2 months ago Updated 17 days ago

e-commerce monitoring gmbh: failure to follow incident report requirements

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
ASSIGNED

People

(Reporter: agwa-bugs, Assigned: ca)

Details

(Whiteboard: [ca-compliance] [policy-failure] [external])

e-commerce monitoring gmbh (aka GLOBALTRUST) has repeatedly failed to adhere to Mozilla's incident reporting requirements.

Leaving questions unanswered for more than a week:

Failing to provide weekly updates (sometimes going months without any update):

Providing insufficient incident reports:

In Bug 1883711 Comment 8, GLOBALTRUST claimed that they couldn't answer 2 simple yes/no questions because it was their audit week. This raises a concern that GLOBALTRUST is not adequately staffed to handle incidents in parallel with their other obligations as a publicly-trusted CA.

As with other CAs (e.g. Bug 1708516, Bug 1563579, Bug 1572992), GLOBALTRUST's repeated failure to adhere to Mozilla's incident reporting requirements warrants its own incident, so we can understand why GLOBALTRUST is failing, and track what they are doing to improve their compliance.

Assignee: nobody → ca
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance] [policy-failure]

Dear All,

this is just to confirm we have received this report. We will provide you with an answer in due time.

Best regards,
Daniel

Thank you Andrew for making this incident.

Looking at ecommerce here: https://crt.sh/cert-populations, I'm not entirely sure what we're getting out of them having keys to the entire web.

So far, this is what I've seen from this CA:

  • The lack of personnel in responding to incidents
  • The months long delay in solving problems
  • Claiming to have solved an incident, and then not having it actually solved

I had started a conversation about this in the past:

https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/HFMEAMUe7v4/

And there have been concerns about this CA in other threads:

https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/P-0JVYdXTDM/

Serious question if root programs are comfortable with this CA being part of their trust stores:

  • What is the incentive for other CAs to commit to the rules?
  • What is the incentive for other CAs to invest in pushing the ecosystem forward? Those efforts cost time & money. If CAs that do not do this are welcomed in the root programs, how does a CA that has been working on pushing the ecosystem forward keep being able to justify the expenses those efforts require?

(In reply to Daniel Zens from comment #1)

Dear All,

this is just to confirm we have received this report. We will provide you with an answer in due time.

Best regards,
Daniel

Do we have an estimate on what 'due time' means?

Flags: needinfo?(ca)

Incident Report

Preface

Since this bug and its responses do not pertain to a single specific incident but rather encompass a range of different aspects spanning over an extended period, which require discussion at a higher level, we would like to provide additional context and explain the underlying background.

We consider this a significant opportunity and appreciate the contributions of all individuals, past and future. Furthermore, we see this bug as a chance to address any questions or concerns that have been raised elsewhere, even if they may be considered off-topic from our perspective.

About

e-commerce monitoring gmbh (in the following: ECM), founded in 2002, initially provided electronic signatures for Austrian companies under local e-invoicing laws under the brand name A-CERT. In 2008, we gained the right to issue e-government signatures in Austria.

Accredited as an EU-compliant certification service provider in 2014 under the EU Signature-Directive, ECM has later been a Qualified Trust Service Provider (QTSP) under the brand GLOBALTRUST in accordance with eIDAS. As such, We undergo continuous oversight by the Austrian Supervisory Body (RTR/TKK) while being regularly evaluated by suitable Conformity Assessment Body using a variety of standards outlined by ISO/IEC, eIDAS/ETSI, national Signature Laws and the CA/Browser Forum and Root Store Policies.

In the course of an increasing internationalization of our customer groups and with the aim of becoming a full service provider, our most recent RootCA GLOBALTRUST 2020 has been included in the major Browser/OS Programs since 2021.

In 2024, ECM has been acquired by AUSTRIA CARD, which also involved a change of the top-level management. The group and its management have a vested interest in ensuring that ECM fully adhere to all the applicable rules.

Summary of this incident

ECM exhibited insufficient procedural and operational standards in regards to incident handling. This has been observed in delayed responses lacking detail and in the slow resolution of technical issues. ECM realizes and acknowledges this claim and has begun to take actions. Further, we have evaluated our organization for potentially related issues to remediate at the same time.

Impact

The policies of the Browser/OS Stores were not adhered to. Further, the expectations of individual participants in the open WebPKI community in regards to timeliness and transparency may have been disappointed. The public, who trust the services of ECM, were not able to obtain a comprehensive picture of the certificate operations and underlying policies and practices.

In the course of incident, ECM has not stopped further issuance of certificates, since this is not related to issuance of certificates.

Timeline

All times are UTC.

2015-05-29

2016-08-31

2017-09-18

2018-02-22

2019-01-18

2019-08-28

2019-08-28

2019-08-29

2019-12-20

2020-01-04

2020-01-04

2020-01-06

2020-01-27

2020-03-30

2020-04-05

2020-12-22

2021-01-08

2021-01-08

2021-02-05

2021-02-27

2021-03-09

2021-03-09

2021-04-28

2021-05-25

2021-06-08

2021-06-12

2021-06-12

2021-06-17

2021-06-17

2021-06-23

2021-07-08

2021-07-20

2021-09-13

2021-09-15

2022-06-20

2023-02-07

2023-04-28

2023-04-28

2023-06-12

2023-10-12

2023-10-30

2023-12-05

2024-02-02

2024-02-08

2024-02-15

2024-02-15

2024-02-23

2024-03-05

2024-03-19

2024-03-28

2024-04-04

2024-04-17

2024-04-26

2024-05-03

2024-05-07

Root Cause Analysis

Prehistory to the Root Cause

2019/20 - 2022:

Upon our initial participation in the root programs, we designated two experienced individuals specifically to oversee external communications related to compliance matters. It is crucial to emphasize that the mention of "two" pertains solely to their authorization for external communication. Behind these appointed individuals, a comprehensive team consistently supported their endeavours. Moreover, both individuals undertook supplementary responsibilities, including representing the company during eIDAS conformity assessments. While acknowledging areas for enhancement as early as during that period, it is noteworthy that they effectively contributed to our participation in the major Root Programs.

2022/23 - now:

The challenges initially arose following the departure of key employees towards the end of 2022 and the beginning of 2023, resulting in an overload of responsibilities for our compliance staff. When we were ultimately acquired by Austria Card, one of the two individuals became unavailable due to management changes affecting their role. The situation resulted in a brief period where only one individual, without backup, was responsible for incident reports, coinciding with the annual full audit and various other compliance matters, such as NIS-2 and eIDAS 2. Unfortunately, this is where the recent incidents occurred.

From this point of view, we consider the staffing deficiencies the root cause of this incident.

Factors contributing to the Root Cause

  • Influx of concurrent projects
    • SSL/TLS
    • S/MIME, Codesigning, Other certificate types
    • eIDAS: QES, QSeal, QWAC, QTimestamping
    • Other (local) projects with relation to cryptography and X509v3.
  • Period of adjustment, as we integrated into
    • new management structures
    • new operational frameworks
  • Evolving regulatory landscape
    • NIS-2, eIDAS 2
    • ETSI revisions
    • CA/Browser Forum
  • Need to adapt to changing technical requirements
    • Focus on future proof key parameters together with customers and partners
  • While maintaining ongoing operations.

Lessons Learned

What went well

  • The new management recognizes the significance of adhering to pertinent regulations. Specifically, our reporting process aligns with Austria Card's communication policy, a recognition that has also been noted in part within the bug reports.

What didn't go well

  • Despite all earnest efforts, while we have improved compared to before, we have still not reached the goal.
  • The incident did not come as a surprise, as signs of its occurrence had already been evident.

Where we got lucky

  • As appropriately highlighted in one of the bug reports, the situation could have been exacerbated by the occurrence of other incidents, for example a simultaneous security incident, which would have further complicated our efforts.

Action Items

Action Items Detail

It is noteworthy that the deficiencies discussed in this bug were identified diligently throughout the entire acquisition process and are currently being resolved or are already resolved:

Staff:

  • All vacant positions that remained unfilled for an extended period have now been filled.
  • In terms of technical compliance, ECM has already obtained approval for the addition of two full-time positions and have begun discussions with some candidates.
  • Similarly, for legal compliance and external communication, two individuals were appointed in mid-March and disclosed to the CCADB. However, their public appearances in this forum are pending. Root Store Operators may verify this here via https://ccadb.my.salesforce.com/500TO00000439vjYAA
  • Moreover, with the recent relocation of ECM's office space, our team is now more closely knit with the group’s team, affording us access to significantly expanded DevOps personnel, as well as various non-human resources. Root Store Operators may verify this here: https://ccadb.my.salesforce.com/500TO000007xHOzYAM
  • We have implemented serious and effective measures such as clear improvements at the job description and work contract level as well as the organigram, in order to relieve compliance personnel from all non-directly relevant tasks and impediments. (CW 7)

Processes and Tools:

  • We have thoroughly revised the internal manuals concerning incident reporting, streamlining several time-consuming aspects. Among these enhancements, we have identified a smart solution to significantly streamline the often laborious task of composing timelines.
  • We are now fully integrated into the collaboration software of Austria Card, providing us with a more efficient method to internally document and address external bugs in our processes, tools, communication channels, and time/reminder management systems.

Other:

  • We have significantly increased the number of participants, scope, and frequency of our regular audit synchronization meetings.
  • Several other trivial but effective measures, such as improvements of our automatic task reminders etc.

All Action Items Overview

Please note that it is not feasible to provide a specific due date for all future action items, as their outcomes are contingent upon community involvement and other external factors beyond our control.

Action Item Kind Due Date
Onboard ECM staff to AC Improve Done
----------- ---- --------
Streamline responsibilities Prevent, Improve Done
----------- ---- --------
Add 2 existing compliance staff members to POCs Prevent, Improve Done
----------- ---- --------
Approval for adding 2 headcounts tech staff Prevent, Improve Done
----------- ---- --------
HR discussions tech staff candidates Prevent, Improve ongoing, completion planned end of May
----------- ---- --------
Company relocation Improve Done
----------- ---- --------
Integration ECM in collaboration/productivity tools Improve Done
----------- ---- --------
Various Bug Fixes in this Forum Resolve, Prevent, Improve ongoing
----------- ---- --------
Ongoing communication in this forum Resolve, Prevent, Improve ongoing
----------- ---- --------
Fix this Bug Resolve, Prevent, Improve ongoing
----------- ---- --------
Revise process, lessons learned Prevent tbd
----------- ---- --------
Regularly scheduled internal Q2 audit Prevent, Detect, Improve CW 29
----------- ---- --------
Improvements based on revision + internal audit Prevent, Improve CW 29-30
----------- ---- --------

Appendix

As no specific certificates are affected, we will repurpose this section as a "Useful Links" section to assist the community in evaluating our policies and practices over the past years. This is particularly important, as we have become aware of a typo in our current list of historic policy URIs that requires correction.

GLOBALTRUST Certificacte Policy

GLOBALTRUST Certificate Practice Statement

Flags: needinfo?(ca)

Still getting through this response but can you please explain the disconnect here:

In the course of incident, ECM has not stopped further issuance of certificates, since this is not related to issuance of certificates.

A few sentences above:

The policies of the Browser/OS Stores were not adhered to.

Flags: needinfo?(ca)

And a small follow up, have you stopped issuance now that all these problems have surfaced?

I can appreciate that ECM's management are freshly aware of this issue and are treating it as mainly a personnel matter. The issue I have with that report is that it doesn't give us any material changes or timeframes for the pre-existing non-compliance.

If ECM have identified they are not compliant and need time to address them all internally, then they must be honest and admit they are not currently capable of issuance that is compliant. I won't address every item posted, but the issues seen so far do not appear to come from a change in personnel solely in late '22. Focusing on addressing process changes for future ballots is missing that the current situation is non-compliant.

Although I did notice a problem that encapsulates this entire situation:

Timeline

...
2022-06-20

...
2023-12-05

Even in your timeline for audits the wrong month is used. The audit wasn't December 5th, it was May 12th. Given your current audit should be happening right now that is... concerning.

Whiteboard: [ca-compliance] [policy-failure] → [ca-compliance] [policy-failure] [external]

I’d expect that on an incident where a CA is being told they’ve been slow to respond, and after the CA acknowledges it, they’d at least try to be fast about responding to incoming questions.

If you have this level of operational issues right now, why have you not yet stopped issuance? Can you guarantee that if you misissue a certificate in the next few days, you’d have the capacity to respond to that effectively?

(In reply to Daniel Zens from comment #4)

2022/23 - now:

The challenges initially arose following the departure of key employees towards the end of 2022 and the beginning of 2023, resulting in an overload of responsibilities for our compliance staff. When we were ultimately acquired by Austria Card, one of the two individuals became unavailable due to management changes affecting their role. The situation resulted in a brief period where only one individual, without backup, was responsible for incident reports, coinciding with the annual full audit and various other compliance matters, such as NIS-2 and eIDAS 2. Unfortunately, this is where the recent incidents occurred.

From this point of view, we consider the staffing deficiencies the root cause of this incident.

Can you clarify this analysis with regards to the comments made by e-commerce about the acquisition in https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/P-0JVYdXTDM/m/AxZMtMd1AAAJ

If there were staffing deficiencies due to management changes, why was e-commerce saying the opposite in February 2024?

On Feb 8, 2024 at 7:19:33 AM e-commerce monitoring wrote:

The takeover of the company also includes the taking over of the established staff which results in no changes except top management and e-commerce monitoring GmbH will continue to adhere and operate according to the respective policies.

On Feb 23, 2024, 9:36:13 AM e-commerce monitoring wrote:

The takeover of the company includes the taking over of the existing, trained and trusted staff which results in no changes except top management. e-commerce monitoring GmbH continues to provide certification and trust services according to the respective policies.

(In reply to amir from comment #5)

In the course of incident, ECM has not stopped further issuance of certificates, since this is not related to issuance of certificates.

A few sentences above:

The policies of the Browser/OS Stores were not adhered to.

An incident may be based on facts that justify or necessitate halting further certificate issuances. The incident we are discussing in this bug, however, is not related to certificate issuance, but rather to a specific expected response behavior, format, and frequency. Therefore, halting certificate issuance is not applicable in this case.

(In reply to amir from comment #6)

And a small follow up, have you stopped issuance now that all these problems have surfaced?

No.

(In reply to Wayne from comment #7)

Even in your timeline for audits the wrong month is used. The audit wasn't December 5th, it was May 12th. Given your current audit should be happening right now that is... concerning.

You are right. This is a clerical error (05-12 versus 12-05). The 2024 on-premise audit has been carried out in CW 16.

(In reply to Mathew Hodson from comment #9)

If there were staffing deficiencies due to management changes, why was e-commerce saying the opposite in February 2024?

We communicated in February that there would be no personnel changes except at the top level management, and that the plan is to continue pursuing the current activities. That is correct, and it does not imply any contradiction.

Flags: needinfo?(ca)

(In reply to Daniel Zens from comment #10)

We communicated in February that there would be no personnel changes except at the top level management, and that the plan is to continue pursuing the current activities. That is correct, and it does not imply any contradiction.

Could you please clarify what you mean? Are you saying that the information provided in February was correct at the time, but your plans have changed? The report provided mentions that staffing deficiencies were an internal concern but this is not what e-commerce were saying at the time. To that end the same question must be asked:

If there were staffing deficiencies due to management changes, why was e-commerce saying the opposite in February 2024?

Further, on the prospect of continued mis-issuance I'll restate my previous comment which is unaddressed:

I can appreciate that ECM's management are freshly aware of this issue and are treating it as mainly a personnel matter. The issue I have with that report is that it doesn't give us any material changes or timeframes for the pre-existing non-compliance.

If ECM have identified they are not compliant and need time to address them all internally, then they must be honest and admit they are not currently capable of issuance that is compliant. I won't address every item posted, but the issues seen so far do not appear to come from a change in personnel solely in late '22. Focusing on addressing process changes for future ballots is missing that the current situation is non-compliant.

ECM are solely focused on the future when the pre-existing issues are still not dealt with. I still cannot see where the logic comes from to continue issuing certificates when ECM are internally aware they are non-compliant and will not be compliant in the short-term.

I will be blunt: the policy issues I have mentioned are from a minor glance on the most simple questions. A more substantial audit would not be kind, and ECM need to restructure how they approach issuance entirely. The current mixture of e-signing and sealing requirements mixed into TLS is doing yourselves no favors. I have grave concerns over what your current audit will say if your past audits are anything to go by.

As you might know, browsers have decided to remove e-commerce monitoring GmbH (ECM) with its Root Certificate "GLOBALTRUST 2020" from their Root Programs as of June 30, 2024. Certificates issued before this date will retain their full validity.

The reasons for the removal have been comprehensively discussed Bugzilla forum. We acknowledged and accepted the decision. We have identified the shortcomings in our processes, particularly related to reaction time. Consequently, we are taking these issues very seriously and are committed to address them. An action plan is being rolled out to restructure our Certificate Authority (CA) functions. Our goal is to be included again in the Root Programs.

ECM’s shareholder, AUSTRIA CARD, is committed to regains full compliance with the Browser/OS Root Store Policies. This commitment, which is strongly supported by our recently changed management, underscores our dedication to maintaining the widest compatibility and coverage.
As an immediate action, and until full remediation, ECM has ceased the issuance of TLS certificates according to the CA/Browser Forum Requirements. TLS certificates will be provided solely based on Regulation (EU) No 910/2014, Annex IV, as recently amended by Regulation (EU) 2024/1183 (“QWACs”). Certificates for interoperability testing purposes are excluded from this decision.

ECM, with its product lines GLOBALTRUST and TRUST2GO, is a Qualified Trust Service Provider (QTSP) according to EU eIDAS regulation and is under continuous supervision by the Austrian regulatory authority (RTR/TKK). Our activities are regularly evaluated by an accredited conformity assessment body based on numerous standards (e.g., eIDAS, ETSI), which include comprehensive logical, physical, and organizational security measures.

Our goal is to rebuild trust and demonstrate our commitment to upholding the highest standards in our industry.

For inquiries, please contact the Compliance & Product Management team, Attn: Mr. Daniel Zens, at ca@globaltrust.eu

You need to log in before you can comment on or make changes to this bug.