Closed Bug 1862004 Opened 2 years ago Closed 1 year ago

e-commerce monitoring GmbH: Delayed revocation

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: bwilson, Assigned: ca)

Details

(Whiteboard: [ca-compliance] [leaf-revocation-delay] [external])

Attachments

(1 file)

Reply to Aaron Gable from comment #12
7.18 KB, text/plain
Details

This bug is required as a follow-up to Bug # 1815534. The CA is required to file a separate incident report for the delayed revocations mentioned there. See
https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation
As outlined in the wiki page linked above, the incident report must include e-commerce monitoring's "analysis to determine the factors that prevented timely revocation of the certificates, and include a set of remediation actions in the final incident report that aim to prevent future revocation delays."

Summary

ongoing discussion about the bugs https://bugzilla.mozilla.org/show_bug.cgi?id=1815534 (open) and https://bugzilla.mozilla.org/show_bug.cgi?id=1830536 (closed)
Concern the same action that was completed in March 2023.

Incident report

Described in detail in

Impact

Was explained in detail in https://bugzilla.mozilla.org/show_bug.cgi?id=1815534 (comment 17)

Timeline

See in

Root Cause Analysis

Root not affected

Lessons Learned

What went well

No internet user had any disadvantages.

What didn't go well

Apparently some participants in the discussion forum did not understand the European approach to the matter. It is a matter of course for us to clearly state different positions and only then take the necessary actions. It is possible that some people do not see differences of opinion this way.

Where we got lucky

not applicable

Action Items

Action Item child Due Date
Report Sending 2023-10-31

###Appendix

Details of affected certificates

none other than those already detailed in https://bugzilla.mozilla.org/show_bug.cgi?id=1830536 (comment 1).

Whiteboard: [ca-compliance] [leaf-revocation-delay] → [ca-compliance] [leaf-revocation-delay] [external]

Thank you for the initial response, but this is a standalone report, and it needs to be improved to include the expected content as described on the CCADB incident report page. The incident report format on ccadb.org was updated in October 2023 to make reports more useful and effective for the community. More detailed guidance was provided, and templates were added to facilitate this intention.

Request for update #1: The Summary section should be updated to reflect the specific nature of this incident, which is a failure to comply with Section 4.9.1.1 of the TLS BRs. This section aims to provide enough context for readers to understand the rest of the report.

Request for update #2: The Impact section should be updated to describe the number of certificates affected by delayed revocation explicitly. Additionally, cross-referencing the contributing incident is okay, but a reader should be able to clearly understand the impact from reading this report.

Request for update #3: The Timeline section needs to be updated and explicitly detailed for this incident. Other related incidents that contributed can and should be included, but this section must consist of all events and actions leading up to and taken during and after the incident. All times should be in UTC and at least have minute-level granularity. The guidance on the CCADB incident report page states the following events must be included:

  • All policy, process, and software changes that contributed to the Root Cause
  • The time at which the incident began
  • The time at which the CA Owner became aware of the incident
  • The time at which the incident ended
  • The times at which issuance ceased and resumed, if relevant

Request for update #4: The Root Cause Analysis section needs to be updated to include a detailed analysis of the combined conditions that created the issue. Comment #20, in the separate, contributing incident, described multiple interpretations of timing, where the most generous interpretation was 7 days, which does not comply with Section 4.9.1.1 of the TLS BRs. In this specific incident, what led to the delay in revocation?

Request for update #5: The Lessons Learned section should be updated to better describe the three subsections (i.e., what went well, what didn’t go well, where you got lucky). The items in “what went well” intend to help others in the community avoid the same situation. The items in “what didn’t go well” and “where we got lucky” should each have an action item assigned in the next section (i.e., Action Items), which intends to ensure a similar incident does not reoccur in the future.

Request for update #6: After improving the Lessons Learned section, the Action Items section needs to be better detailed. The Actions Items list should be updated to minimally include a Prevent, Mitigate, or Detect "kind" of action for each thing under "What didn't go well." It should consist of an action to address "Where we got lucky" (so that one does not have to rely on luck in the future). Ideally, one of each of these three types is included in the list of action items. This guidance is reflected in the CCADB incident report page.

Flags: needinfo?(ca)

Dear All,

Thank you for your feedback and your clear update requests. It helps e-commerce monitoring GmbH to improve our understanding of the required level of incident documentation. e-commerce monitoring GmbH also welcomes this opportunity to enhance our incident handling in the future in order to better meet the community’s expectations.
We will provide updates to this bug no later than Friday 16th.
Thank you for your patience.

Best regards,
Daniel Zens

Flags: needinfo?(ca)

Incident Report

Summary

On 2023-02-07 e-commerce monitoring has issued multiple signed data structures with the serial number 10:45:67:49:88:c4:db:00:76:89:b9:.Two Mozilla Incident Bugs had been filed to cover the aspects of duplicate serial number and SCT extension in a pre-certificate. After discovering the issue, e-commerce monitoring GmbH has quickly fixed the bug and analyzed the underlying causes. However, those three data structures (considered to be one and the same certificate serial number 10:45:67:49:88:c4:db:00:76:89:b9) were revoked not before 2023-03-30.

Impact

The revocation deadline of BR 4.9.1.1 was missed for two pre-certificates, one existing leaf certificate and one presumed-to-exist certificate (MRSP 5.4). All share the same serial number. All were issued under the OV SSL Policy. e-commerce monitoring GmbH interrupted the issuance of SSL certificates for a short time in the course of updating our system in the overall context, but not specifically because of the aspect of delayed revocation as described in this ticket.

Timeline

2023-01-03

  • 10:23 UTC An update was made to our issuance tool in order to use the then-current CT log servers

2023-02-07

  • 10:23 UTC: An SCT query was made for "10:45:67:49:88:c4:db:00:76:89:b9" which was insufficient according to our specifications (only one SCT instead of three). Since this SCT was suitable, it was stored internally. (SCT response 1)
  • 10:38 UTC: Another SCT query was made, the response of which met our specifications These SCT values were stored internally. (SCT response 2)
  • 10:50 UTC: Based on the correct SCT entries from SCT response 2, the leaf certificate was issued The SCTs were used as embedded SCTs.
  • 14:04 UTC: Internal controls showed that two pre-certificates were issued for one leaf certificate. (10:45:67:49:88:c4:db:00:76:89:b9)
  • 14:30 UTC (rounded) A detailed analysis revealed that some Google ct-log-servers changed the format of their URLs at the turn of the year 2022-2023. Our software was converted to the new URL format and the entire module was subjected to a comprehensive revision. No server end-entity certificates were issued during this conversion and revision process.
  • 15:00-16:00 UTC (rounded) A subsequent review was carried out, primarily to avoid future disruptions due to unexpected changes in the URL format and to speed up the issuing process of server end-entity certificates.
  • 19:33 UTC Bug 1815534 was filed https://bugzilla.mozilla.org/show_bug.cgi?id=1815534

2023-02-08

  • 9:30 UTC Internal acknowledgement of Bug 1815534. Forward to the responsible staff.
  • 10:00-11:30 UTC (rounded) internal discussion, coordination of the next steps. A deeper post issuance lint tests was carried out, which did not show any warning nor error. However it can be confirmed the leaf certificate includes the inappropriate SCTs (as in https://bugzilla.mozilla.org/show_bug.cgi?id=1815534#c0). It has been discussed the certificate is likely problematic, however it also could be confirmed it is successfully used by the subscriber.
  • 14:25 UTC first response to Bug 1815534, more detailed response later the same day. Ongoing discussions for the next days in Bug 1815534 and internally with our staff.

2023-02-20

  • 12:57 UTC e-commerce monitoring reaches out to the community asking if revocation is required. At this stage, e-commerce monitoring considered this was not necessarily the case.

2023-03-22

  • 20:30 UTC confirmation by community members that revocation is necessary and should already have been done.

2023-03-27

  • from 9:00 UTC several contacts between our customer service and the subscriber. Again it is confirmed the certificate, though problematic, is usable. Moreover, the is part of an infrastructure with high availability requirements and subject to specific policies and processes at the subscriber level. The customer was advised that revocation is necessary and will happen. A deadline of no later than 2023-03-30 was agreed.
  • 20:09 UTC A certificate record for the use in issuance tool was prepared in order to replace the problematic certificate. Issuance itself had to be postponed to the following day due to the late time of day, as otherwise the dual control principle could not have been processed. (According to our policies, any issuance or revocation requires a four-eyes principle. This is enforced directly within the issuance tool which requires 2-FA login by two authorized individuals.)

2023-03-28

  • 10:28 UTC A correct certificate was issued in order to replace the revoked.

2023-04-28

2023-10-30

  • 15:06 UTC This bug was filed

Root Cause Analysis

The requirements set forth by the BR requirements in regards to the revocation deadline are known and implemented.

In our certificate issuance and revocation tool we use a two step concept:

  • revocation is requested. This can be triggered by the subscriber via a web form, by customer service staff based on communication with the subscriber (e.g. revocation request via phone) and addition by staff in certain trusted roles (e.g. a malformed certificate is detected during internal audits).
  • revocation processed: a second staff member validates the revocation request and execute.

In general, e-commerce monitoring GmbH executes every revocation, regardless of the reason, within 24 hours as standard. For certain policies, shorter deadlines must be provided according to eIDAS and Austrian Signature Laws (maximum of 6hours). Revocation service is provided as a 24/7 service and supported by a 24-hour on-call service. Revocation requests triggered by the web form provoke notifications to several emergency addresses (email, SMS). Notifications repeat until revocation is completed.

Obviously, is it neither implemented nor much gain to enforce any revocation request be executed under any circumstance under 24hours. Reasons for postponing may differ widely. (E.g. circumstances are unclear, right to request revocation remains unclear, subscriber requests revocation for a specific date in the future, … )

In this case, we had to carefully balance:

  • subscriber's interests and those of their users
  • the policy requirements,
  • the expectations of the community,
  • the fact that the certificate, though problematic, is usable and is already in use in a special infrastructure.
  • the security impact of not revoking the certificate earlier

To that end we have come to the well-founded conclusion that the interests of the subscriber and its users respectively clearly outweigh the interests of a revocation in due time.

Lessons Learned

We realize a detailed disclosure, in particular by proactively filing a Bug in advance, should have been made. If for whatever reason a similar situation re-occurs, we will not go the same way. We have thoroughly discussed our approach to future incident handling in general. Due to the fact that first two and then three bugs were open due to the same incident, in which the different dimensions were already discussed, we were of the opinion that large-scale references were sufficient here in this bug. This view is incorrect. ("The incident report may well repeat things previously said in discussions or Bugzilla comments. This is entirely expected.") We are committed to transparently communicate any failures including full disclosure of remediation measures and continuously improve our incident responding under the Root Program Policies.

What went well

After discovering the issue that originally led to the problematic certificate, e-commerce monitoring GmbH has quickly fixed the bug and analyzed the underlying causes and has taken measures to avoid the incident from happening again. This is demonstrating well-documented and rapidly available processes maintained by highly experienced staff. All customer data and cryptographic procedures/methods of all certificates were correct and secure at all times.

What didn't go well

It would have been better to assign a higher priority to documentation and external communication.

Where we got lucky

Only one case was affected

Action Items

No pending actions

Appendix

Details of affected certificates

Correctly-formed pre-certificate with serial number 10:45:67:49:88:c4:db:00:76:89:b9:
https://crt.sh/?sha256=6f1fd13b3c72fd064e01b4fa810d44a06bde36f44876179a45bf752ae988d6cf

Precertificate with serial number 10:45:67:49:88:c4:db:00:76:89:b9 containing an SCT extension,
in violation of RFC6962 which only defines the SCT extension for final certificates, not pre-certificates:
https://crt.sh/?sha256=c967a3109b953f8c14ffda5901db5f76d1bc0b6f84a3b6163ce14cd16d12086a

Certificate with serial number 10:45:67:49:88:c4:db:00:76:89:b9 containing SCTs with invalid signatures:
https://crt.sh/?sha256=5ee21e3bb051f280dec1339e83f60969919837da84a554007553edfb77c74a5f

"To that end we have come to the well-founded conclusion that the interests of the subscriber and its users respectively clearly outweigh the interests of a revocation in due time."

I feel this is clear enough evidence that this CA has utter disregard for the Mozilla, their users, root-programme requirements, as well as CA-B-Forum guidelines.
I do not believe they can be in the position of a trusted CA any longer, and a discussion must be initiated to distrust this CA and remove them from Mozillas trust-store.

It is my personal opinion that the Root Cause Analysis in this incident report is insufficient and arguably factually incorrect. I believe the fundamental cause of this incident is reflected in this item from the timeline:

2023-02-20

  • 12:57 UTC e-commerce monitoring reaches out to the community asking if revocation is required. At this stage, e-commerce monitoring considered this was not necessarily the case.

At this point it was already approximately 13 days after the CA had been informed (by Bug 1815534 being filed) that the certificate was mis-issued. The BRs are very clear that certificates which are "not issued in accordance with these Requirements" must be revoked within 5 days.

The issue is less that e-commerce monitoring GmbH decided not to revoke the certificate, and much more that they didn't know they had to. This is in my opinion a gross failure of understanding the Baseline Requirements. I don't believe it is acceptable for a CA to be presented with clear evidence of misissuance and then take 13 days to even ask if revocation is necessary.

As no further questions have been raised on the specific issue, we ask that the ticket be closed.

Flags: needinfo?(bwilson)

My Comment #6 was never addressed.

The incident covered in this report is a "failure to revoke within the required time period" incident. The timelines presented here and in issue 1815534 make it clear that e-commerce monitoring GmbH failed to revoke this certificate in the required time period because they did not understand that revocation was necessary. Again, 51 days passed between e-commerce monitoring GmbH becoming aware that the certificate was misissued and the certificate actually being revoked.

However, the root cause analysis does not discuss why e-commerce monitoring GmbH believed revocation was not necessary. And the Action Items (of which there are none) do not present any steps that e-commerce monitoring GmbH is taking to ensure that they better understand the requirements in the future.

As stated in Comment #6, I believe this Root Cause Analysis is insufficient and incorrect, and it has not been updated since that time.

Dear all,

thank you very much for your contributions so far. We honestly value all constructive feedback and respect all opinions. We subject every comment to thoughtful analysis and, if necessary, derive binding recommendations for our future actions.

However, please understand that it is not possible for us to respond to all inputs, in particular such that literally use the phrases "It is my personal opinion" (https://bugzilla.mozilla.org/show_bug.cgi?id=1862004#c6) or "I feel" (https://bugzilla.mozilla.org/show_bug.cgi?id=1862004#c5).

That said, what is written in the incident report remains unchanged.

Best regards,
Daniel

To date we still have no action items on what, if anything, e-commerce monitoring GmbH will do to stop a future delayed revocation. I'm still not convinced they're fully aware of their obligations. I was writing this comment as I reviewed, so let's start:

As outlined in #1815534 it takes the community over the course of a year to get this CA to do the absolute bare minimum and product a report for their original issue.

Beyond the abysmal incident report that was forced out of the company over months after very restrained prodding by Ben I'm not seeing any attempt at a serious response to these issues.

Obviously, is it neither implemented nor much gain to enforce any revocation request be executed under any circumstance under 24hours. Reasons for postponing may differ widely. (E.g. circumstances are unclear, right to request revocation remains unclear, subscriber requests revocation for a specific date in the future, … )

In this case, we had to carefully balance:

  • subscriber's interests and those of their users
  • the policy requirements,
  • the expectations of the community,
  • the fact that the certificate, though problematic, is usable and is already in use in a special infrastructure.
  • the security impact of not revoking the certificate earlier

To that end we have come to the well-founded conclusion that the interests of the subscriber and its users respectively clearly outweigh the interests of a revocation in due time.

Will e-commerce monitoring GmbH explain where these factors come from? Please note in detail:

  • Your own Certificate Practice Statement applicable to 'GLOBALTRUST SERVER OV 1' at time of issuance
  • Baseline Requirements
  • Root Program Policies for: Apple, Chrome, Microsoft, Mozilla given that is where you are currently trusted

Additionally I am noting that the certificate policy page does not have a list of older CPS documents, such as what would apply to this pre-cert. This is a further violation of the Baseline Requirements v2.04:

2.3 Time or frequency of publication
The CA SHALL develop, implement, enforce, and annually update a Certificate Policy and/or Certification Practice Statement that describes in detail how the CA implements the latest version of these Requirements. The CA SHALL indicate conformance with this requirement by incrementing the version number and adding a dated changelog entry, even if no other changes are made to the document.

2.4 Access controls on repositories
The CA shall make its Repository publicly available in a read‐only manner.

In looking at the most recent CPS covering 'GLOBALTRUST SERVER OV 1', I am deeply concerned as they note on page 18:

Version 2.0h Changes 15th January 2021

  • Minor alignments according to browser requirements
  • Additional technical clarifications
  • Editorial correction of spelling and phrasing errors

Version 2.0i Changes 14th January 2022

  • Annual review with no material changes
  • Editorial correction of spelling and phrasing errors

Version 3.0 Changes 13th January 2023

  • Annual review with no material changes
  • Editorial correction of spelling and phrasing errors

Version 3.0a Changes 6th November 2023

  • Annual review with no material changes
  • Editorial correction of spelling and phrasing errors

Version 3.0b Changes 16th February 2024

  • Annual review with no material changes
  • Editorial correction of spelling and phrasing errors

We have no links to any of the prior documents there, and as I'm sure any other CA will understand many things have changed across those years. While reviewing their CPS I did find a revocation page, and points at an older version of their policies, or rather it should instead they all point at the latest version.

I will stop here as actually looking further I am finding far too many issues to note on-the-fly in a quick comment. Suffice to say a quick glance under 'GLOBALTRUST 2020 SERVER OV 1' does not show a link to valid/old/revoked certificates. Even worse, looking at 'GLOBALTRUST 2015 SERVER OV 2' provides:

– valid certificate https://testok-2015-server-ov-2.e-monitoring.at/
– old certificate: https://testold-2015-server-ov-2.e-monitoring.at/
– revoked certificate: https://testrevoked-2015-server-ov-2.e-monitoring.at/

Rather than spend more time documenting where e-commerce monitoring GmbH have gone wrong, it may be more fruitful to document where they have went right.

I support removing trust in this CA and a thorough inspection of their operating practices. Pulling on a single thread like this is not a good sign for what an audit would do.

Flags: needinfo?(ca)

Glancing at the full Certificate Policy I have even more questions:
(Page 21)

Version 3.0 Changes 28th April 2022

  • qualified signatures and seals as Server-based signature service
  • introduction of term S/MIME certificate
  • Editorial correction of spelling and phrasing errors

Version 3.1 Changes 27th April 2023

  • Terminological specifications electronic seals (alignment with eIDAS, sole control vs control, ...)
  • Replacement of obsolete terms (trust services instead of certification, ...)
  • Editorial correction of spelling and phrasing errors

Version 3.2 Changes 18th August, 2023

  • Integration of browser requirements on S/MIME certificates
  • Precision on name forms in certificates
  • Consolidation of certificate profiles
  • Editorial correction of spelling and phrasing errors

Version 3.2a Changes 16th November, 2024

  • Update company information e-commerce monitoring gmbh
  • Editorial correction of spelling and phrasing errors

This might seem like a simple mistake, except it's also in the Austrian German on the right-hand side of the page too. There have been quite a few changes in the baseline requirements over this time period, here I am vaguely checking the baseline requirements against the certificate policy:

Compliance Section Summary
2022‐09‐01 7.1.4.2.2 CAs MUST NOT include the organizationalUnitName field in the Subject

7.1 Certificate profile (pg 164)
...
The subject can contain the following inputs: countryName (mandatory), localityName (mandatory), stateOrProvinceName (optional), organizationName (if the certificate is being issued for an organisation), organizationalUnitName (optional)

Compliance Section Summary
2023‐07‐15 4.9.1.1 and 7.2.2 New CRL entries MUST have a revocation reason code

Literally the only place that reasonCode is mentioned (pg 182):

7.2.2 CRL and CRL entry extension
Revocation lists can contain extensions that are specified in [RFC5280] or are compatible with [RFC5280].
Effective 30st September 2020, Certificate Revocation Lists for end user certificates may, Certificate Revocation Lists for CA certificates must contain the non critical extension reasonCode (OID 2.5.29.21) demonstrating the appropriate value according to [RFC5280], whereas the value unspecified is not not supported at all and the value certificateHold is not supported for server and S/MIME certificates.
Certificate Revocation Lists for qualified certificates contain the X509 extension “expiredCertsOnCRL” which correspondends to the OID string 2.5.29.60.

In the event of two languages providing different obligations in a Certificate Policy, which apply? The 'not not' line quoted above is not a typo on my behalf, and I'm not doing a thorough comparison here.

(In reply to Daniel Zens from comment #9)

However, please understand that it is not possible for us to respond to all inputs, in particular such that literally use the phrases "It is my personal opinion"

I used the phrase "it is my personal opinion" to indicate that the contents of that comment do not reflect the views of my employer.

The fact that the contents of a comment are one person's personal opinion do not make them unworthy of response. CAs are beholden to all users of the WebPKI, not just to root programs, and the comments of community members who post here need to be taken just as seriously as the comments of root program representatives. I will note that the Mozilla incident response policy does not care who posted a comment, and requires response to all of them equally:

Once the report is posted, you should respond promptly to questions that are asked, and in no circumstances should a question linger without a response for more than one week

But perhaps the fault is mine, for not phrasing my earlier comments as questions. So here are four specific questions that I would like you to address:

  1. Why did e-commerce monitoring GmbH believe that revocation was not necessary for this certificate?
  2. Why does the root cause analysis not include e-commerce monitoring GmbH's apparent lack of understanding of the Baseline Requirements as a contributing cause?
  3. What actions does e-commerce monitoring GmbH plan to take to prevent a future "delayed revocation" incident?
  4. Why does e-commerce monitoring GmbH believe it is acceptable to file an incident report with no action items?

Can someone else in the community attempt to interpret e-commerce monitoring GmbH's Certificate Policy via regards to key compromise and when they'll do it? I have my notes below and can't find a simple answer. I'm focusing on a single important question that should have a single quick answer, as any doubt has security implications in response times.

I want to assure the community in advance these are direct quotes. Page 41 (emphasis mine):

1.6 Definitions and acronyms / Definitionen und Kurzbezeichnungen
...
24/7/365, round-the-clock service, office hours
Services are provided all year round, seven days a week and 24 hours a day. Service disruptions or failures are documented. In isolated cases, additional limitations on availability can be defined, such as a tolerated disruption period of 1% per month or year.

Where not otherwise described in the respective GLOBALTRUST Certificate Practice Statement, published on the website of the operator or one of the websites referred to in the respective certificate policy, the following minimum office hours apply for every kind of query, application or order, including applications for suspension and revocation: working days, Monday to Friday 9:00 – 17:00.

Outside of these times an on-call service is available to address technical disruptions, defects or other emergencies that are critical to Trust Services.

So there's a physical office, but otherwise on-call handle other emergencies out-of-hours in a very vague way that may not include revocations. Let's delve a bit deeper. Page 72:

4.5.1 Subscriber private key and certificate usage / Nutzung des privaten Schlüssels und des Zertifikates durch den Signato
...
14. In the event that the CA key or subscriber key has been compromised, the subscriber should follow the instructions of the CA within 48 hours. This time span is shortened if specific security risks are expected. In this event, the subscriber will be informed of the shortened reaction time by telephone, email or other suitable means.

Okay, this section seems to be about subscriber reaction so 'within 48 hours' is just a very odd choice of phrase. This should be within 24 hours in 4.9 right? Page 84:

4.9.1 Circumstances for revocation
A certificate is revoked if the further use of the key in the sense of this policy is no longer ensured.
Reasons for revocation include:

I will do readers a service, but all 24 hour and 5 day revocation reasons are mixed together, the closest we get to a timeframe in 4.9.1 is Page 86:

A CA-certificate or enduser-sub-certificate will be revoked within 7 days (unless shorter time limit is mandatory in this policy specified) if one of the following criteria has been fulfilled:

A list of 11 items is then provided which includes compromise and misuse. Looking ahead to 4.9.5 (Page 89-90), we almost get an answer:

4.9.5 Time within which CA must process the revocation request / Reaktionszeit des VDAs auf einen Widerrufsantrag

Requests by telephone, fax, post and email will be received and processed within the office hours given in the GLOBALTRUST Certificate Practice Statement and will be subject to all necessary reviews after completion.

Revocation requests can be made via a web interface around the clock.

The maximum duration permitted between receiving the request and performing it depends on the current legal conditions, but is less than 24 hours.

For qualified certificates, the revocation is executed within three hours of the request on business days from 9am to 5pm, with the exception of Saturdays. Outside of these times, a suspension is performed within six hours.

Note that the suspension line was included for completeness, they do make sure to not apply those to TLS certs. Additionally a reference to the CPS is made, but nowhere are office hours listed in that document.

I am not quite sure what they mean about Saturdays. They seem to hold themselves to office hours, unless it's through the web interface, which seems intended for Subscribers specifically. There is an email address listed in the CP and CCADB, but given the above it's only checked within office hours.

While I trust we will get a response from e-commerce monitoring GmbH, I feel like it shouldn't require a direct answer.

To partially answer my own earlier question: where the policies are archived? I have a partial answer. The latest Certificate Practice Statement has properly versioned links on page 85 dating to the most recent version. However, for the Certificate Policy on page 209 they're all pointing at the same latest policy with bad copy/pasted version information. So we're back to square one.

I will leave any further inquiries and research until we have more information and updated policies.

This CP and CPS are likely not compliant.

From the CP

If an application for revocation is made by someone other than the
operator, a complete verification of identity and authorisation to request
revocation is compulsory.

No, you can't do a "complete verification of identity".

The CA retains the right to request further proof of authorisation and to
suspend a certificate rather than revoke it,

I don't think this is allowed either.

If the applicant cannot provide sufficient information to reliably confirm his
identity and authorisation to request a revocation between the time of the
request and the maximum permitted response time, the revocation will be
rejected and a suspension performed instead.
Again, no. You can't "confirm identity".

Yeah I was keeping my question very specifically focused otherwise all of these tangents would come up. There's also the small problem that e-commerce monitoring GmbH last audit was 2023-05-12 and for Root 3: GLOBALTRUST 2020 allegedly the Baseline Requirements v1.8.7 were considered.

During that Certificate Policy v3.1 was checked, but the only version of that I can find is non-authoritative. It is full of the same defects highlighted above, which calls their auditing practices into question. Everything mentioned above still applies to BR v1.8.7.

Alarmingly this is a CA trusted in the Apple, Chrome, Microsoft, and Mozilla Root Programs. A cursory glance at these documents or their audits would make these issues jump out a mile. I won't request information from them here as I'm sure they're already aware and working behind the scenes, but at least consider a review of these paper-thin audits for other CAs.

Any audit in the pipeline by e-commerce monitoring GmbH should never pass as it stands.

Comment #13 notes that the latest CPS has properly versioned links on page 85, but I had trouble opening the links without some trial and error by trying different naming patterns. Whereas on pages 208-209 of the Certificate Policy the links appeared to work, but as noted, the last few (CP from v.3.0 through 3.2a) just pointed to the current CP. However, I was able to locate and download version 3.1 of the CP by following the date pattern that I could see was used for other CPs - http://www.globaltrust.eu/static/globaltrust-certificate-policy.20230427.pdf. e-commerce monitoring should open another incident in Bugzilla for not "maintain[ing] links in their online repositories" to historic versions, as required by Item 7 in section 3.3 of the Mozilla Root Store Policy, which if had been done would likely have avoided this issue.

Somewhat related, here is e-commerce monitoring's last CP/CPS self-assessment - https://service.globaltrust.eu/static/globaltrust-ccadbselfassessment-20320628.xlsx. Self-assessments are required to be performed on an annual basis. Soon e-commerce monitoring will be expected to submit another self-assessment using the current self-assessment template, which is v. 1.3 and is located here: https://www.ccadb.org/cas/self-assessment. It covers v. 2.0.2 of the TLS Baseline Requirements (dated January 8, 2024).

Concerning the revocation procedure, at least for key compromise, Mozilla requires that section 4.9.12 of the CP or CPS "clearly specify the methods that parties may use to demonstrate private key compromise". While Section 4.9.12 of the e-commerce monitoring CP appears to comply, the methods for demonstrating key compromise do not appear to be as easy as what a few other CAs provide—although the CP says that the methods “may include but are not limited to”. e-commerce monitoring should amend various parts of its section 4.9 accordingly.

Flags: needinfo?(bwilson)

Ben: Why are we still discussing this and not prepare for distrust?

It is clear that this CA should not be trusted. Many community members (Amir, Wayne and more) are showing that e-commerce monitoring are absolutely not capable of being trusted CA. These volunteers are doing the work you and Mozilla should be doing.

They do not have a big number of certificates. Distrust will impact no-one and make the webPKI safer.

When and where and how can we start the process to remove them as a trusted CA, or do we wait for proper trust-programms like Google or Apple or Microsoft to take action?

We are a week on since e-commerce monitoring GmbH made their comment saying things would change. Nothing has changed.

I sincerely hope we are not just hoping that e-commerce monitoring GmbH submits a self-assessment alongside their regular audit and we all act like everything is fine. They have been utterly failing for over a year and are being treated as if they're new and just need a little training. They've been dragged through this entire process with kid's gloves - if their finance department were getting issues raised like this do you expect a similar level of response? The internal response reflects the level of enforcement expected by Root Programs and how little resources the company needs to put to meet the bare minimum.

I do not say this lightly but this is a CA where active measures must be taken by all Root Programs. CAs all have to be held to the same high standards, and every day is lowering the bar of what is considered acceptable practice. The same could also be said for other CAs claiming exception to the rules, but this is an extreme case ongoing for over a year. If self-regulation is the plan then enforcement needs teeth and a willingness to take on even the most simplest of cases like this one.

I appreciate the Mozilla Root Program have opted to give a response here, but where are the Chrome Root Program? Apple Root Program? I've never even seen the Microsoft Root Program do anything other than look at the code-signing ballots and give an automatic Yes to them. If those Root Programs wish to not have different governments take over their roles then they need to allocate resources and start enforcing their own policies. Issuing CAs should understand why this is a concern to their business model going forward.

In response to Comment #17 and Comment #18, I am working on a summary of issues for posting to mdsp. Meanwhile, issues that don't fit into the category for this bug (i.e. delayed revocation) should be posted in the appropriate bug, have their own bug created, or otherwise be discussed in mdsp in the thread "Public Discussion of Acquisition of e-commerce monitoring GmbH by AUSTRIA CARD-Plastikkarten und Ausweissysteme GmbH". Thanks! Ben

(In reply to Aaron Gable from comment #12)

But perhaps the fault is mine, for not phrasing my earlier comments as questions. So here are four specific questions that I would like you to address:

  1. Why did e-commerce monitoring GmbH believe that revocation was not necessary for this certificate?

We have already outlined the factors we considered in making the decision earlier. We are unable to assess any further thought processes from former officials who have since departed.

  1. Why does the root cause analysis not include e-commerce monitoring GmbH's apparent lack of understanding of the Baseline Requirements as a contributing cause?

In this bug report, both the critics that the requirement was not known and that it was intentionally disregarded have already been made; but the latter presupposes knowledge of the requirement. What does the general consensus lean towards?

Our position is as stated at the beginning of the root cause section. The requirements were known but not adhered to due to the reasons described. This differs from not being aware of or misunderstanding the requirements.

  1. What actions does e-commerce monitoring GmbH plan to take to prevent a future "delayed revocation" incident?

We will continue to mitigate misissuance through various measures, thereby reducing the likelihood of revocations in general, and consequently, also decreasing the occurrence of delayed revocations. Measures we already have in place, including prior to this bug, comprise:

  • Email and SMS notifications to the on-call service personnel outside office hours, as long as revocation requests remain unresolved.
  • Work instructions
  • Regular training sessions
  • 4 eyes decisions

It is evident that these measures address the risk of forgetting revocations.

  1. Why does e-commerce monitoring GmbH believe it is acceptable to file an incident report with no action items?

If we consider there are no pending action items, it is consistent that we cannot list any open points.

(In reply to Ben Wilson from comment #16)

Comment #13 notes that the latest CPS has properly versioned links on page 85, but I had trouble opening the links without some trial and error by trying different naming patterns. Whereas on pages 208-209 of the Certificate Policy the links appeared to work, but as noted, the last few (CP from v.3.0 through 3.2a) just pointed to the current CP. However, I was able to locate and download version 3.1 of the CP by following the date pattern that I could see was used for other CPs - http://www.globaltrust.eu/static/globaltrust-certificate-policy.20230427.pdf. e-commerce monitoring should open another incident in Bugzilla for not "maintain[ing] links in their online repositories" to historic versions, as required by Item 7 in section 3.3 of the Mozilla Root Store Policy, which if had been done would likely have avoided this issue.

Thank you Ben.
Regarding the historical policy documents, we have identified a typo in our current list of historic policy URIs on page 209 of https://www.globaltrust.eu/static/globaltrust-certificate-policy.pdf, which will be addressed in accordance your guidance. In the interim, below are the documents for evaluating the policies and practices at that time. Note that we have not arbitrarily selected but have listed all versions included in major browsers since 2020.

GLOBALTRUST Certificacte Policy

GLOBALTRUST Certificate Practice Statement

Somewhat related, here is e-commerce monitoring's last CP/CPS self-assessment - https://service.globaltrust.eu/static/globaltrust-ccadbselfassessment-20320628.xlsx. Self-assessments are required to be performed on an annual basis. Soon e-commerce monitoring will be expected to submit another self-assessment using the current self-assessment template, which is v. 1.3 and is located here: https://www.ccadb.org/cas/self-assessment. It covers v. 2.0.2 of the TLS Baseline Requirements (dated January 8, 2024).

This is on track as part of our routine compliance processes.

Concerning the revocation procedure, at least for key compromise, Mozilla requires that section 4.9.12 of the CP or CPS "clearly specify the methods that parties may use to demonstrate private key compromise". While Section 4.9.12 of the e-commerce monitoring CP appears to comply, the methods for demonstrating key compromise do not appear to be as easy as what a few other CAs provide—although the CP says that the methods “may include but are not limited to”. e-commerce monitoring should amend various parts of its section 4.9 accordingly.

The current CP in 4.9.12 Special requirements related to key compromise states:
[...]may include but are not limited to:

  • Presentation of the private key or the QSCD containing the private
    key by a third party who is not the legitimate owner of the key,
  • Documentation of a practical procedure for generating the private
    key identified as being compromised,
  • Studies, analyses or comparable qualified research documentation
    from an accredited technical institution (e.g. NIST, BSI (GB), BSI
    (DE), ...) or a research institution describing the compromisability of
    the key generation method specifically used by the TSP.[...]

After a brief review of the CP/CPS of other CAs, it appears that bullet 1 method is sometimes the only one offered.
We understand Points 2 and 3 are, of course, not something that the anyone can practically apply in everyday life.
We understand and are going to take into account any proposal for providing easier ways for our subscribers and the public.

(In reply to Ben Wilson from comment #19)

In response to Comment #17 and Comment #18, I am working on a summary of issues for posting to mdsp. Meanwhile, issues that don't fit into the category for this bug (i.e. delayed revocation) should be posted in the appropriate bug, have their own bug created, or otherwise be discussed in mdsp in the thread "Public Discussion of Acquisition of e-commerce monitoring GmbH by AUSTRIA CARD-Plastikkarten und Ausweissysteme GmbH". Thanks! Ben

This bug has now largely evolved into a collection of general expressions of discontent; Thank you, Ben, for channeling the communication.

Flags: needinfo?(ca)

(In reply to Daniel Zens from comment #21)

  1. Why did e-commerce monitoring GmbH believe that revocation was not necessary for this certificate?

We have already outlined the factors we considered in making the decision earlier. We are unable to assess any further thought processes from former officials who have since departed.

I'm sorry, we appear to be misunderstanding each other. I'm not asking why e-commerce monitoring GmbH decided to delay revocation after they finally determined that it was necessary on 2023-03-22. I'm asking why e-commerce monitoring GmbH was unable to come to the conclusion that revocation was necessary on its own, all the way back on 2023-02-08 when the misissuance was first reported. None of the details provided here address why e-commerce monitoring GmbH misunderstood the requirements in this way.

  1. Why does the root cause analysis not include e-commerce monitoring GmbH's apparent lack of understanding of the Baseline Requirements as a contributing cause?

In this bug report, both the critics that the requirement was not known and that it was intentionally disregarded have already been made; but the latter presupposes knowledge of the requirement. What does the general consensus lean towards?

In order to function, CAs need to be aware of the requirements. Lack of awareness of the requirements is a root cause of this delayed revocation, and fixing this lack of awareness should be one of the action items.

Our position is as stated at the beginning of the root cause section. The requirements were known but not adhered to due to the reasons described. This differs from not being aware of or misunderstanding the requirements.

The requirement that revocation happen within 5 days was known, yes, but that's not the requirement I'm talking about. The requirement that a certificate not contain both the CT Poison extension and SCTs is the requirement that I'm talking about. The fact that e-commerce monitoring GmbH did not know or did not understand this requirement is what led to revocation being delayed by 51 days.

  1. What actions does e-commerce monitoring GmbH plan to take to prevent a future "delayed revocation" incident?

We will continue to mitigate misissuance through various measures, thereby reducing the likelihood of revocations in general, and consequently, also decreasing the occurrence of delayed revocations.

Preventing misissuance does not prevent CA staff from misunderstanding requirements and making incorrect decisions about whether revocation is necessary. Steps to prevent misissuance are appropriate action items for Bug 1815534. They are not appropriate action items for this incident.

  1. Why does e-commerce monitoring GmbH believe it is acceptable to file an incident report with no action items?

If we consider there are no pending action items, it is consistent that we cannot list any open points.

The CCADB Incident Report policy says that "Each item [in the "What didn't go well" section] must have at least one corresponding Action Item". It also says that the Action Items "must create additional protections to prevent future incidents". I do not see any action items detailing how e-commerce monitoring GmbH is "assign[ing] a higher priority to documentation and external communication", as per your "What didn't go well" section. I also do not see anything about what actions e-commerce monitoring GmbH has taken, is taking, or will take to ensure that CA staff understand the requirements and do not misjudge whether a certificate needs to be revoked.

Attachment #9402467 - Attachment description: (In reply to Aaron Gable from comment #12) > But perhaps the fault is mine, for not phrasing my earlier comments as questions. So here are four specific questions that I would like you to address: > 1. Why did e-commerce monitoring GmbH believe that r → Reply to Aaron Gable from comment #12

As you might know, browsers have decided to remove e-commerce monitoring GmbH (ECM) with its Root Certificate "GLOBALTRUST 2020" from their Root Programs as of June 30, 2024. Certificates issued before this date will retain their full validity.

The reasons for the removal have been comprehensively discussed Bugzilla forum. We acknowledged and accepted the decision. We have identified the shortcomings in our processes, particularly related to reaction time. Consequently, we are taking these issues very seriously and are committed to address them. An action plan is being rolled out to restructure our Certificate Authority (CA) functions. Our goal is to be included again in the Root Programs.

ECM’s shareholder, AUSTRIA CARD, is committed to regains full compliance with the Browser/OS Root Store Policies. This commitment, which is strongly supported by our recently changed management, underscores our dedication to maintaining the widest compatibility and coverage.
As an immediate action, and until full remediation, ECM has ceased the issuance of TLS certificates according to the CA/Browser Forum Requirements. TLS certificates will be provided solely based on Regulation (EU) No 910/2014, Annex IV, as recently amended by Regulation (EU) 2024/1183 (“QWACs”). Certificates for interoperability testing purposes are excluded from this decision.

ECM, with its product lines GLOBALTRUST and TRUST2GO, is a Qualified Trust Service Provider (QTSP) according to EU eIDAS regulation and is under continuous supervision by the Austrian regulatory authority (RTR/TKK). Our activities are regularly evaluated by an accredited conformity assessment body based on numerous standards (e.g., eIDAS, ETSI), which include comprehensive logical, physical, and organizational security measures.

Our goal is to rebuild trust and demonstrate our commitment to upholding the highest standards in our industry.
For inquiries, please contact the Compliance & Product Management team, Attn: Mr. Daniel Zens, at ca@globaltrust.eu

Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: