Open Bug 1883711 Opened 4 months ago Updated 17 days ago

e-commerce monitoring gmbh: precertificate validity does not match leaf certificate

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
ASSIGNED

People

(Reporter: ca, Assigned: ca)

Details

(Whiteboard: [ca-compliance] [ov-misissuance])

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:108.0) Gecko/20100101 Firefox/108.0

Steps to reproduce:

on 2024-03-04 16:34 UTC e-commerce monitoring has received a Certificate Problem Report pointing out that the precertificate and the leaf certificate of serial number 32daef359edc5cdaa0b9fa have been issued with different validity periods. We have investigated the issue and are in the process of documenting a full incident report, which we will provide as soon as possible.

Leaf: https://crt.sh/?sha256=6181C4F11C4F7E1FF22ECBD3F9A001A85162527D4E934EC75A3CB419F50E46F7
Precertificate: https://crt.sh/?sha256=650A940633DC9B3A8CCB201198238EE093C3EC7F53898873AFAB51AC859060AC

Assignee: nobody → ca
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Whiteboard: [ca-compliance] [ov-misissuance]

The certificate has been revoked with CRLReason 5 (cessationOfOperation), which is not the appropriate revocation reason code as per Section 4.9.1.1 (12) of the BRs: "The CA is made aware that the Certificate was not issued in accordance with these Requirements... (CRLReason #4, superseded)".

Incident Report

Summary

On Jan 30 2024, e-commerce monitoring gmbh issued a Pre-Certificate signed by GLOBALTRUST 2020 SERVER OV 1. This precertificate has a Not Before date set to Jan 30 00:00:00 2024 GMT and a Not After set to Feb 2 02:00:00 2025 GMT. However the corresponding leaf certificate issued the next day has a validity period of Not Before: Jan 31 00:00:00 2024 GMT and Not After: Feb 3 02:00:00 2025 GMT, hence does not match the Pre-Certificate in violation of rfc6962.

Impact

One TLS certificate under the CA/Browser Forum Organization Validated Policy was issued in non-conformance to the rfc6962, and revoked before delivery to the Subscriber. E-commerce monitoring gmbh has decided not to stop the issuance of further TLS certificates until resolving the bug, hence was once knowing about it, we have found an easy and secure way to avoid further similar incidents.

Timeline

All times are UTC.

2024-01-30

2024-01-31

2024-03-04:

  • 16:34 Receipt of a Certificate Problem Report
  • 16:47 Forward to other responsible staff, confirmation of receipt to reporter

2024-03-05:

  • 09:00 Staff meeting discussing including discussion of the Certificate Problem Report, manual certificate analysis confirmation that this is a misissuance
  • 10:00-16:00 Investigation of the root cause and information gathering on how to resolve the issue together with DevOps
  • 17:00 Filing of this bug (https://bugzilla.mozilla.org/show_bug.cgi?id=1883711)

Root Cause Analysis

Upon investigation of our data acquisition in our issuance system, we have discovered a bug which causes current date and time to be used for the Leaf Certificate, instead of retrieving the date and time directly from the Pre-Certificate. Obviously, this means that if a Pre-Certificate and its corresponding Leaf Certificate are not issued on the same day (as happened in this case) the times do not match. Our post-issuance linting did not catch this, however one Registration/Revocation Officer noted the problem and revoked the certificate within approximately 30 Minutes after the Leaf was issued, and before the certificate was delivered to the Subscriber.

Lessons Learned

What went well

  • Early immediate action (revocation) concerning the specific misissued certificate
  • Fast and competent problem analysis and clear approach to get rid of the bug.

What didn't go well

  • Registration/Revocation Officer though behaving correctly in part, missed to inform the in house compliance team, so no holistic incident handling could be processed. Actually we were notified by an external Problem Report.
  • Registration/Revocation Officer selected an inappropriate value for the revocationReason CRL extension.

Where we got lucky

  • Only one certificate was affected
  • We became aware of the bug before the problem could spread

Action Items

Action Item Kind Due Date
Revocation Resolve 2024-01-31
----------- ---- --------
Retraining Prevent 2024-03-12
----------- ---- --------
Code Analysis Prevent, Improve 2024-03-12
----------- ---- --------
Bug Fix Prevent 2024-04-15
----------- ---- --------
4-eyes Review by Security staff Prevent 2024-04-17
----------- ---- --------
Internal Test system Prevent 2024-04-24
----------- ---- --------
Approval by Compliance staff Prevent 2024-04-24
----------- ---- --------
Deployment productive Systems by DevOps Prevent 2024-04-29

Appendix

Details of affected certificates

Leaf Certificate: https://crt.sh/?sha256=6181C4F11C4F7E1FF22ECBD3F9A001A85162527D4E934EC75A3CB419F50E46F7
Pre-Certificate: https://crt.sh/?sha256=650A940633DC9B3A8CCB201198238EE093C3EC7F53898873AFAB51AC859060AC

How did you fix this bug?

What are you doing to prevent similar incidents in the future?

Were you aware of Bug 1838667

Have you examined other certificates that you issued to see if they have similar issues?

Flags: needinfo?(ca)

It has been 9 days since I posed the questions in Comment 3. Leaving questions unanswered for more than a week is a violation of https://wiki.mozilla.org/CA/Responding_To_An_Incident

e-commerce monitoring was aware of this problem in January but did not file this bug until after they received a certificate problem report in March.

Furthermore, on 2024-03-08, they signed another mismatched certificate/precertificate pair: https://crt.sh/?q=eecac177af1c9ebe630134a9fe78fbe787c7361db91135eafb4b2e317d9e2f02 and https://crt.sh/?q=41E9D1810A191C9EC5D4AB90FB9EBBA553AC0AD232D5A8FD6AE781202C048866

This certificate was revoked on 2024-03-10 but it is not mentioned in the incident report posted on 2024-03-19.

Clearly, e-commerce monitoring has not done a proper investigation into this incident nor fixed the root cause.

Considering also Bug 1815534 and Bug 1862004, especially Bug 1862004 Comment 1, it is evident that e-commerce monitoring has appalling incident response procedures.

(In response to https://bugzilla.mozilla.org/show_bug.cgi?id=1883711#c3 and https://bugzilla.mozilla.org/show_bug.cgi?id=1883711#c4)

Thank you Andrew for following up on this.

How did you fix this bug? What are you doing to prevent similar incidents in the future?

The due date according to our Action Items list is 2024-04-15, we will provide a response on time.

Were you aware of Bug 1838667

Yes

Have you examined other certificates that you issued to see if they have similar issues?

Yes

e-commerce monitoring was aware of this problem in January but did not file this bug until after they received a certificate problem report in March.

This has been adressed in our incident report. Please see the "What didn't go well" section in https://bugzilla.mozilla.org/show_bug.cgi?id=1883711#c2.

Furthermore, on 2024-03-08, they signed another mismatched certificate/precertificate pair: https://crt.sh/?q=eecac177af1c9ebe630134a9fe78fbe787c7361db91135eafb4b2e317d9e2f02 and https://crt.sh/?q=41E9D1810A191C9EC5D4AB90FB9EBBA553AC0AD232D5A8FD6AE781202C048866
This certificate was revoked on 2024-03-10 but it is not mentioned in the incident report posted on 2024-03-19.

Thank you for bringing this to our attention. We have begun investigating this and will decide the next steps on the short term.

Flags: needinfo?(ca)

Quick status update: In the meantime, we have added more DevOps resources for a sustainable solution that addresses the root cause. Our next action Item, dated 2024-04-15, is on track.

Thank you for the incident report and the updates so far.

Can you please update the bug with more details on the actions until now based on your action plan?

  • Retraining: What parts of your training did you change as part of this retraining process? Did you add specific training material and in which areas? This would be useful information for other CAs to know so they can possibly adjust their training program as well.
  • Code Analysis: What parts of the code did you check and what were the results?

Based on the action plan and your previous answer, is it fair to assume that the bug has not been fixed and will be fixed by 2024-04-15?

I was not able to detect whether you have stopped issuance of new certificates until you fix the issue. Can you please confirm whether you have stopped issuance (and when), or not?

Dear all,

as this week is e-commerce monitoring gmbh's audit week, our compliance resources which include incident reporting, were mainly focused on that. Our DevOps have been working s with high priority on various improvements that address current Mozilla bugs.

For upcoming week, we have scheduled a test session and remain confident to rollout into the production environment soon. We will provide an update before Friday 26th EOB.

Best regards,
Daniel

(In reply to Daniel Zens from comment #8)

Dear all,

as this week is e-commerce monitoring gmbh's audit week, our compliance resources which include incident reporting, were mainly focused on that. Our DevOps have been working s with high priority on various improvements that address current Mozilla bugs.

For upcoming week, we have scheduled a test session and remain confident to rollout into the production environment soon. We will provide an update before Friday 26th EOB.

Best regards,
Daniel

Dear Daniel,

My last two questions (submitted 10 days ago in https://bugzilla.mozilla.org/show_bug.cgi?id=1883711#c7) should have very simple answers (yes/no) that should be affected by your audit week. In any case, you could have responded last week which was before your audit week.

Regardless of that, a public CA should have adequate resources to handle multiple tasks and incidents in parallel with other activities, including audit tasks. What if you had a critical security incident during your audit week?

Best regards,
Dimitris.

It's very unfortunate that e-commerce has not been providing easy clarification responses to easy questions which are important for the community to understand if there is an unmitigated risk.

Dear Dimitris,

thank you for providing your input to this bug.
Some of your questions are not immediately clear to us.

(In reply to Dimitris Zacharopoulos from comment #7)

Can you please update the bug with more details on the actions until now based on your action plan?

  • Retraining: What parts of your training did you change as part of this retraining process? Did you add specific training material and in which areas? This would be useful information for other CAs to know so they can possibly adjust their training program as well.

Yes, training material about Certificate Transparency tailored to our needs (e.g. we only support embedded SCT currently.)

  • Code Analysis: What parts of the code did you check and what were the results?

This is described in the Root Cause section of the report.

Based on the action plan and your previous answer, is it fair to assume that the bug has not been fixed and will be fixed by 2024-04-15?

By the Action Item "Bug Fix" we mean a correction at the code level, which is followed by a few other steps before being deployed and used. Please refer to the Action Items section, this should be clear. In the future we will make this more transparent by regularly updating, aligned with our Action Items List.

I was not able to detect whether you have stopped issuance of new certificates until you fix the issue. Can you please confirm whether you have stopped issuance (and when), or not?

Please have a second look at the full report that states "E-commerce monitoring gmbh has decided not to stop the issuance of further TLS certificates until resolving the bug" https://bugzilla.mozilla.org/show_bug.cgi?id=1883711#c2

(In reply to Dimitris Zacharopoulos from comment #9)

My last two questions (submitted 10 days ago in https://bugzilla.mozilla.org/show_bug.cgi?id=1883711#c7) should have very simple answers (yes/no) that should be affected by your audit week. In any case, you could have responded last week which was before your audit week.

In the future we will make this more transparent by regularly updating.

Regardless of that, a public CA should have adequate resources to handle multiple tasks and incidents in parallel with other activities, including audit tasks. What if you had a critical security incident during your audit week?

We fully agree, however at our organization there is a strict segregation of Compliance and Security.

(In reply to Dimitris Zacharopoulos from comment #10)

It's very unfortunate that e-commerce has not been providing easy clarification responses to easy questions which are important for the community to understand if there is an unmitigated risk.

This matter could be addressed in https://bugzilla.mozilla.org/show_bug.cgi?id=1893546

Thank you for the responses. I read the report again and couldn't locate the part that explains the "easy and secure way to avoid further similar incidents". Can you please help me to identify which part of your incident response describes the way to avoid further similar incidents?

Regarding the "Code Analysis", I took this from your Action Items which had a due date "2024-03-12". So, when I posted my first comment in 2024-04-09, this part was supposed to be completed. I was hoping to get some more details on this analysis, findings and fixes. Your response was "This is described in the Root Cause section of the report." and I assume you are referring to this part:

  • "Upon investigation of our data acquisition in our issuance system, we have discovered a bug which causes current date and time to be used for the Leaf Certificate, instead of retrieving the date and time directly from the Pre-Certificate. Obviously, this means that if a Pre-Certificate and its corresponding Leaf Certificate are not issued on the same day (as happened in this case) the times do not match"

Is this is the extent of your code analysis for this issue?

How would your CA system react to a precertificate issued at 23:59:00 and the certificate issued 2 minutes later? Would the same problem occur?

(In reply to Dimitris Zacharopoulos from comment #12)

Dear Dimitris,

thank you for responding.

Is this is the extent of your code analysis for this issue?

What level of detail or insight into internal operations a) is really beneficial to others than ourselves b) is not confidential or commercially sensitive c) can be reasonably expected at this forum?

How would your CA system react to a precertificate issued at 23:59:00 and the certificate issued 2 minutes later? Would the same problem occur?

Obviously yes, though highly unlikely. We (still) issue this type of certificates by human input only, and our respective staff do not work around midnight.

Best regards,
Daniel

(In reply to Daniel Zens from comment #13)

(In reply to Dimitris Zacharopoulos from comment #12)

Dear Dimitris,

thank you for responding.

Is this is the extent of your code analysis for this issue?

What level of detail or insight into internal operations a) is really beneficial to others than ourselves b) is not confidential or commercially sensitive c) can be reasonably expected at this forum?

Hi Daniel,

Since you are responding to a question with a question, I must reply with "whatever you deem appropriate to assure this community that you have performed basic analysis to detect this specific and potentially other issues of a similar nature" :)

I'm sure you can understand that by stating that you performed "Code Analysis" in your action items, is overly broad and not very helpful.

How would your CA system react to a precertificate issued at 23:59:00 and the certificate issued 2 minutes later? Would the same problem occur?

Obviously yes, though highly unlikely. We (still) issue this type of certificates by human input only, and our respective staff do not work around midnight.

I'm sorry, I understood that this affects all SSL/TLS Certificates issued. Based on your https://bugzilla.mozilla.org/show_bug.cgi?id=1883711#c2 it doesn't specify that all SSL/TLS certificates are issued by human input only. Does that mean that you do not currently support any method that does not involve some sort of manual validation by your staff?

As you might know, browsers have decided to remove e-commerce monitoring GmbH (ECM) with its Root Certificate "GLOBALTRUST 2020" from their Root Programs as of June 30, 2024. Certificates issued before this date will retain their full validity.

The reasons for the removal have been comprehensively discussed Bugzilla forum. We acknowledged and accepted the decision. We have identified the shortcomings in our processes, particularly related to reaction time. Consequently, we are taking these issues very seriously and are committed to address them. An action plan is being rolled out to restructure our Certificate Authority (CA) functions. Our goal is to be included again in the Root Programs.

ECM’s shareholder, AUSTRIA CARD, is committed to regains full compliance with the Browser/OS Root Store Policies. This commitment, which is strongly supported by our recently changed management, underscores our dedication to maintaining the widest compatibility and coverage.
As an immediate action, and until full remediation, ECM has ceased the issuance of TLS certificates according to the CA/Browser Forum Requirements. TLS certificates will be provided solely based on Regulation (EU) No 910/2014, Annex IV, as recently amended by Regulation (EU) 2024/1183 (“QWACs”). Certificates for interoperability testing purposes are excluded from this decision.

ECM, with its product lines GLOBALTRUST and TRUST2GO, is a Qualified Trust Service Provider (QTSP) according to EU eIDAS regulation and is under continuous supervision by the Austrian regulatory authority (RTR/TKK). Our activities are regularly evaluated by an accredited conformity assessment body based on numerous standards (e.g., eIDAS, ETSI), which include comprehensive logical, physical, and organizational security measures.

Our goal is to rebuild trust and demonstrate our commitment to upholding the highest standards in our industry.

For inquiries, please contact the Compliance & Product Management team, Attn: Mr. Daniel Zens, at ca@globaltrust.eu

You need to log in before you can comment on or make changes to this bug.