1963629 - HARICA: One of the two Certificate Problem Report email aliases not working

Status: ASSIGNED

(CA Program :: CA Certificate Compliance, task)

Description

[Dimitris Zacharopoulos]

Preliminary Incident Report

Summary

• Incident description: HARICA accepts Certificate Problem Reports via 2 email addresses as described in section 1.5.2 of the HARICA CP/CPS. CCADB records showed one of the two email addresses. Due to an internal procedure failure, the email alias disclosed in CCADB “expired”, and emails were not being delivered. The problem was resolved a few minutes after it was reported via the second email address mentioned in our CP/CPS. An internal investigation has started to identify a) why didn’t the CCADB record include both email addresses, and b) why the email alias wasn’t renewed on time.
• Relevant policies: Section 1.5.2 of the HARICA CP/CPS (possibly, check below)
• Source of incident disclosure: Externally reported

We decided to disclose this issue for increased transparency. However, we are not 100% certain if this is a violation which is why we seek some guidance.

Unless we missed it, the CCADB and Mozilla Root Store policies do not require CAs to update CCADB with their certificate problem report reporting information. Relying Parties and Subscribers usually find this information within section 1.5.2 of the CA's CP/CPS.

The TLS BRs have requirements in Section 4.9.3 that the CA must have procedures to support Certificate Problem Reports. HARICA satisfies this requirement by pointing to the two email addresses mentioned in section 1.5.2 of our CP/CPS. One of those email addresses had an issue and could not receive emails properly.

The reporter ultimately submitted the certificate problem report using the second email address.

HARICA will treat this as an incident internally using its documented procedures, independently of this bug. However, we were curious if the community considers this a violation of CCADB or Root Store Program Policies that requires following the CCADB incident reporting guidelines.

Comment 1

[Jeremy]

Yes - this is an incident. This is in the same category as https://bugzilla.mozilla.org/show_bug.cgi?id=1955365 where the CA committed to more than the minimum in their CPS. Because the CA did not follow their CPS, the discrepancy should be filed as a bug. However, I believe these bugs deserve a slightly different treatment than violations of the BRs. I think we should encourage more detail in CPS docs, and we want CAs to be stricter than the BRs require. Therefore, we should encourage disclosure of these incidents while encouraging these more stringent self-imposed requirements.

The incident (IMO) is associated with the 4.9.3:
"The CA SHALL provide Subscribers, Relying Parties, Application Software Suppliers, and other third parties with clear instructions for reporting suspected Private Key Compromise, Certificate misuse, or other types of fraud, compromise, misuse, inappropriate conduct, or any other matter
related to Certificates. The CA SHALL publicly disclose the instructions through a readily accessible online means and in Section 1.5.2 of their CPS."
Despite two methods being listed, having one of the emails not working means the instructions were not clear to the third party reporter.

You also have an issue under Mozilla 5.2:
"CA operations MUST at all times be in accordance with the applicable CP and CPS (or combined CP/CPS)."

Comment 2

[Dimitris Zacharopoulos]

Thanks Jeremy for the quick response.

(In reply to Jeremy from comment #1)

Yes - this is an incident. This is in the same category as https://bugzilla.mozilla.org/show_bug.cgi?id=1955365 where the CA committed to more than the minimum in their CPS. Because the CA did not follow their CPS, the discrepancy should be filed as a bug. However, I believe these bugs deserve a slightly different treatment than violations of the BRs. I think we should encourage more detail in CPS docs, and we want CAs to be stricter than the BRs require. Therefore, we should encourage disclosure of these incidents while encouraging these more stringent self-imposed requirements.

I don't believe this is the same issue. Apple had a stricter requirement that didn't follow. HARICA didn't have a stricter requirement. It listed two email addresses that could be used to contact the CA in case of an issue, one that worked and one that didn't work (for a limited time).

It sounds overly strict to consider it a violation when there is a single control failure in a process that has redundant controls. Does that make sense?

The incident (IMO) is associated with the 4.9.3:
"The CA SHALL provide Subscribers, Relying Parties, Application Software Suppliers, and other third parties with clear instructions for reporting suspected Private Key Compromise, Certificate misuse, or other types of fraud, compromise, misuse, inappropriate conduct, or any other matter
related to Certificates. The CA SHALL publicly disclose the instructions through a readily accessible online means and in Section 1.5.2 of their CPS."
Despite two methods being listed, having one of the emails not working means the instructions were not clear to the third party reporter.

Do you believe HARICA didn't have "_clear _instructions" in the CP/CPS? It reads as follows:

Contact HARICA for Certificate Problem Reports by sending an email to “cert-problem-report at harica.gr”.
HARICA also maintains a continuous 24x7 ability to respond internally to a high-priority Certificate Problem Report to the email address “high-> priority-cert-problem-report at harica.gr”, and where appropriate, forward such a complaint to competent public authorities, and/or revoke a Certificate that is the subject of such a complaint.

The instructions were clear enough and effective enough that the reporter was able to submit the CPR using the second email address. Therefore, I don't think there is a question about whether the instructions were "clear". Obviously they could be more clear, saying something like "if the first email address doesn't work for any reason, please use the second one". Would you consider it a violation then?

Just to be clear, I'm trying to get a sense of your expectations when it comes to clarity in CP/CPS -and by extension the BRs- language for Relying Parties to understand. We've already established that Relying Parties reading an entire CP/CPS document must be at a certain level (above rudimentary?) to fully understand it.

You also have an issue under Mozilla 5.2:
"CA operations MUST at all times be in accordance with the applicable CP and CPS (or combined CP/CPS)."

This is known and it's the main clause in MRSP enabling all the CP/CPS stricter requirements to be part of all Program incident handling procedures.

I suppose there is agreement that CCADB or MRSP doesn't have a requirement associated with the disclosure of a Certificate Problem Report reporting mechanism?

Comment 3

[Wayne]

As the reporter in question I do want to clarify a couple of points:

• CCADB contact details for certificate reporting problems have been mentioned as a source of truth for a while now, as well as ensuring that information on CCADB is correct
• When the email address listed on CCADB failed I actually used the webform rather than looking at the CPS. A rare time I didn't check the CPS, but I figured I'd give a CA's webform a try.
• The webform's timeout for recaptcha is a bit strict as you complete the captcha prior to filling out the form. If you're not quite quick enough it'll silently fail as the failure is not presented to the client. You have to refresh to get a fresh captcha - and have the rest of the form ready to copy/paste.

I am a bit puzzled at HARICA being adamant that a second email address was found and used. Is it not clear on their side that the webform was used?

Looking at the CPS though:

Contact HARICA for Certificate Problem Reports by sending an email to “cert-problem-report at harica.gr”.

HARICA also maintains a continuous 24x7 ability to respond internally to a high-priority Certificate Problem Report to the email address “high-priority-cert-problem-report at harica.gr”, and where appropriate, forward such a complaint to competent public authorities, and/or revoke a Certificate that is the subject of such a complaint.
Also see sections 4.9.3.2 and 4.9.3.3.

The 1st email address is what was listed on CCADB (now updated to show both). The [url=https://harica.gr/en/Contact/CPR]webform[/url] lists CPR and high-priority CPR contact forms. I do want to note that there is no confirmation of any ticket raised sent via email, or even presented on the website, when sending in this form. Presumably if I used the regular CPR webform at that time it would also have silently bounced and it'd be even less obvious something went wrong. The confirmation I received was HARICA support contacting a couple of hours later.

Comment 4

[Jeremy]

I don't believe this is the same issue. Apple had a stricter requirement that didn't follow. HARICA didn't have a stricter requirement. It listed two email addresses that could be used to contact the CA in case of an issue, one that worked and one that didn't work (for a limited time).

I still see this as the same category. Harica and Apple were both giving the community something more than the Baseline required. However, both had an issue in delivering that expectation. The incident report 1) details how the CPS was not implemented, 2) provides the community (including other CAs) an opportunity to see how other CAs are implementing improvements, and 3) allows the community to make the severity/security judgment of the issue. In this case, how many CPRs might have been missed with one of the methods not working?

It sounds overly strict to consider it a violation when there is a single control failure in a process that has redundant controls. Does that make sense?

I think treating it like a normal incident or failure under the BRs is overly strict for sure. However, I think having an incident report and bug is good as it clarifies how the CPS diverged from reality. I'm also of the opinion that filing a bug isn't bad by default (or shouldn't be). These posts provide transparency and opportunity for improvement. Would anyone claim that the Google bug on MPIC wasn't an awesome learning opportunity for the entire ecosystem?! (https://bugzilla.mozilla.org/show_bug.cgi?id=1959867)

Do you believe HARICA didn't have "_clear _instructions" in the CP/CPS? It reads as follows:

I think Wayne addressed this one.

The instructions were clear enough and effective enough that the reporter was able to submit the CPR using the second email address. Therefore, I don't think there is a question about whether the instructions were "clear". Obviously they could be more clear, saying something like "if the first email address doesn't work for any reason, please use the second one". Would you consider it a violation then?

But how many people may not have realized their first submission didn't go through? But, yes - I think that language would help prevent an incident, but then I would question as why you don't just have one working email instead of one potentially broken email and one non-broken email.

Just to be clear, I'm trying to get a sense of your expectations when it comes to clarity in CP/CPS -and by extension the BRs- language for Relying Parties to understand. We've already established that Relying Parties reading an entire CP/CPS document must be at a certain level (above rudimentary?) to fully understand it.

My personal opinion is that the CPS should be written in clear language and issues in CPS should be file a bugzilla and fix it. I dislike that revocation creeps into manually written documents as typos do occur. Personally, I would love it if all CPS docs were written in plain language and were overly perspective on what the CA does with bug reports and audit findings being the primary consequence of having typos or accidentally failing to only meet the CPS requirements that are stricter than the BRs.

I suppose there is agreement that CCADB or MRSP doesn't have a requirement associated with the disclosure of a Certificate Problem Report reporting mechanism?

There is a requirement under 4.9.3 of the BRs to have the CPR mechanism clearly disclosed in section 1.5.2 of the CPS. CCADB implicitly requires a contact point as well.

Comment 5

[Dimitris Zacharopoulos]

Wayne, Jeremy, you both made good points.

We do support transparency which is the reason we decided to open this bug (err on the side of "safe"), and following good practice, when we are not 100% certain of something, we prefer to ask the community rather than assuming everything is fine.

By the end of next week a complete incident response will be provided that will hopefully address your questions as well.

Comment 6

[HARICA]

Full Incident Report

Summary

• CA Owner CCADB unique ID: A000035
• Incident description: Third parties can submit a Certificate Problem Report (CPR) via 2 email addresses as described in section 1.5.2 of the HARICA CP/CPS. CCADB records showed one of the two email addresses. Due to an internal procedure failure, the email alias disclosed in CCADB "expired", and emails were not being delivered. The problem was resolved soon after it was reported. An internal investigation revealed that a series of failures resulted in the email alias getting disabled but the good news was that the reporter found alternative methods to submit the report.
• Timeline summary:
  • Non-compliance start date: 2025-04-18
  • Non-compliance identified date: 2025-04-30
  • Non-compliance end date: 2025-04-30
• Relevant policies: Section 1.5.2 of the HARICA CP/CPS, Section 5.2 of the Mozilla Root Store Policy
• Source of incident disclosure: Externally reported

Impact

• Total number of certificates: N/A
• Total number of "remaining valid" certificates: N/A
• Affected certificate types: N/A
• Incident heuristic: N/A
• Was issuance stopped in response to this incident, and why or why not?: N/A
• Analysis: A third-party attempted to submit a Certificate Problem Report. The reporter checked the CCADB report CSV List of CA problem reporting mechanisms and obtained one of the two addresses listed in section 1.5.2 of the HARICA CP/CPS. When attempted to send an email to that destination email address, the email was returned as "undelivered". The reporter found an alternative method to submit the report via HARICA's web portal https://www.harica.gr/en/Contact/CPR which currently sends emails to the official email addresses listed in section 1.5.2 of the CP/CPS.
• Additional considerations: Based on our assessment, we do not consider this incident to be a TLS Baseline Requirements violation or a CCADB policy violation. However, improvements will be made internally to keep CCADB CPR reporting methods in sync with the HARICA CP/CPS.

Timeline

• Timeline (UTC+3):
  • 2025-04-03: A reminder about the upcoming expiry of email alias cert-problem-report [at] harica [dot] gr was sent to the administrator responsible for the alias
  • 2025-04-13: A second reminder about the upcoming expiry of email alias cert-problem-report [at] harica [dot] gr was sent to the administrator responsible for the alias
  • 2025-04-18 23:59:59: The email alias cert-problem-report [at] harica [dot] gr expired and emails could not be delivered to the intended recipients
  • 2025-04-30 01:57: HARICA received a CPR in the email alias high-priority-cert-problem-report [at] harica [dot] gr. This email arrived in both the internal ticketing system and the internal HARICA online collaboration tool. It was originated by the CPR web form. The reporter mentioned the issue with the email address not working, and an additional issue that is not related to this incident.
  • 2025-04-30 06:15: The email alias cert-problem-report [at] harica [dot] gr was restored
  • 2025-04-30 11:13: The reporter was notified that the CPR reporting email address was fixed
  • 2025-04-30 18:32: A preliminary incident report was submitted to bugzilla

Related Incidents

Bug	Date	Description
1959733	2025-04-10	Email filter errors can cause a CPR delivery failure. HARICA's CPR reporting email addresses do not currently have any blocking filters, but the administrator in charge of renewing the email alias had filters enabled. The alias renewal email was misplaced due to those filters
1894111	2024-04-29	CPR reporting methods were not included in CCADB. HARICA's CCADB CPR reporting field contains one of the two email addresses located in the CP/CPS so we do not consider this a policy violation. However, consistency between CP/CPS and CCADB fields is a reasonable expectation by the community so we already updated the CCADB field as soon as we realized the second email address was missing.

We couldn't find an incident in Bugzilla that shares exactly the same causes (expired/disabled email alias for a CPR reporting email address). HARICA actively monitors all incidents in Bugzilla.

Root Cause Analysis

Contributing Factor #1: One of the two email aliases used for reporting CPRs expired

• Description: All email addresses used by HARICA are managed by an in-house developed management system. Each email alias entry includes the alias email address, recipient email addresses, startdate and enddate, along with two persons ("owner" and "technical administrator") responsible for the alias. When this email alias was created in 2005, the owner arbitrarily set the expiration date to 2025. Email aliases associated with HARICA are usually valid for decades.
• Timeline (UTC+3):
  • 2025-04-18 23:59:59: The email alias cert-problem-report [at] harica [dot] gr expired and emails could not be delivered to the intended recipients
  • 2025-04-30 01:57: HARICA received a CPR in the email alias high-priority-cert-problem-report [at] harica [dot] gr. This email arrived in both the internal ticketing system and the internal HARICA online collaboration tool. It was originated by the CPR web form. The reporter mentioned the issue with the email address not working, and an additional issue that is not related to this incident.
  • 2025-04-30 06:15: The email alias cert-problem-report [at] harica [dot] gr was restored
  • 2025-04-30 11:13: The reporter was notified that the CPR reporting email address was fixed
  • 2025-04-30 18:32: A preliminary incident report was submitted to bugzilla
• Detection: The email address failure was reported by a third-party who submitted a CPR with an alternative method.
• Interaction with other factors: No

Contributing Factor #2: The email alias expiration email was missed by the person responsible

• Description: 15 and 5 days before the expiration of the email alias, the aliases management system sends an email to the "owner" and "technical administrator" responsible for the alias. In this particular case, the same person was recorded for both roles. The person's email filters misplaced the reminder emails and were overlooked.
• Timeline:
  • 2025-04-03: A reminder about the upcoming expiry of email alias cert-problem-report [at] harica [dot] gr was sent to the administrator responsible for the alias
  • 2025-04-13: A second reminder about the upcoming expiry of email alias cert-problem-report [at] harica [dot] gr was sent to the administrator responsible for the alias
• Detection: This factor was detected after the incident
• Interaction with other factors: The administrator's failure to read and respond to the alias expiration warning email, is what caused factor #1 to manifest.

Contributing Factor #3: Unclear language in the CP/CPS to instruct reporters to use one or the other reporting available mechanisms

• Description: Section 1.5.2 of HARICA's CP/CPS lists two email addresses that can be used to submit a CPR. The current language does not clearly say that a reporter should use the second email address if the first one doesn't work.
• Timeline: This language has been in HARICA's CP/CPS for years.
• Detection: N/A
• Interaction with other factors: No

Lessons Learned

• What went well:
  • HARICA reacted quickly and fixed the issue a few hours after it was reported
  • Additional reporting methods were available (two email addresses in the CP/CPS and two web forms on HARICA's website)
• What didn’t go well: The problem was not detected internally
• Where we got lucky: The reporter completed the submission using alternative methods, namely the second webform in https://www.harica.gr/en/Contact/CPR (High Priority Certificate Problem Reporting)
• Additional: Section 1.5.2 of HARICA's CP/CPS lists only the "email option" as the official way to submit a CPR. After reviewing several CPR-related incidents, HARICA decided to offer a web form that can be used to submit CPRs. The current implementation sends an email to the official aliases described in the CP/CPS. Each of those aliases, includes a recipient email address that causes a post to be automatically submitted to an internal HARICA online collaboration tool that gives instant alerting to the officers-on-duty in charge of handling the CPRs (we have 24x7x365 coverage). Although this implementation is effective, we were looking for ways to remove the dependency to the email service and therefore didn't include it in the CP/CPS as a distinct way of submitting CPRs. We are still trying to improve the solution using web-hooks so the form posts directly to the internal HARICA online collaboration tool without using the mail service.

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Review all HARICA PKI-related email aliases and ensure they are set to expire at the maximum allowed value (currently maxed at 2040)	Prevent	Root Cause # 1	This is a simple action item to ensure other email aliases are not affected by this issue	2025-05-09	Ongoing
Review all HARICA PKI-related email aliases and ensure two different persons are responsible to get the alerts	Prevent	Root Cause # 2	When the next email alias expiration takes place in 2040, two individuals will receive the expiration warning email	2025-05-09	Ongoing
Update the CP/CPS to update section 1.5.2 with clear instructions for third-parties to use any of the reporting methods described, and if one method fails for any reason, they should try to use another. The goal is for the third-party to successfully submit a CPR to the CA, even if they have to try again in case of one of the methods failing (for any reason)	Mitigate	Root Cause # 3	This information will be included in the publicly available CP/CPS for third parties to review	2025-05-23	Ongoing
Add the web form reporting option to section 1.5.2 of the CP/CPS, even in its current state (using the email service)	Mitigate	Root Cause # 3	This information will be included in the publicly available CP/CPS for third parties to review	2025-05-23	Ongoing
Implement web-hooks in the web form reporting option to remove dependency from the email service	Prevent	Root Causes # 1 and 2	Direct posting to the HARICA internal online collaboration will be tested before released to production	2025-06-27	Ongoing

Appendix

Comment 7

[HARICA]

Quick update on the action items.

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Review all HARICA PKI-related email aliases and ensure they are set to expire at the maximum allowed value (currently maxed at 2040)	Prevent	Root Cause # 1	This is a simple action item to ensure other email aliases are not affected by this issue	2025-05-09	Completed
Review all HARICA PKI-related email aliases and ensure two different persons are responsible to get the alerts	Prevent	Root Cause # 2	When the next email alias expiration takes place in 2040, two individuals will receive the expiration warning email	2025-05-09	Completed
Update the CP/CPS to update section 1.5.2 with clear instructions for third-parties to use any of the reporting methods described, and if one method fails for any reason, they should try to use another. The goal is for the third-party to successfully submit a CPR to the CA, even if they have to try again in case of one of the methods failing (for any reason)	Mitigate	Root Cause # 3	This information will be included in the publicly available CP/CPS for third parties to review	2025-05-23	Ongoing
Add the web form reporting option to section 1.5.2 of the CP/CPS, even in its current state (using the email service)	Mitigate	Root Cause # 3	This information will be included in the publicly available CP/CPS for third parties to review	2025-05-23	Ongoing
Implement web-hooks in the web form reporting option to remove dependency from the email service	Prevent	Root Causes # 1 and 2	Direct posting to the HARICA internal online collaboration will be tested before released to production	2025-06-27	Ongoing

Ben, if there is no objection, can you please set the next upate on 2025-05-23? HARICA will continue to monitor this bug for additional questions.