Closed Bug 1957962 Opened 1 year ago Closed 1 year ago

Telekom Security: QCStatement with http link to PDS

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: stefan.kirch, Assigned: stefan.kirch)

Details

(Whiteboard: [ca-compliance] [uncategorized])

Preliminary Incident Report

Summary

Incident description:
In preparation for our audit, we found today that two certificates issued for testing purposes are incorrect. Within the QCStatement extension, the link to the respective PKI Disclosure Statements (PDS) is defined as an http link instead of an https link. The certificates have never been used and were both immediately revoked. No customer certificates are affected.

Relevant policies:
ETSI 319 411-2, see ETSI EN 319 412-5 for the relevant requirement QCS-4.3.4-03 “As a minimum, a URL to a PDS provided in this statement shall use the "https" (https://) scheme, IETF RFC 2818 [5] or later documents updating this specification.”

Source of incident disclosure:
Self Reported

Assignee: nobody → stefan.kirch
Status: NEW → ASSIGNED
Whiteboard: [ca-compliance] [uncategorized]

Full Incident Report

Summary

  • CA Owner CCADB unique ID: A000059
  • Incident description:
    In preparation for our audit in combination with the launch of a new issuing platform for this type of certificates, we found that two certificates issued for testing purposes were incorrect.
    The error was that within the QCStatement extension the link to the respective PKI Disclosure Statements (PDS) was defined as an http link instead of an https link.
    As there were only issued two certificates for testing purposes, these certificates have never been used in a producitve environment and were both immediately revoked. No customer certificates are affected.
  • Timeline summary: (all Times are CEST, i.e. GMT+2)
    • Non-compliance start date: 2025-02-13 10:01
    • Non-compliance identified date: 2025-04-02 07:58
    • Non-compliance end date: 2025-04-02 11:34
  • Relevant policies: ETSI EN 319 411-2 which refers to ETSI EN 319 412-5 for the relevant requirement QCS-4.3.4-03 “As a minimum, a URL to a PDS provided in this statement shall use the "https" (https://) scheme, IETF RFC 2818 [5] or later documents updating this specification.”.
  • Source of incident disclosure: Self Reported

Impact

Timeline

(all Times are CEST, i.e. GMT+2)
2024-03-05 12:00 The initial configuration of the new platform for this certificate type was set.
2025-02-13 10:01 The first certificate of this type was issued on the new platform in preparation of the launch.
2025-02-13 13:15 The QA for the first certificate was done without finding an error using zlint, certlint, x509lint.
2025-04-01 10:35 The second certificate was issued as part of a final quality check for the audit.
2025-04-02 07:58 The error was found by linting the certificate with pkimetal within crt.sh: https://crt.sh/?id=17552531562&opt=pkimetal .
2025-04-02 08:29 Both affected certificates were revoked.
2025-04-02 08:44 An incorrect entry in the certificate template that led to the misissuance was found, a CR to correct the template was planned.
2025-04-02 10:15 Our analysis showed that zlint, certlint and x509lint do not have a lint for this requirement.
2025-04-02 11:34 The certificate template was corrected.
2025-04-02 16:09 A Preliminary Incident Report was filed in Bugzilla.
2025-04-02 16:49 Our software supplier was assigned to create a pull request for zlint that checks the existence of an https link to the PDS within the QCStatement extension.
2025-04-03 09:10 Our Conformity assessment body was informed about the bug.
2025-04-03 11:19 As part of the audit, a new certificate was issued and successfully linted with pkimetal: https://crt.sh/?id=17600815196&opt=pkimetal.

Related Incidents

D-Trust: QCStatement with http link of PKI Disclosure Statements
Related Bug ID: 1939809
Related Bug URL: https://bugzilla.mozilla.org/show_bug.cgi?id=1939809
Opened date: 2025-01-03
Description: The incident is similar, as the affected certificates had the same error (http link instead of https link), even if the root cause was different.
Remark: As we monitor all incidents in Bugzilla, we have also checked our certificates for this error in respect of this incident. Since the certificates on the old platform were correct, we did not pursue this bug any further and did not keep it in mind when launching the new platform.

Root Cause Analysis

Contributing Factor 1: Insufficient QA within the initial configuration of the new certificate templates

  • Description:
    For the initial configuration of the certificate template for this type of certificate within the new platform the template from the old platform was used.
    Unfortunately the handling of the PDS-URL had changed within the old platform so that the defined URL from the template was ignored and instead a URL configured in another setting was used. As part of the adaptation of the old platform, however, this ignored part was not deleted from the template, so that there was still an old comment there that referred to an old version of the ETSI standard that did not require an https path to the PDS.
    The error was not seen during the QA of this first certificate, although the certificate was checked using the four-eyes principle and all certficates are linted against the latest versions of zlint, certlint and x509lint.
  • Timeline:
    From 2024-03-05 12:00 to 2025-04-02 11:34
  • Detection:
    The error was found within the QA of the second certificate using the pkimetal lint.
  • Interaction with other factors:
    zlint, certlint and x509lint do not find this error until today (see Contributing Factor 2).

Contributing Factor 2: zlint, certlint and x509lint do not find this error

  • Description:
    zlint, certlint and x509lint are used for pre- and post-sign-linting of all our TLS server certificates, and thus also when issuing the first certificate, but zlint, certlint and x509lint do not find this error.
  • Timeline:
    n/a
  • Detection:
    The error was found within the QA of the second certificate using the pkilint within pkimetal.
  • Interaction with other factors:
    none

Lessons Learned

  • What went well:
    The misissuance was detected before launching the new plattform, so that only two certificates for testing were affected.
    All affected certificates were revoked in time.
  • What didn’t go well:
    Despite a QA according to a four-eyes principle and presign-linting all certficates with zlint, certlint and x509lint the error was not found within the first QA.
  • Where we got lucky:
    The affected certificates were never used in a productive environment.
    No customers were affected.
  • Additional:
    none

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Adaptation of the testing requirements to lint certificates from the test resp. pre-productive environment against pkimetal in addition to manual testing in a four-eyes principle Prevent Root Cause # 1 no error message from pkimetal 2025-04-15 Ongoing
Contribute to zlint with a PULL-request to implement a corresponding lint Prevent Root Cause # 2 error message from zlint when linting the misissued certificates again 2025-04-15 Ongoing

Appendix

None, see the list of affected certificates above.

Our software supplier has added the pull request to github: https://github.com/zmap/zlint/pull/935
Action item#2 is settled with that.

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Adaptation of the testing requirements to lint certificates from the test resp. pre-productive environment against pkimetal in addition to manual testing in a four-eyes principle Prevent Root Cause # 1 no error message from pkimetal 2025-04-15 Ongoing
Contribute to zlint with a pull request to implement a corresponding lint Prevent Root Cause # 2 error message from zlint when linting the misissued certificates again 2025-04-09 Completed

We have included in our knowledge base a description of how to lint certificates that need to be tested manually using the pkimetal web interface.
Also Action item#1 is settled with that.

Action Items

Action Item Kind Corresponding Root Cause(s) Evaluation Criteria Due Date Status
Adaptation of the testing requirements to lint certificates from the test resp. pre-productive environment against pkimetal in addition to manual testing in a four-eyes principle Prevent Root Cause # 1 no error message from pkimetal 2025-04-14 Completed
Contribute to zlint with a PULL-request to implement a corresponding lint Prevent Root Cause # 2 error message from zlint when linting the misissued certificates again 2025-04-09 Completed

We are monitoring this bug for feedback. Please let us know if there are any comments or questions.

I think PKIlint catches this error. Have you thought about using PKIlint?

We have thought about it in the past but considered zlint to have a very high quality and appreciated the broad community behind it (which also motivated our pull request to zlint).
However, in recent past and due to this bug in particular, we are reconsidering the implementation of pkilint or pkimetal.

I realize that this may seem like a minor quibble, but I suspect that the Subject:organizationIdentifier value ("VATDE-DE254595345") in the first affected certificate is not entirely correct; in fact, the ISO 3166 identifier of the country that issued the VAT number ("DE") appears twice in the organizationIdentifier. Adding the country code as a prefix to a VAT number is common practice, especially in intra-community transactions, for disambiguation purposes, but, as far as I know, it is not an integral part of the VAT number itself, especially when the country that assigned it is already known from the context (in this case, it is specified by the two letters "DE" following "VAT"). But I may be wrong. This aspect is not addressed in the CABF BRs, nor in the ETSI standards, apparently.

From our point of view, the country code is an integral part of the VAT ID, see Article 215 of EU Directive 2006/112
https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32006L0112

FWIW, I agree with Adriano on this one. If the author of ETSI EN 319 412-1 wanted to include the country identifier within the "number" itself, it would explicitly state it. IMO including the country identifier in the number itself is redundant because it is already included in the scheme.

Since you mentioned the EU Directive, please note that for Greece, the exception was for the country identifier in the VAT scheme. The expected value would be "VATEL-123456789". To confirm the expected behavior, please check the examples in section 5.1.4 of ETSI EN 319 412-1. There is an explicit example for VAT.

Correct, the country code is only included after the scheme used as stated in the ETSI EN 319 412-1. As Dimitris says, the examples can give you a better view of what´s required.

Hi Adriano, Dimitris, Inigo,

I fully agree with you that the second “DE” is redundant, and so we also had some internal discussions about it before we decided to include it in the registration reference in the future.

In the past we have not set the country code again in the registration reference, conform to the examples in the EVGL and the S/MIME BR. The discussion came up last year when we reviewed the new requirements in ETSI 319 412-4, e.g. WEB-4.1.2-4, which requires the organizationIdentifier to be set in QWACs and therefore the VAT registration scheme has become more important to us.

On the one hand there are the examples in the EVGL and in the S/MIME BR in which the prefix of the identifier is not set, but on the other hand we have (besides the definition in Article 215 quoted above in comment#8) the fact that the country code is also part of the tax number in the official assignments from the Federal Central Tax Office (“...the VAT identification number assigned to you is: DE1234...”). When validating against these official assignments, it seemed “more correct” for us to consider “DE” as an integral part of the registration reference and to set it again.

At the same time, the use of the EUID as part of the NTR scheme in S/MIME certificates was also discussed and there we see a similar implementation: within the registration reference the country code as part of the business register identifier is set again "NTRDE-DER3306.HRB12345". So it seemed to be consistent to set the country code also again in the VAT scheme.

But as stated above, we see the repetition of the country code also as redundant information and therefore we can happily do without it. However, I would appreciate if we could clarify this in ETSI EN 319 412-1 as well as in the EVGL and S/MIME BR, as in my view both variants are currently within the scope of interpretation of the specifications and none of the variants is really wrong.

Note: Last but not least, it would of course be desirable if we could make the rules for the organizationIdentifer consistent for all certificate types since the requirements in ETSI, EVGL and S/MIME BR currently deviate from each other (e.g. the EVGL do not allow the use of LEI, the S/MIME BR also allow GOV and INT in some cases, the requirements in ETSI 412-1 are binding only if the legal semantics identifier is also set, which is usually not the case). However, this discussion goes beyond the scope of this bug and must be dealt with elsewhere. As we are not yet a member of the SCWG or the SMCWG, we cannot provide clarification there, but we can bring it to ETSI ESI for clarification in ETSI EN 319 412-1.

(In reply to Stefan Kirch from comment #11)

At the same time, the use of the EUID as part of the NTR scheme in S/MIME certificates was also discussed and there we see a similar implementation: within the registration reference the country code as part of the business register identifier is set again "NTRDE-DER3306.HRB12345". So it seemed to be consistent to set the country code also again in the VAT scheme.

We agree with this interpretation. If the German authorities assign a company the VAT number DE12345678 (i.e. this is the number included in an invoice, for example), the logical interpretation of the SBR is to build the OrgID as VATDE-DE12345678

This is not in line with the ETSI standard(In reply to Pedro Fuentes from comment #12)

(In reply to Stefan Kirch from comment #11)

At the same time, the use of the EUID as part of the NTR scheme in S/MIME certificates was also discussed and there we see a similar implementation: within the registration reference the country code as part of the business register identifier is set again "NTRDE-DER3306.HRB12345". So it seemed to be consistent to set the country code also again in the VAT scheme.

We agree with this interpretation. If the German authorities assign a company the VAT number DE12345678 (i.e. this is the number included in an invoice, for example), the logical interpretation of the SBR is to build the OrgID as VATDE-DE12345678

Pedro, with all due respect, this is not in line with the ETSI text and example. If you want to argue about "relaxing" this rule or to clarify that the intent is not to repeat the country identifier, regardless if the government "prints" or "assigns" this number in some document (invoice or otherwise), you can definitely bring it up.

I believe the EUID is different because this scheme is not identified on its own (there is no EUID prefix).

Ultimately I see and appreciate all the arguments but this is one of those rare cases where I believe the ETSI standard is pretty clear, and the example provided makes it even more clear. If you have specific language to propose to make it even clearer, please contact me offline and I can bring it up in ETSI ESI.

Pedro, with all due respect, this is not in line with the ETSI text and example. If you want to argue about "relaxing" this rule or to clarify that the intent is not to repeat the country identifier, regardless if the government "prints" or "assigns" this number in some document (invoice or otherwise), you can definitely bring it up.

This conversation, while interesting, is not relevant to this bug. The certificate in question does not assert the legalPerson semantics identifier, so the syntax requirements in EN 319 412 1, clause 5.1.4 are not applicable. And even if the certificate contained the semantics identifier, I believe it would still comply with clause 5.14. Happy to discuss this in another venue.

The EV Guidelines provides the following guidance for the Registration Reference in a VAT orgId:

Reference allocated by the national tax authorities to a Legal Entity.

I think this definition is broad enough to allow either form of the VAT.

Following on from the last comment, we would like to support the discussion about the organizationIdentifer syntax within ETSI ESI as well as in the CA/Browser Forum, but take it out of the scope of this bug.

In the meantime, the pull request contributed to zlint (see Action Item#2) is also committed, see https://github.com/zmap/zlint/pull/935/commits/9fd17798e55b71585afb9ff156b409d6d9639229.
Are there any other comments on the original scope of this bug?

As there are no further comments or questions and all Action Items have been completed, we would like to request the closure of this bug. Accordingly, here is the Incident Report Closure Summary.

Incident Report Closure Summary

  • Incident Description: 2 Certificates were misissued: Within the QCStatement extension, the link to the respective PKI Disclosure Statements (PDS) was set as an http link instead of an https link.
  • Incident Root Cause(s): The certficates were misissued due to an insufficient QA within the initial configuration of the new certificate templates in a new certificate management platform. The linting tools used did not prevent this error as there was no rule set for this requirement, although a current version was used.
  • Remediation Description: To address the issue, internal processes for testing were updated to lint certificates from the test resp. pre-productive environment against another linting tool (pkimetal). Furthermore a pull request for a corresponding lint was contributed to zlint (in the meantime committed).
  • Commitment Summary: We will continuously strengthen our processes, follow the updates of ETSI and CA/Browser Forum specifications as well as Root Policies, and regularly train staff to avoid such incidents in the future.
Flags: needinfo?(incident-reporting)

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2025-05-21.

Flags: needinfo?(incident-reporting)
Whiteboard: [ca-compliance] [uncategorized] → [close on 2025-05-21] [ca-compliance] [uncategorized]
Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Whiteboard: [close on 2025-05-21] [ca-compliance] [uncategorized] → [ca-compliance] [uncategorized]
You need to log in before you can comment on or make changes to this bug.