1972887 - A-Trust: TLS non-compliance detected during linter implementation

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Ramin Sabet]

Attachment: affectedCertificates.csv

Full Incident Report

Summary

• CA Owner CCADB unique ID: A000005
• Incident description: During implementation of the linting mechanism we realized that the certificates issued do not meet the necessary requirements as described by CA Browser Forum. SSL certificates are not the core business of A-Trust; In total, only 20 certificates are currently valid and will be revoked on Monday 2025-06-23.

Following issues have been found:

• Extension basic constraints not marked critical

• Some Subscriber Certificate do include Subject Alternative Name

• Invalid Subject Order

• Missing Organisation ID

• Timeline summary:

  • Non-compliance start date: 2022-04-06 (Application to CCADB)
  • Non-compliance identified date: 2025-06-16
  • Non-compliance end date: 2025-06-23

• Relevant policies:

  • 7.1.2.1.2 Root CA Extensions
  • 7.1.2.7.12 Subscriber Certificate Subject Alternative Name
  • 7.1.4.2 Subject Attribute Encoding

• Source of incident disclosure: Internal

Impact

• Total number of certificates: 66
• Total number of "remaining valid" certificates: 20
• Affected certificate types: TLS
• Incident heuristic:
  All TLS certificates issued by the following CAs were affected.

  https://crt.sh/?caid=161706 
  https://crt.sh/?caid=261525
  https://crt.sh/?caid=294116
  https://crt.sh/?caid=268764
  

  The affected Leaf-Certificates are attached in the appendix.
• Was issuance stopped in response to this incident, and why or why not?:
  Yes, in order to prevent the issuance of further non-compliant certificates, the issuance of new certificates was paused.
• Analysis: During the configuration of the CA, some details were not configured as the baseline requirement would imply and the absence of a linter mechanism allowed non-compliant TLS certificates to be issued.
• Additional considerations: Precertificates and certificates are affected equally. All customers were contacted and since many of the certificates are issued to our selfes, we will be able to revoke all the non-compliant certificates within five days.

Timeline

• 2025-06-16 - Identification of non-Compliance
• 2025-06-16 - Stop issuing of non-Compliant certificates
• 2025-06-16 - Root Cause Analysis
• 2025-06-17 - Root Cause Analysis & Fixing the issues
• 2025-06-17 - Issuing of one compliant test certificate using our test environment
• 2025-06-18 - Report of non-Compliance
• 2025-06-18 - Customer Information
• 2025-06-18 - Re-issuing of new compliant certificates
• 2025-06-18 - Since the fixes were only applied to one product two non-compliant certificates were issued due to human error and got revoked.
• Planned:
  • 2025-06-23 - Revocation of issued non-compliant certificates

Related Incidents

N/A

Root Cause Analysis

Root Cause Analysis methodology used: Fault Tree Analysis

• Top Event: Non-compliant certificate issued
  • AND
    • No linting
    • Auditor overlooked non-compliance
    • OR
      • Inadequate document analysis
      • Failed to detect non-compliance in test certificates

Contributing Factor #: 1

• Description: "No linting"
• Timeline:
  • 2025-06-16 - Finalization of linter development
  • 2025-06-17 - integrate linter to certificate issuing process
• Detection: Necessity of baseline requirements
• Interaction with other factors: N/A

Contributing Factor #: 2

• Description: "Auditor overlooked non-compliance"
• Timeline:
  • Yearly audit with sample certificates
• Detection: N/A
• Interaction with other factors: N/A

Contributing Factor #: 3

• Description: "Inadequate document analysis"
• Timeline:
  • 2022-04-06 Application to CCADB with a non-compliant certificate profile due to inadequate document analysis
• Detection: N/A
• Interaction with other factors: 4

Contributing Factor #: 4

• Description: "Failed to detect non-compliance in test certificates"
• Timeline:
  • 2022-04-06 When issuing test certificates their non-compliance was not detected by A-Trust
• Detection: N/A
• Interaction with other factors: 3

Lessons Learned

• What went well: Linter has found non-compliances and reported them
• What didn’t go well: Non-compliances were not detected during repeated manual checks.
• Where we got lucky: Only a small number of certificates are affected.
• Additional: N/A

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Implement linting mechanism in issuing process	Prevent	Root Cause # 1	Compare and monitor linter result with CA configuration and Baseline Requirements	2025-06-17	Complete
Implement stricter sample certificate testing strategy	Detect	Root Cause # 2	No non-compliant certificates will be issued	Next Audit	Ongoing
Assign more resources to certificate configuration	Prevent	Root Cause # 3	Product management will be involved in configuring the certificate profile in addition to development	2025-06-17	Complete
Implement linting mechanism in issuing process	Prevent	Root Cause # 4	Compare and monitor linter result with CA configuration and Baseline Requirements	2025-06-17	Complete

Appendix

Non-compliant certificates affected by the incedent and revoked

see attached file affectedCertificates.csv

Compliant certificates NOT affected by the incedent

N/A

Comment 1

[Malcolm D]

Shouldn't the revocations occur before you have planned? As the non-compliance was recognised on 2025-06-16, shouldn't the revocation occur by the same time (unspecified) on 2025-06-21? At the very least, 5 days after 2025-06-17,which is when the timeline identified the fix required.

Note: The timeline should have hh:mm resolution wherever possible.

Comment 2

[Daniel McCarney]

SSL certificates are not the core business of A-Trust

This reads as a deflection of your responsibilities. If certificates are not your core business, perhaps you don't need to be a root CA in the Mozilla root program?

Comment 3

[Daniel McCarney]

Analysis: During the configuration of the CA, some details were not configured as the baseline requirement would imply and the absence of a linter mechanism allowed non-compliant TLS certificates to be issued.
..
Contributing Factor #: 1
Description: "No linting"
Timeline:
2025-06-16 - Finalization of linter develop

Can you speak to why you waited until 2025 to begin linting your certificates? There have been numerous incidents between 2022 and today uncovered by linters that affected a variety of other CAs. Pre-issuance linting has also been a Mozilla root program recommended practice since at least 2019.

It seems like the timeline of this contributing factor should begin earlier than 2025.

Comment 4

[Wayne]

As others have noted the 5-day timeline should have started when identification of non-compliance started.

The incident report notes that:

Non-compliance end date: 2025-06-23

Which seems to be based on when revocation would occur, not when the certificate profile was fixed (2025-06-18 according to the report).

The related incident section is empty, which is strange as there are countless incidents from the past two years of these kinds of errors. CAs are expected to follow these incidents to understand what others did wrong, and how they can avoid similar issues.

On glancing at certificates the CA has issued I note the intermediary being used 'a-sign-SSL-07':
https://search.censys.io/certificates/789e625ae7bf6b604df93ca8eb99a30b0378a8f82a67562c66d8013926121529

I am interested in how you managed to configure your infrastructure to produce Subject Key IDs short enough to violate RFC 5280. I haven't seen that one before, and does mean that all certificates are still violating it - including ones issued today. Given this intermediary was produced in 2018 and it hasn't been noticed yet, perhaps a new auditor is in order.

The CCADB records for the audits all need the Audit Statement Date, Period Start Date and Period End Date updated. I do wish to note that while the audit mentions the baseline requirements are to be considered, it only cover ETSI policy. There are more parts of CCADB that need updated too, but let's not dwell on it.

The CCADB records note the following for 'Certificate Practice & Policy Statement':
https://www.a-trust.at/downloads/Downloads/Certificate%20Practice%20Statement/a-sign-ssl-ev/a-sign-ssl-ev_cps.pdf

This is listed as version 2.5, dated: 2024-08-29. Although the CCADB notes 2023-03-27 as the last updated date. This is found on the website under 'downloads' for 'a-sign-ssl-ev'. The version history implies a non-yearly update cadence, which I hope is improved going forward.

The OID listed for this CPS is: 1.2.40.0.17.2.22. Yet the OID for the policy included in a certificate issued today ends in 0.17.1.20. This seems to correspond to a 'a.sign premium mobile' qualified certificate held in a different cps: https://www.a-trust.at/downloads/de/Certificate%20Practice%20Statement/a-sign-qualified/a-sign-qualified_cps.pdf

Strictly speaking .040. and .40. are not byte-for-byte identical OIDs, but let's not dwell on that.

Moving back to the listed CPS in CCADB:

1.5.2 only lists a physical address as a contact method, and no method for handling a certificate problem report. The website does mention a revocation service that is available over the phone Mon-Fri 10am - 6pm. There is also an email address listed on CCADB that is the general support address.

2.3 notes that the CPS is updated "at least on an annual basis". This is not reflected in the version history, but is perhaps aspirational.

The rest of the CPS also seems aspirational, but does mention a 24x7 revocation service in 4.9.5? There is no mention of the revocation time periods, and the list of revocation reasons is also lacking.

4.10.1 notes:

Certicate status information can be obtained via CRL and OCSP. Expired certicates remain on the CRL, by the extension ExpiredCertsOnCRL.
CRLs are archived for thirty years after expiration.

Which presumably means that revoked certs are held nigh-indefinitely on the CRL rather than being pruned when they expire.

7.1.2 notes that:

The subordinate CA is using id-kp-serverAuth and id-kp-clientAuth as extkeyUsage.

Given that the subordinate CA is the intermediary, it is not, however subscriber certificates are using that extkeyUsage.

'qc-Statement' is also listed as present in subscriber certificates. It is not.

There will be more issues than that, but a glance at an example certificate and the policy brought up those issues. I strongly recommend restarting a large amount of your practices from scratch if inclusion in root programs is intended in the near future.

Comment 5

[Corey Bonnell]

I am interested in how you managed to configure your infrastructure to produce Subject Key IDs short enough to violate RFC 5280.

I admittedly only looked at and linted a few certs referenced in the spreadsheet, but I don't think the key identifiers violate RFC 5280. RFC 5280 does not mandate any particular key identifier generation method, and instead only defines two possible methods that can (not must) be used. There are more key identifier methods specified in RFC 7093, some of which are in use today by some CAs. The short key identifiers appear to be generated using the second method defined in RFC 5280; this is indicated in the pkilint output, as pkilint detects the key identifier generation method used (assuming the generation method is defined in RFC 5280 or 7093).

Which presumably means that revoked certs are held nigh-indefinitely on the CRL rather than being pruned when they expire.

As far as I am aware, there is no requirement in RFC 5280 or the TLS BRs to prune expired CRL entries. In fact, some PKIs require that CRL entries remain on the CRL for some period of time after expiry. Retaining expired entries is not useful for an online protocol like TLS, but is useful when previously created signatures need to be verified. For example, the CS Baseline Requirements requires that CRL entries remain on the CRL for 10 years after certificate expiry so that previously signed binaries can be verified even after the corresponding code signing certificate has expired.

Pre-issuance linting has also been a Mozilla root program recommended practice since at least 2019.

It has also been required by all TLS-issuing CAs since March 15th of this year, according to the TLS BRs.

Comment 6

[Ramin Sabet]

Thank you all for your contributions, we value your insights.

Answering your comments was our top priority after returning from yesterdays bank holiday.

AD Comment 1:

We started counting days from reporting date, but we will rescedule revocation to 2025-06-21

AD Comment 3:

You are absolutely right, if we had implented linting earlier, we would have cought these non-compliances earlier.

AD Comment 4:

We missunderstood the related incident section and thought we had to link other incidents involving our CA.
As Corey Bonnell in Comment 5 pointed out, the short SKI complies with RFC5280. The SKI is generated by bouncycastle. There is the CreateTruncatedSha1KeyIdentifier function, which states “Return a RFC 3280 type 2 key identifier.”
We will improve our process to update the CPS at least yearly
The different OID was a human error and was fixed.
Right now further contact Methods are in our PDS. We will update the Information in the next Version of the CPS.
AD Comment 4/4.10.1 notes ExpiredCertsOnCRL
Yes that is correct.
The qc-statement is only present in QWACs. We will clarify it in the next Version of the CPS.

best regards
ramin

Comment 7

[Ramin Sabet]

Saturdary 2025-06-21 17:32 (GMT+2) all Certificates reported have been revoked.

regards
ramin

Comment 8

[Daniel McCarney]

AD Comment 3:
You are absolutely right, if we had implented linting earlier, we would have cought these non-compliances earlier.

Does this response mean we will see this incident report's timeline and root cause analysis updated with a discussion of why linting wasn't prioritized earlier?

Comment 9

[Ramin Sabet]

Reason for the delayed implementation of SSL certificate linting
The implementation of linting for issued SSL certificates was delayed due to several factors:

• The trust center primarily issues a large volume of user certificates, while SSL/TLS certificates account for only a smaller share of the overall portfolio.
• Changes required for certificates in the Austrian e-government solution necessitated adjustments that postponed the planned upgrade of the certification authority.
• In addition, the specific requirements for SSL/TLS certificates were not fully integrated into the linting process due to a human error during implementation.

To address this, dedicated measures have been introduced to ensure that TLS certificates receive more focused handling going forward., as described in the action items of the issue.

regards
ramin

Comment 10

[Jeremy]

Have you run a linter on all user certs? The linters look for a variety of compliance issues, including compliance with 5280 (which even non-TLS certs need to comply with), ICA compliance, compliance with teh SMIME BRs, etc. I think you need to run a linter over all certs to see what else might have been missed without linter implementation.

Comment 11

[Ramin Sabet]

Dear Jeremy,

we did not run the Linters against the millions of issued qualified certificates, but when implementing the linters we also tested some sample (non TLS) certificates.

Although we used the profile (pkimetal) [RFC 5280 Document signing certificate] the only issues are TLS related like:

• validity longer than 398 days, which is true for TLS certificates.

regards
Ramin

Comment 12

[Rob Stradling]

(In reply to Ramin Sabet from comment #0)

Following issues have been found:

A plurality of "issues", specifically...

• Extension basic constraints not marked critical
• Some Subscriber Certificate do include Subject Alternative Name
• Invalid Subject Order
• Missing Organisation ID

https://www.ccadb.org/cas/incident-report says:
"How are reports scoped?
There SHOULD be a single Incident Report for each distinct matter"

Q1: Can A-Trust explain why it chose to bundle multiple distinct issues into a single incident bug?

Root Cause Analysis methodology used: Fault Tree Analysis

• Top Event: Non-compliant certificate issued
  • AND
    • No linting
    • Auditor overlooked non-compliance
    • OR
      • Inadequate document analysis
      • Failed to detect non-compliance in test certificates

I find the primary focus on linting somewhat troubling.

In my view, the quality (or, in this case, the complete lack) of pre-issuance linting can never be considered a root cause of certificate misissuance. The purpose of pre-issuance linting is "to test the technical conformity of each to-be-signed artifact prior to signing it", but the responsibility for that "technical conformity" lies entirely with the CA, not with the linter(s). (Relatedly, see https://www.sectigo.com/resource-library/root-causes-437-dont-blame-the-linter).

Q2: Does A-Trust accept this viewpoint?

Contributing Factor #: 1

• Description: "No linting"
• Timeline:
  • 2025-06-16 - Finalization of linter development
  • 2025-06-17 - integrate linter to certificate issuing process

Q3: Is A-Trust able to share details of which linter(s) it is integrating into its certificate issuing process?

(Comment 11 mentioned pkimetal, but IIUC only in the context of testing some non-TLS sample certificates).

Action Items

Action Item	Kind	Corresponding Root Cause(s)	Evaluation Criteria	Due Date	Status
Implement linting mechanism in issuing process	Prevent	Root Cause # 1	Compare and monitor linter result with CA configuration and Baseline Requirements	2025-06-17	Complete
...
Implement linting mechanism in issuing process	Prevent	Root Cause # 4	Compare and monitor linter result with CA configuration and Baseline Requirements	2025-06-17	Complete

Duplicate action item ^^.

Comment 13

[Dimitris Zacharopoulos]

(In reply to Ramin Sabet from comment #11)

Dear Jeremy,

we did not run the Linters against the millions of issued qualified certificates, but when implementing the linters we also tested some sample (non TLS) certificates.

Although we used the profile (pkimetal) [RFC 5280 Document signing certificate] the only issues are TLS related like:

• validity longer than 398 days, which is true for TLS certificates.

regards
Ramin

Dear Ramin,

The fact that you issued millions of qualified (or other types of) certificates without linting, and with the findings disclosed in this incident for the TLS certificates, makes it very likely that more of your already-issued certificates deviate from your CP/CPS.

When a new version of a specific linting software is released, it usually includes additional checks that the previous version didn't have. It is therefore extremely useful to check the entire corpus of issued certificates (especially the valid ones) against the new version of the linter to make sure they don't contain any of the issues detected by the updated linter. If the entire corpus is unmanageable to run with the new version, some sampling might be applied.

I don't have a specific question but I'd like you to consider to adopt good practices when updated versions of linting software are released, and I'd be curious to know how you plan to address deficiencies in existing valid non-TLS certificates.

Comment 14

[Ramin Sabet]

(In reply to Rob Stradling from comment #12)

Q1 multiple distinct issues
in our opinion this is one issue which was discovered as described in the initail incident Report, which includes several non conformities.

Q2 We definitely did not blame the linter, but we Blame us for not having implemented the linting process, which is OR-ed to "failed to detect ..." in the fault tree.

Q3 we implemented pkimetal in the following Profiles: TLS BRs (Precertificate/Certificate) DV/OV/EV respectively.

We choose to create two Action items since they tackle different root causes, but you are Right we could also have referred to both root causes in one Action item.

Comment 15

[Ramin Sabet]

(In reply to Dimitris Zacharopoulos from comment #13)

you are absolutely right that It is only a benefit to take advantage of the fact that linters exist. As mentioned we have already used the linters to test sample certificates when we implemented the linting process for BRs.

We will implement rules to lint sample sizes of certificates when

• we change certificate profiles
• new versions of linters are released

Comment 16

[Ramin Sabet]

Report Closure Summary

• Incident description: During the implementation of the linting mechanism it had become apparent that the certificates issued did not meet the necessary requirements as described by CA Browser Forum. In total, only 20 certificates were valid and have been revoked within the 5 day period.

• Incident Root Cause(s):

• Top Event: Non-compliant certificate issued

  • AND
    • No linting
    • Auditor overlooked non-compliance
    • OR
      • Inadequate document analysis
      • Failed to detect non-compliance in test certificates

• Remediation description:

  • All valid non-compliant TLS certificates have been revoked
  • Implemented linting mechanism to benefit from the community efforts
    • mandatory linting process for all TLS certificates
    • linting of random samples of other issued certificates, when new linters are released or the certificate profile is changed

• Commitment summary:

  • Established a process to update CPS on time
  • Appointed a dedicated Owner for TLS Certificates
  • Regular review of new developments and incorporation of best practices

All Action Items disclosed in this report have been completed as described, and we request its closure.

Comment 17

[incident-reporting]

This is a final call for comments or questions on this Incident Report.

Otherwise, it will be closed on approximately 2025-07-08.