1936908 - DigiCert: Encoded HTML entities in attribute values

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Tim Hollebeek]

Six certificates were issued with HTML-encoded attribute values instead of the correct values. This should have been caught by pkilint, but the version of pkilint running in production was not the latest version. We have updated to the latest version of pkilint in production, and are in the process of revoking the mis-issued certificates.

We will file a full report by Monday, December 23rd.

Comment 1

[Tim Hollebeek]

On track for full report.

Comment 2

[Tim Hollebeek]

Incident Report

Summary

This incident is similar to bug 1912225 filed by Sectigo where HTML encoded characters appeared in subject values. In response to the issue, DigiCert added a check to its open source pkilint that can detect HTML encoded characters that are not permitted. We released the new version of pkilint on October 2. However, engineering did not update pkilint on October 2, and we were unaware of an issue similar to Sectigo’s within our code. Prior to this bug, pkilint internal deployment lagged the public release by two months and was scheduled for an update on December 11th. This lag in update was for part of our internal testing and approval process. Our post-issuance scans, conducted as part of DigiCert’s compliance/internal audit regime, use the newer version of pklint and detected 6 certs with this issue. All certs were revoked within the mandated five-day timeframe.

Impact

Six certificates were issued with SubjectDN information that contained HTML encoded characters.

Timeline

All times are UTC.

2024-10-02

• 15:18 PKILint (v0.12.0) released, includes update for this failure mode
  2024-11-23
• 22:36 PKILint-Internal (v0.14.0) introduced to main branch for testing
  2024-12-11
• 20:15 PKILint (v0.12.5) released to production
  2024-12-12
• 04:34 Trust Assurance finds 6 misissued certificates in post-issuance scans
  2024-12-17
• 01:16 All certs revoked.

Root Cause Analysis

These incident reports are most useful when they help other issuers improve their operations, and that happened here.
We regularly read reports filed by other CAs and update pkilint with findings. In this case, DigiCert updated pkilint to detect non-allowed HTML encoded entries in certificates.
Usually, we retroactively scan our existing certificate population when we add new checks to pkilint, but that did not happen in this instance. Finding the handful of misissued certificates that existed at that time would have alerted us to the fact that our systems had a flaw similar to Sectigo’s.
This issue also highlights the importance of keeping linters up to date, as that accelerates the speed at which newly discovered failure modes can be eliminated from the ecosystem. Effective next March, there is a three-month SHOULD in the Baseline Requirements for updating such linters, and we strongly encourage other CAs to update their linters to make sure they are not impacted by this issue as well.
In addition to updating the pre-issuance linter, we are also adding the same check to the issuance pipelines as well. We’re also continuing to improve our documented procedures for pkilint releases and related automated alerts and management. Pkilint development is being integrated into our standard engineering release management procedures, allowing things like automated dependency checking and alerts.

Lessons Learned

What went well

• DigiCert updated pkilint rapidly to detect the issue noted in the original Sectigo bug.
• DigiCert has implemented pre-issuance linting well in advance of the deadlines required by the TLS BR. DigiCert also has implemented the post-issuance linting recommended/optional in the TLS BR.
• Deployment of pkilint was proceeding at a speed which is already in line with recommended future best practices according the most recent Baseline Requirements.

What didn't go well

• We didn’t immediately scan for existing certificates after updating pkilint. Certs weren’t found until the next scheduled post-issuance linting.
• Pkilint was running on our production systems (v0.9.11), but not the latest version that was available (v0.12.5). This was within the 90 day recommendation set by Section 6.6.1 of the TLS BR. Running the latest linter versions would reduce misissuance.
  Two months is a long time for testing and deployment of updated linting software. Considering linting is a compliance issue, the time period between a linter version update and internal deployment should be minimized.

Where we got lucky

• We regularly conduct post-issuance linting, as recommended by the CABF, to find issues like these where newer linters retroactively find older issues that had gone undetected. Although post-issuance linting does not prevent mis-issuance, it allows us to rapidly detect and remediate issues.

Action Items

Action Item	Kind	Due Date
Create formal process to use trackable tickets for internal pkilint updates	Prevent	Already completed
Automated alert mechanism to notify if pkilint is out of date	Detect	Already completed
Add the same html entity check from pkilint to issuance pathways	Prevent	2025-01-07
Explicitly add re-scanning existing certificates to pkilint release procedures/checklist	Prevent	2025-01-07

Appendix

Details of affected certificates

serial_number	sha256_hash
022afc5af1ff527a92c014a452bf1d0f	3f2d01b1bd0b2d92d5cd808479c054601f31fc739ea98d13d2d371eed01baacb
06474ab58eb1a3574de9dc1f6bc75490	b49dce76495cfca9e87b098546cbeee6c5cff585519b409535cd5667b6f073fc
07bf7a2f019dfe96f51af7c0823aa276	704ebe53e66a62f309bb4d00d5eceebccb5222c72929cfca3937725722c8b5a1
0ccc134c7668b358048a53a748dbbc32	838f4c286a2552c569231e3d264a9c815f8495e88b1b2fefe0971851e0289f70
0cfeb4470921ee843f85caf422e68db5	fcdf331fffd14db2e64b7c288543399bdcd3c43459befcca1da540775b41a9c8
0e2731d28dcba23dd3201cd4bff5a286	cf774d95adfde49c740d94d53523f1e334091eae776ffab4e5ea45da1eb614db

Comment 3

[Tim Hollebeek]

Happy holidays! Will answer any questions next Monday.

Comment 4

[Tim Hollebeek]

Onward to 2025!

Comment 5

[Tim Hollebeek]

Nothing new here.

Comment 6

[DigiCert]

We will have an update on the action items here later this week.

Comment 7

[DigiCert]

I have confirmed with our engineering teams that both action items are complete.

Comment 8

[Rob Stradling]

I'm very pleased to see that as part of responding to this incident DigiCert undertook an Action Item to "Add the same html entity check from pkilint to issuance pathways"!

Pre-sign linting of each to-be-signed certificate, which is now required by TLS BR 4.3.1.2 (ballot SC75), is intended only to be a last line of defence against misissuance. Since the CA Owner bears full responsibility for any certificates it misissues, it befits the issuance pathways to implement as many controls as possible to minimize the chances of any nonconformities ever reaching the to-be-signed certificates that will be checked by the linter(s).

I realise that in the case of this bug the linter was created by the affected CA Owner itself rather than by an external party; nevertheless, since linters do not bear any responsibility for misissuance, Sectigo's advice to all CA Owners is always: Don't Blame the Linter.

With that in mind, I have a small suggestion:
Would DigiCert consider editing this bug's Summary field to remove the (in my view) inappropriate "due to outdated pkilint" suffix?

Comment 9

[DigiCert]

Sure, fair enough.

We did want to highlight the value of always making sure the linter is up to date, though, especially with the upcoming deadline. It's the lesson other CAs could learn the most from.

Comment 10

[DigiCert]

Closing summary coming soon on this one too.

Comment 11

[DigiCert]

DigiCert completed all action items listed for this bug.

Action Items

Action Item	Kind	Due Date
Create formal process to use trackable tickets for internal pkilint updates	Prevent	2024-12-20
Automated alert mechanism to notify if pkilint is out of date	Detect	2024-12-20
Add the same html entity check from pkilint to issuance pathways	Prevent	2025-01-07
Explicitly add re-scanning existing certificates to pkilint release procedures/checklist	Prevent	2025-01-07

Incident Report Closure Summary
Incident Description:
This incident is similar to bug 1912225 filed by Sectigo where HTML encoded characters appeared in subject values. In response to the issue, DigiCert added a check to its open source pkilint that can detect HTML encoded characters that are not permitted. We released the new version of pkilint on October 2. However, engineering did not update pkilint on October 2, and we were unaware of an issue similar to Sectigo’s within our code. Prior to this bug, pkilint internal deployment lagged the public release by two months and was scheduled for an update on December 11th. This lag in update was for part of our internal testing and approval process. Our post-issuance scans, conducted as part of DigiCert’s compliance/internal audit regime, use the newer version of pklint and detected 6 certs with this issue. All certs were revoked within the mandated five-day timeframe.

Incident Root Cause(s):
These incident reports are most useful when they help other issuers improve their operations, and that happened here.
We regularly read reports filed by other CAs and update pkilint with findings. In this case, DigiCert updated pkilint to detect non-allowed HTML encoded entries in certificates.
Usually, we retroactively scan our existing certificate population when we add new checks to pkilint, but that did not happen in this instance. Finding the handful of misissued certificates that existed at that time would have alerted us to the fact that our systems had a flaw similar to Sectigo’s.
This issue also highlights the importance of keeping linters up to date, as that accelerates the speed at which newly discovered failure modes can be eliminated from the ecosystem. Effective next March, there is a three-month SHOULD in the Baseline Requirements for updating such linters, and we strongly encourage other CAs to update their linters to make sure they are not impacted by this issue as well.
In addition to updating the pre-issuance linter, we are also adding the same check to the issuance pipelines as well. We’re also continuing to improve our documented procedures for pkilint releases and related automated alerts and management. Pkilint development is being integrated into our standard engineering release management procedures, allowing things like automated dependency checking and alerts.

Remediation Description:
We built and added an automated alerting system to monitor the version of pkilint. Pkilint is also now part of our regular engineering processes, and is no longer a separately maintained tool. The issuance pathways have been updated to block this bug, and the release checklist for pkilint has been updated.

Commitment Summary:
These enhancements should prevent pkilint from being out of date in the future, in advance of the upcoming deadline.

@Ben Wilson – As all action items are completed, will you please close this bug?

Comment 12

[Ben Wilson]

I'll schedule for this to be closed Wed. 12-Feb-2025, unless there are additional questions or issues to discuss.

Comment 13

[Mike Shaver (:shaver emeritus)]

Thank you for the clear description of what is most relevant for other CAs! It really helps make sure that the ecosystem learns from these incidents.