Sectigo: HTML encoded characters in subject attribute values
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: martijn.katerbarg, Assigned: martijn.katerbarg, NeedInfo)
Details
(Whiteboard: [ca-compliance] [ov-misissuance])
Preliminary Incident Report
Summary
On 2024-08-06, we received a Certificate Problem Report reporting a single certificate. The included subject:organizationName attribute value contained the HTML encoded version of the “&” character.
We are currently investigating this incident. A full incident report will be posted no later than 2024-08-16.
The reported certificate has been scheduled for revocation on 2024-08-11.
|
Assignee | |
Updated•2 months ago
|
|
||
Updated•2 months ago
|
|
|
Assignee | |
Comment 1•1 month ago
|
||
Incident Report
Summary
On 2024-08-06, we received a Certificate Problem Report (CPR) reporting a single certificate. The included subject:organizationName attribute value contained the HTML character entity “&” instead of the “&” character.
Impact
11 certificates issued between 2023-07-13 and 2024-03-05.
Timeline
All times are UTC.
2022-02-16:
- 16:04 We announce our completed deployment of an automatic pre-issuance check for Mojibake characters by utilizing ftfy in bug 1724458.
2024-08-06:
- 07:01 We receive a CPR reporting a single certificate.
- 08:43 We respond to the received report and start an internal investigation.
- 08:45 We confirm the certificate has been misissued.
- 09:40 We request a database report from our DBA team, listing all certificates containing possible HTML character entities within the subject:organizationName attribute value. The request asks for all certificates whose Subject names contain both an “&” and a “;” character to be reported.
- 11:06 We confirm the confirmed misissuance to the reporter.
- 11:11 An internal request is sent to one of our senior developers asking for information related to our implementation of ftfy, because during the original implementation it was discovered that ftfy also corrects HTML encoded strings.
- 11:16 We schedule a revocation event for the reported certificate. Revocation is scheduled for 2024-08-11 at 05:00 UTC.
- 11:25 Our developer confirms that the “&” string is indeed corrected by ftfy when called from the command line, but that our pre-issuance pipeline does not flag a linter warning.
- 11:27 We test pkimetal and find that it doesn’t catch “&” either. Since the pkimetal integration of ftfy is an enhanced version of the ftfy integration code in our pre-issuance pipeline, we hypothesise that the same root cause applies in both cases.
- 11:48 An update is implemented and pushed to the pkimetal integration of ftfy, which fixes the issue there. (Note that we do not yet use pkimetal for our pre-issuance linting, but we intend to switch to using it soon).
- 14:54 We receive the requested database report and review the listed certificates.
- 15:30 We confirm an additional 10 misissued certificates based on the database report. One of the certificates does not contain an HTML character entity, but rather a typographical error fitting bug 1910451. However, since the certificate was discovered during the investigation of the current bug, we are choosing to incorporate the certificate within this report.
- 15:37 A revocation event is created, scheduling revocation of the additional 10 discovered certificates on 2024-08-11 at 14:00 UTC.
2024-08-11:
- 05:06 The initial reported certificate is revoked.
- 14:06 The additional 10 discovered certificates are revoked.
2024-08-13:
- 12:19 We complete analysis of our current pre-issuance integration of ftfy, concluding that the root cause is in fact different to the issue already fixed in pkimetal.
Root Cause Analysis
Sectigo uses ftfy as a pre-issuance linter, to prevent Mojibake characters from being included in certificate subjects. When we integrated ftfy into our pre-issuance pipeline, we discovered that it also detects HTML character entities, which we noted as an added bonus and useful feature.
When entering the “&” string into ftfy, it will change the output, correctly, to “&”, since “&” is the HTML character entity that corresponds to the “&” character.
However, during the investigation for this incident we discovered that a backslash escape character is introduced by the code that integrates ftfy into our pre-issuance pipeline: specifically, “&” is turned into “&\;” by this function. Since “&” is not an HTML character entity, ftfy does not correct it. This escape character is introduced for every semicolon, so it affects all HTML character entities.
The original design and requirements for our ftfy-based pre-issuance linter was focused on preventing Mojibake characters from being included. As such, testing and QA of our original implementation found that this approach worked correctly for the purpose for which it was originally built - preventing the inclusion of Mojibake characters in certificate subject attribute values - and did not consider the bonus feature of detecting HTML character entities as a test case.
Lessons Learned
What went well
- The ftfy implementation has been highly effective in preventing Mojibake characters, which was the original motivator for the implementation.
What didn't go well
- When ftfy was originally integrated as a pre-issuance linter, we missed the opportunity to test that HTML character entities were detected.
Where we got lucky
- The number of affected certificates is low.
Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Deploy on-premise instances of pkimetal to our production CA environment. | Prevent | 2024-09-15 |
| Replace our current pre-issuance linter integration code with calls to the pkimetal REST API. | Prevent | 2024-09-15 |
Note: Since ftfy’s processing relies on heuristics, pkimetal returns a warning on ftfy findings and not an error. Our issuance system is built to halt issuance on all errors and on nearly all warnings. For other CAs looking to adopt pkimetal, this may be good to know.
Appendix
The crt.sh links below for the affected Certificates and Precertificates showcase crt.sh’s newly added integration with pkimetal. This demonstrates the efficacy of our proposed Action Items.
Details of affected certificates
Note: The markdown conversion within the bugzilla preview screen converted html character entities into their decoded equivalent. As such, we had to add escape characters in several places. We do not know how this will be translated into any outgoing bugzilla email. We consider the post as shown on bugzilla to be authoritative.
|
|
Assignee | |
Comment 2•1 month ago
|
||
UPDATED Details of affected certificates
Immediately after posting comment 1 we noticed an editorial mistake. The first certificate in the list had been inadvertently substituted for an unaffected certificate for the same domain. The correct entry is:
| Serial Number | Certificate | Precertificate |
|---|---|---|
| 009A72D69C80BA2FED5EC7755080524218 | Certificate | Precertificate |
|
|
Assignee | |
Comment 3•1 month ago
|
||
Ben, based on the due dates in our action items, we’d like to request a next-update for 2024-09-15.
|
|
||
Updated•1 month ago
|
|
|
Assignee | |
Comment 4•10 days ago
|
||
This is to announce we are still on track to complete our action items through a scheduled deployment this upcoming weekend.
|
|
Assignee | |
Comment 5•6 days ago
|
||
Our planned deployment last weekend was completed successfully. All remaining action items have now been completed.
Ben, since there have not been any questions or comments so far, we would like to request closing this bug.
|
|
||
Comment 6•5 days ago
|
||
I'll schedule this for closure on Friday, 20-Sept-2024, unless there are questions or issues still needing discussion.
Description
•