1943596 - HARICA: S/MIME certificate issuance with incorrect commonName

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Dimitris Zacharopoulos]

Steps to reproduce:

Preliminary Incident Report

Summary

On 2025-01-22, HARICA became aware that a flaw in a recently deployed S/MIME certificate issuance workflow, included the subject:organizationName value in the subject:commonName value of SV S/MIME Certificates instead of the combination of givenName and surname.

SV S/MIME certificate issuance was suspended until a fix was deployed on 2025-01-23.

All active affected certificates will be revoked in accordance with §4.9.1.1 of S/MIME BRs. The subscribers have been informed about the upcoming revocation and the need to issue replacement certificates.

A full incident report will be posted no later than 2025-01-31.

Comment 1

[Dimitris Zacharopoulos]

Incident Report

Summary

On 2025-01-22, HARICA became aware that a flaw in a recently deployed S/MIME SV certificate issuance workflow, included the organizationName value in the commonName subject value of SV S/MIME certificates instead of the combination of givenName and surname.

The team identified an error in recently committed code that copied the value of an incorrect attribute (organizationName) to the commonName attribute, instead of using the concatenation of givenName and surname attributes.

Impact

S/MIME certificate issuance was suspended until a fix was deployed on 2025-01-23.

HARICA searched through the logs and detected 68 mis-issued certificates of which 43 were non-expired, non-revoked, which were scheduled for revocation within 5 days.

Timeline

All timestamps in EET (GMT+2)

• 2024-12-15 New feature request was submitted, to implement code that allows for requesting multiple pre-validated S/MIME SV Certificates by authorized Enterprise administrators
• 2024-12-16 Started coding the feature and add-ons
• 2024-12-31 The code was merged to development branch for testing
• 2024-12-31 - 2024-01-07 The code was reviewed and tested in the intended feature.
• 2025-01-08 09:24 The feature was approved, and the tested code was merged to the production environment
• 2025-01-22 18:20 We received an email from a subscriber in our standard support email queue, who used the new “bulk Enterprise S/MIME SV certificate request” process and observed that the commonName subject field value in the produced SV S/MIME certificates contained the organizationName value instead of the concatenation of givenName and surname.
• 2025-01-22 18:51 The HARICA team reproduced the problematic behavior in the reported workflow and stopped the issuance of SV S/MIME certificates.
• 2025-01-23 22:22 A patch was deployed that fixed the bug.
• 2025-01-23 22:58 SV S/MIME certificate issuance resumed.
• 2025-01-24 The Subscribers of active and affected certificates were informed about the upcoming revocation.
• 2025-01-24 16:42 Preliminary incident report was submitted to Bugzilla.
• 2025-01-27 18:20 Deadline for the revocation of affected certificates.
• 2025-01-31 09:28 Discovered that some of the affected certificates were not revoked. A separate delayed revocation bug will be filed.
• 2025-01-31 12:12 All remaining affected certificates were revoked.

Root Cause Analysis

Here we present the results of the “5 whys” root cause methodology that was followed:

1. Why did HARICA issue SV S/MIME certificates with the Organization as the CommonName?

Because there was a flaw in the Bulk S/MIME certificate issuance workflow that included the organizationName value in the commonName subject value of SV S/MIME certificates instead of the combination of givenName and surname.

2. Why was there a flaw in the Bulk S/MIME certificates issuance workflow?

The RA development team used the wrong elements from the already-validated information to add to the commonName, but mainly because the review/testing phase didn’t catch the issue.

3. Why didn’t the review/testing phase catch the issue?

Because, with the dummy values used to test the workflow, it was difficult for the reviewer to notice the bug.

4. Why was it difficult for the tester to notice the bug?

Because the CSV test vectors included random names like "test1" and "test2" that were similar to the organization name “test”.

5. Why did the CSV test vectors include random names?

Because test vectors with realistic names and organization names had not been prepared and were not an explicit part of the test process.

Lessons Learned

What went well

• HARICA quickly identified the affected certificates

What didn't go well

• Code review didn’t catch the bug
• QA didn’t see the problematic common names in the certificates
• Not all affected certificates were revoked in the pre-determined timeline

Where we got lucky

• The bug infected only bulk issued SV S/MIME certificates
• The new workflow was recently deployed so the problem was quickly detected and solved

Action Items

Action Item	Kind	Due Date
Patch code base to fix bug	Prevent	Already implemented
Add realistic test vectors to the test process like “John Doe”, “Jane Doe”, “Example Company”, “ACME Corp”, “Test Legal Entity”	Prevent	2025-02-14
Implement Unit test for SV S/MIME certificates that checks that the organization name is not part of the common name	Mitigate	2025-04-08
Implement Unit test for SV S/MIME certificates that checks that common name includes the givenName and surname	Mitigate	2025-04-08

Appendix

Details of affected certificates

Serial, SHA-256 fingerprint
220385BBA7F9300EF2EFCEC3BC639D9F,6E54254E3A029C378AD2754B9B0E0E41C6E81D18F66DB22FFA317ED54C3AA30C
4425F43E0630691141CB822D71F0AED9,9A1923BAD348BCB019F0A1B85E5E25E7C23259FFBDCDB06E1F9BF4241A64A370
3204D1C5EC18D72E10EBACF39E3FCE0D,D8EE9DE017245832150B7A60B1ACAD7319C392EBB0E62C43BA1204D2EF52387A
7310758EF762272EE474342744D0EC7B,73A35DD319A36A2B99E4D23D077D857D320AEAB120112F9E74EC051AFF6747AD
154371260896DF0FDD33F7B640A77888,B57104C755ACB8368818EF4FBB87C8B537059DACE30A3E8CC5B1B15F9A6B88C6
08FD07F27A2ECF2BDB7649B1FE3185BF,55793C73CF907274A8ECD5566566DD239086E074EE1DE52837B3FF17A789B5C5
6BBF2D017FE1B92095032D81CF48FBCF,E4630CE8D04B6653CA10BF7231894A3E71F7C0896684B882EFA0AC5FDE63225F
5865CBF6CEDDF1E4FE17B3555BB95B2E,4BD3443AE78E508DC6B4C2AC69A8E702126E5F0F3507367DC947B87FD1D731EF
3E2EA1E1309A4A4575E5E13D11322914,646790B3386489268498BB3F7E223114BF971688661F2F2DFD6A98C8D7A2BB22
1735B41434CA5625826159C80CC6244F,A94C805B558EDBC77835352135671EE784357C7222549FAE5E94BCF9EC2497C4
78841F189F0583098B8983CFAD2E57DD,14A884E772B8AB2E534DE737F089FEF16501057EC98DE05A82062D0B96835253
13F5D1F98D20D8917803AD313BE2AEBB,9D8A9F3FA90710ED4BF5F63B50DC9E8DB675AC8944C39AB1B994A91803E0C541
3D79D7E55AA178A506ABEF2BD4E40E28,9DD7F72103AFCC4099A256CBFFA8C91D5C4A01628A9892C217429EF760206F89
49917FA4976174A927282ABFE98F77EB,6F8B41D09996C3B84A3FD1ABF36751A2E544C581D96A25F20BDB78D547443B09
4424A6B1B96FEBC6F4279590C5F19B74,AD57B9F2F8FC948248280C84DE7AC71280ED5E2C2DD48D808A9F0F9A4E6CE41F
7BEAF31156B253AB07BAFB877DBB3BBE,5B94B83EA24ECE9906581351886C6BD41D05A99432228CF5027697F70C6C74E7
233B9A91A47754A673C5AE7778465C22,84333324D4136396B568EA8F75E726DEB27855EC33413DD3ADF45FBC861075A6
5007257E3133C308AAE6B3AA45785A42,4AD4A14618C4D607EC35843CA9F95A55A63CB796FC82D6C0E0A8C6280DC0F5C9
0EAE3D2FF9BF399DF1806D8660EA4947,435B2F2FF3D5188749B4414EC08C29A098214DD75FC666C821C89879EAAC4B26
418B4AB2D6EEA5A95180F2FAF51EFCB9,1EFE9EA7993AE13C3A3A743446A25C4AE9A1AACB00F059B97B40C2123224A81A
7E52BCBAEFE942832A816FED3D8E169B,D9A44FAA5BF64DF04C31BAB2E332EBC75C1A6AE3F78F898FA0B35945EA2F70B6
1C8B3010F4FE354D1C0E61920E99F140,8DC9C6D7B0D3A386000DEC2AF596A56CE2807917EAB6A03A54005D9F545BA2C8
602F2551F8FEEC2BF4E4FD1BA3F273F8,63C728AC293881117227AA88F368603C636882298E3776C6E0EDAE3EA5E10E5C
618D4B081D73CE1B65AF0F833E1A47A6,D41597B98BB84CEEC01979F5920EE11BB9F518452153B5468AC51668F9C6A5CF
5E4153C598C9CCC7574CFE395FD31044,CE9D290C25EEBB515E39559E1390A4108D74513C65AF8074C9480C6F3C8ACBE7
30D184816E74E430FD2B178E388A1C54,6BE66FCC02895185775D67C88D10ABE4201D80491A1F2C7B1FFF6DDB232FBFE0
4B60E9929EBC95FFCD7804236BBB2B5C,A725832E81728651BE7E344194B01424D8E1AC84082D48747B3756BBA73387FC
0AD281D0A1B4A2E03F48892E932A7A6,D391358E6DC0E3EA089323215B7AD5A25298D8BE4364F75F543035FA953990FA
751AB6B25656762B4A19E77B4426C59F,6AD7DFDC90424CEEA9725412200E8A3176447A2D6EF02B0FC81E7F005ADB7EF3
5F449E90C9EA87F93B94C9AC7ECEA659,A903B7E16AD19A2D8F9A03F928E4B6343BA6FFC91829C6319205021A9D40C28F
0A792031E5C4A4EBE3CE66BFC5E4A199,36BB511735E3A28BEB49CD926E4931C4F6A71A13333C64276BA9CA2CCB4FBF66
5B7EEDD689CAF36FCE02042EEB806F62,8AD14EF6C4D5DD140AC009E1E013E9365BBEED18F964B6506E105E2210632B6D
26B5A82EBF0C11E106F51767041021FD,8CF56378820B418B78BB41343B5334868EEA9B3F8F60090A52105E7E761C734B
55B1F5BA7E7BA00EE2CFA70C5FB9C87,FE9F09632823B2925E21B71CFBC1988EA6F3AAEB207A11830FC70CF977C73D9F
2ACCB8D152E226F4E2BBB2E604A384E0,B7941EDF9D8D1B0F5C70F7A64BEEE4F4978DB329CA44599C942EBEB2D4917E1A
52A694732F979C42938E98C0D26D5361,19916B0048FC6C3C0A3BEF2BDC078CBB51AF2C55344F739D6E5D676AC773FC9C
788C8CB640D13F26B66816A9978099D3,08CC06D2DCD33AA512A9EA7D77F25B0C9098411B7F77E1A89409BECCAE74D0DE
78058E42C1B922DADAA43A5E8CD24FA8,A48C17E47C50140A409E29F8BDA9712DF7B9ADCC729CE15D3FBD87416B7AFF1C
535C97622936ACF4A4A9C32489606882,B811082DE00223B7AE3DCFE9A8996B4F08D89860CF27887B3A73DB558B02E353
54928B5D1E163EB39351987B31CF00D9,39CB0AE0E9B1F566FE9B1C586D53C3653939E10134A6046841D61111F4864849
2596AA6EE7B9282E0E87051C740AEEEB,407667F44286714D2167B384DBA235D035C56015E645C81C22F3ED3038D1816E
709EB4F75CFAAD72826700719D6DBB04,1F2FAFAE6054EEDA57200322B87079F284D01A3F0BAC57C30362EF035DF8F404
4376DC37CE6AA031E1B5E9D4415B0484,82EFCED2E0AF4A0B66556CAFA5B1A467EDE930E0A1688793EEBCF46BEC98A5FD
7D08764BBB7DE98FAA536E128E34EBB,3AA094CBBD4BDC47F626210702EF6125A98CC6587B9ECFF3F3F74C52F25FFE0C
5E816D4C1AD4D9600D1A9DC8D1FCD51E,83D7D8F0CED74BCBADB743A3A596FDE70435C87E3A9DECF6B37A262699E1EB63
64622C03943F5C9D28583D99811BF693,EDBD3D9ECA2CB0DED04A709E4B63F2224C44FB904A0326BE76F91C6CCCFD6573
13666C611DB8D1058E45B34D57A238FE,FCEADF30BB5DFB1000C9989A575EEA72E8D9DD9BF56A795A317A628C84CCF25A
22EE43093F56C033584E563283A1C2EE,1192D83949356829AB2DE0B1230FDBA906361CAB8A2326DF4FD3644C83818EF2
6C1DABDBA539ACD88BF111765E889C8D,B986241909C280B09D7A7C1F8BD59B2817E1837161E2DCD313D3117C5A5D0513
471B7D127F4AB9EBAD4ACC3F19C37830,AC2794D6507C0CB3207F46D7ED0A515A426630C3E2E4867982A9948AE0341AD5
7DE55775FCB34B0BF475DC0309F368C9,26DA0DF33943AD8D588CDC3F4CD901ADC5D7774E39567D9315D1ABDB93A98076
36F1E45BD75AF26D037C36177CF3ECE8,E44104C624FDDC96D9009DF91CED43FC9148A4FA3D1995388C9397444560CE8B
4F36D2A01580D955FDD95C6D0E7FFB21,ABB54A26CE491DFF90784DD8B5BB2B14E40860B55C8C5C0FCB92EF97456CDE18
73F7F46F54DAFC4DF89D2A818A025FBC,924B346D419D8FA2475C0E9DC1DCD2E0E281F265B000CFE5451EA9059CBAE24B
6C4A3E70893F86F93913EEEAC8C48CE5,87EB53E5DFA25CD0C80D13D2A3BF9B9D7320A4C04E100C1371E5AE54BCD18047
3AEE0829E8AFD69AB107598EDB22891F,3020510F10369B3C09595789425E877D6AD4D2E35369ED0601A708791AD30FFC
727B50FD6CFD039EBDDE0170DBA8B9FE,B93CC2875C2C937880F790A0056D9DC49D76CA3DEF3EF9F0A86CF4FAB2445536
7FD0697BCEBE4462AB61E417FE36FD2D,CE16DF8D809E6ED9AF9D132CBB7A502F685AA1202905F2DA98BDE83E10E3B62C
0EA634CBF8043564EACB2D45BD2EF7F9,9B24033583023974AA490D97344E87F014383A6C1AEB35C6A1D039BC80A6D46F
0C9269AD8B5D6EE591DF4302FFF7D430,3502EE88747F81BD70C1C15AFFC279DE09F02A5C6582B0E763DE09AC4470D798
24F9BA60E83EB04A80C28EA5233049DA,E42EBF5C46C786BFEE85B8F772C73245708DE9F8BDC52393097D26983F63122E
66465BF21B87680A28A9CFCBAC4D5075,C36221A1A16D116E4BFFFDF5980E35696AEECE45AB6561D3F16273C0FDBFBE63
2163B6BCEF0B0FA27E4BF6A3DAD02CF8,BC74678788CBF057F946E8AE9B1A35AE38643F4EDFB772A7DA6F66774045C5D3
644B3A88DBB3A3EB1BF2D8226B6AC4F2,E087846233D615CCCA0256E45236CAEA8B0C23E21391148B67BC66AB32056339
5045C26624284F73A822A9399AD6A00B,CB9EC95B0FFD1C47E8C84E6498B359DB5809A01F119A5B6ED2257DE2B5D14EE8
69A5978BCDD6FDCEDC8E060DD79164CF,B7C287CECC59681BA532D0AA4E979C0E594D1246B862F574306716EF59B03691
6802B79D5ACA294392361CB7D7A96D92,E5E11E31C9D15EA1B379F5626BED9F8E45F6A56932B214F6EA17A84F8D6602FB
6A5D7C80FE06E0CF465145E894284BAC,772B341E88A470E5F5F6E369BC6C59A240108CB643A8B8709E39815B7BA87A11

Based on Incident Reporting Template v. 2.0

Comment 2

[Dimitris Zacharopoulos]

We are preparing an internal document describing that testing must include realistic values. This document will provide example values for at least the following:

• Domain Names
• Email Addresses
• Base Domain Names
• givenName, surname
• countryName
• localityName
• stateOrProvinceName
• organizationName
• serialNumber
• organizationIdentifier
• JOI

Comment 3

[Dimitris Zacharopoulos]

The internal documentation with realistic testing values for natural persons, legal entities, domain names and email addresses, is completed.

Action Items

Action Item	Kind	Due Date
Patch code base to fix bug	Prevent	Already implemented
Add realistic test vectors to the test process like “John Doe”, “Jane Doe”, “Example Company”, “ACME Corp”, “Test Legal Entity”	Prevent	Completed 2025-02-14
Implement Unit test for SV S/MIME certificates that checks that the organization name is not part of the common name	Mitigate	2025-04-08
Implement Unit test for SV S/MIME certificates that checks that common name includes the givenName and surname	Mitigate	2025-04-08

If there are no further questions or concerns with the incident report, we would like to kindly ask the Next Update to be set for 2025-03-14 so we can provide an update on the progress. If there is meaningful progress before that date, we will post an update sooner.

Comment 4

[Dimitris Zacharopoulos]

Status update:

• On the “Implement Unit test for SV S/MIME certificates that checks that the organization name is not part of the common name”, we are preparing two unit tests to check that “organization name is not part of the common name”.

• On the “Implement Unit test for SV S/MIME certificates that checks that common name includes the givenName and surname”, we are preparing two unit tests to check that “common name includes the givenName and surname”

We are on track with the proposed timeline, and we kindly request setting the next update to 2025-04-01.

Comment 5

[Dimitris Zacharopoulos]

Here is a status update for the pending actions.

Action Items

Action Item	Kind	Due Date
Patch code base to fix bug	Prevent	Already implemented
Add realistic test vectors to the test process like “John Doe”, “Jane Doe”, “Example Company”, “ACME Corp”, “Test Legal Entity”	Prevent	Completed 2025-02-14
Implement Unit test for SV S/MIME certificates that checks that the organization name is not part of the common name	Mitigate	Completed 2025-03-21
Implement Unit test for SV S/MIME certificates that checks that common name includes the givenName and surname	Mitigate	Completed 2025-03-21

All action items have been completed. Please let us know if there are any additional questions or concerns related to this bug.

Comment 6

[Ben Wilson]

Does anyone have any follow-up questions or requests? If not, then HARICA should file an Incident Closure Summary and request that this bug be closed.

Comment 7

[Dimitris Zacharopoulos]

Incident Report Closure Summary

Incident Description

On January 22, 2025, HARICA received an email from a subscriber in our standard support email queue, who used the new “bulk Enterprise S/MIME SV certificate request” process and observed that the commonName subject field value in the produced SV S/MIME certificates contained the organizationName value instead of the concatenation of givenName and surname.

Incident Root Cause(s)

The RA development team used the wrong elements from the already-validated information to add to the commonName, but mainly because the review/testing phase didn’t catch the issue.

Remediation Description

HARICA revoked 8 affected certificates immediately and contacted Subscribers of the remaining 35 certificates to arrange revocation by January 27, 2025. All certificates were subsequently revoked (note: delay in revocation of the remaining 35 certificates is separately addressed in Bug 1945389). HARICA also:

• Implemented additional Unit tests for SV S/MIME certificates
• Added realistic test vectors to the test process

Commitment Summary

HARICA will ensure that validation methods are tested and verified before deployment. We will continue using unit tests and realistic test vectors to catch configuration or implementation issues early in the process.

All action items have been completed as described, and we respectfully request closure of this incident.

Comment 8

[Ben Wilson]

I'll close this next Wed. 23-Apr-2025, unless there are questions or issues to discuss.