SwissSign: MPKI step-up process sets wrong JoI Locality
Categories
(CA Program :: CA Certificate Compliance, task)
Tracking
(Not tracked)
People
(Reporter: sandy.balzer, Assigned: sandy.balzer)
Details
(Whiteboard: [ca-compliance] [ev-misissuance] Next update 2024-06-10)
Incident Report
Summary
While performing the walkthrough of the MPKI step-up process during the yearly external Audit, we have issued 8 EV certificates (4 pre and 4 leaf certificates) with wrong JOI locality fields.
This violates EV Guideline 7.1.4.2.4 "Jurisdiction of Incorporation or Registration Field", because the Registration Agency is on state level, the jurisdiction must not include locality information.
The walkthrough was performed between the 24th and the 26th of April.
This mis-issuance was discovered during today's (29.04.2024) Audit review session.
Impact
We assign a low risk to the ecosystem by these mis-issued certificates because the process that resulted in the mis issuance has not been used so far and the certificates are revoked.
Pre and leaf certificate
https://crt.sh/?id=12833464224
https://crt.sh/?id=12833464313
https://crt.sh/?id=12833417306
https://crt.sh/?id=12833417305
https://crt.sh/?id=12833382160
https://crt.sh/?id=12833382159
https://crt.sh/?id=12859109894
https://crt.sh/?id=12859109893
Serial Numbers:
039BB49B6FAB28E4ACB25D2B17829D6F064E1AC4
3796F321A1B9FB09B66E0D13A749E4227EFE4DE9
2ED0BA1C4278B7F6B45427B19D4D95CFF2BEE3FE
79AE68135264728C2986B0B572347CFFD62E9A72
Timeline
All times are UTC.
2024-03-04
Guidelines for the Issuance and Management of Extended Validation Certificates Version 1.8.1 have been released
2024-04-24
Walkthrough of the MPKI step-up process and issuance and revocation of the EV certificates
08:58 UTC First mis-issuance
2024-04-26
09:46 UTC Last mis-issuance during issuance and revocations of EV certificates
2024-04-29
10:50 UTC Review of the certificates in the audit session and discovery of mis-issuance
17:00 UTC Post of this Bugzilla
Root Cause Analysis
We have an audited process for creating new DV/OV/EV MPKIs. For the creation of EV MPKI, the UI correctly prohibits the setting of JOI Location if the organization belongs to a jurisdiction on State level.
During adaption of this existing code for the step-up process the check for the JOI location on state level was also copied but the UI was not correctly prohibiting entry of a JOI location and the code in the backend also didn't check for this case.
During the audit of this step-up process it was overlooked that the organization was registered on state level and thus the UI should have prohibited the user from entering a JOI location. Additionally, the backend also didn't reject this.
The mis-issuance was detected in today's audit session when all certificates issued during the various walkthroughs were reviewed. The affected 8 certificates were already revoked last week during the session because revocation was also part of the audit. The certificates were all for our SwissSign MPKI.
Lessons Learned
What went well
- It was discovered before the step-up process was used.
What didn't go well
- Our test-cases for the step-up process did not include this constellation
Where we got lucky
Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| correction of error in the step-up process | Prevent | to be defined |
| review test-coverage to include all test-cases of existing code parts if they are re-used in new process | Prevent | to be defined |
Appendix
Details of affected certificates
see above in section Impact
While performing the walkthrough of the MPKI step-up process during the yearly external Audit, we have issued 8 EV certificates (4 pre and 4 leaf certificates) with wrong JOI locality fields.
So to understand this correctly, this wasn't to setup a new MPKI stack but to show the process to the auditors?
Was this MPKI process tested in a test environment before testing it in a production environment?
Additionally, the backend also didn't reject this.
There are several incidents on Bugzilla regarding the various fields in the Subject field of a certificate. Do you have a triage log of these incidents, and why this wasn't caught earlier?
Have you done a review of all of your issued certificates to ensure you've not made this mistake before? I did find https://crt.sh/?q=ce6233646aa2c0af95b919899117b20467e73699e3e90314f8e5dea4ba6961cb Is this also impacted by this same problem, but last year?
|
||
Updated•2 months ago
|
|
Assignee | |
Comment 2•2 months ago
|
||
Hi Amir, thank you for your questions, please find our answers below:
"So to understand this correctly, this wasn't to setup a new MPKI stack but to show the process to the auditors?"
Correct.
"Was this MPKI process tested in a test environment before testing it in a production environment?"
Yes it was but we didn't have a test case with Juridiction on state level.
We will of course include such test cases from now on.
"There are several incidents on Bugzilla regarding the various fields in the Subject field of a certificate. Do you have a triage log of these incidents, and why this wasn't caught earlier?"
We monitor all Bugzillas for improvments, potential problems, learnings which could be applicable to identify points where we can improve our processes. The recent Bugzillas regarding various certificate fields were checked against our systems and certificate profiles and we didn't detect any issues. There was no Bugzilla with a similar issue to the one we detected now and even if there had been, because the step-up process was not used, we wouldn't have found the problem based on previous Bugzillas.
We believe that the root cause was really incomplete tests and not missed learnings from Bugzillas.
"Have you done a review of all of your issued certificates to ensure you've not made this mistake before? I did find https://crt.sh/?q=ce6233646aa2c0af95b919899117b20467e73699e3e90314f8e5dea4ba6961cb Is this also impacted by this same problem, but last year?"
We did an initial review yesterday and need to investigate more. We will report back.
|
|
Assignee | |
Comment 3•1 month ago
|
||
Incident Report
Summary
We have found below listed additional 10 (5 pre and 5 leaf certificates) no longer valid EV certificates, including the one mentioned by Amir.
This were revoked by a previous Bugzilla:
https://bugzilla.mozilla.org/show_bug.cgi?id=1860750
Impact
We still assign a low risk to the ecosystem by these mis-issued certificates because the certificates are already revoked.
Leaf and pre certificate links
https://crt.sh/?q=83552D1BF9BEE82792EF6F6FD3B428DAB4015E6394D7C4CEC2741D7CE4541ADC
https://crt.sh/?q=0CE0B409C2F33FF91F82A2D40667E4F7C7F464BA00761899964660334252EF51
https://crt.sh/?q=35DA2A713CE8AFF6794697290652F36B4AA2D12E07A19150C0288E2AC30DD90F
https://crt.sh/?q=210D0787A1EE8A82F97D9E041236FDB626EA933387733960016F677A9EFC5C01
https://crt.sh/?q=C32E903A575755B6B9DF9841A20517D397FB8AF75C69E158E49F11794E180702
https://crt.sh/?q=0B0D41217A0AB8EA7D9E98185D72B1C9EE557B02C854E36B82A13FFC88A7D3EF
https://crt.sh/?q=E9127D45FAC07670CE22E968C8E17401C8869D18D7B7E01BA07813CE4980FFF5
https://crt.sh/?q=A40B683A1DA64243C0D2F7DE65C7DAA4501054D04700B0B92975E6FCE27B9F9B
https://crt.sh/?q=ce6233646aa2c0af95b919899117b20467e73699e3e90314f8e5dea4ba6961cb
https://crt.sh/?q=C67695EF9D56201046D3E02B43E0552518688F15416620AF7DE061EAFE6DF74E
https://crt.sh/?q=60CE72FD3E3F2390815817E17BD870827EAA4A5AB25B84D8A8284C32842F709C
https://crt.sh/?q=86F0BE48135F704357F22B7C98CBA0563378734AC1752D66B9ABBB2CF4F924E7
https://crt.sh/?q=2A64897392B12D8B774A9902A4DC45FE1873CEA1449924DB0E6C0596AE15C8D8
https://crt.sh/?q=EAF34CAFA4D0BDBB3C09ED97B8C8CDE5ED500FA08A64F6C437492C9E8D088309
https://crt.sh/?q=672F2C25C1C999E723F004C70F85B8AAEF7DF9C6CD155E757BE41DC122012415
https://crt.sh/?q=A561B83C5B62CC0B8138E598F7762F77A07DA79E73F62BC05250044229EA257D
https://crt.sh/?q=E9127D45FAC07670CE22E968C8E17401C8869D18D7B7E01BA07813CE4980FFF5
https://crt.sh/?q=A40B683A1DA64243C0D2F7DE65C7DAA4501054D04700B0B92975E6FCE27B9F9B
Serial Numbers:
039BB49B6FAB28E4ACB25D2B17829D6F064E1AC4
3796F321A1B9FB09B66E0D13A749E4227EFE4DE9
2ED0BA1C4278B7F6B45427B19D4D95CFF2BEE3FE
79AE68135264728C2986B0B572347CFFD62E9A72
59E14FFA80509465FA24BE6BCEB891B5CA144C0E
7F8742DB161D2DE6D8AEF40C3948748EC81E60F6
6418D42AC7A18FF22E65DC0FA76F5DCBC7F4F039
0733874D7BE5B7831704F6FB78D7660D71B6D2BA
0A810BC76A7500E219B926DE2E62E089FA7102B8
Timeline
2024-03-04
Guidelines for the Issuance and Management of Extended Validation Certificates Version 1.8.1 have been released
2024-04-24
Walkthrough of the MPKI step-up process and issuance and revocation of the EV certificates
08:58 UTC First mis-issuance
2024-04-26
09:46 UTC Last mis-issuance during issuance and revocations of EV certificates
2024-04-29
10:50 UTC Review of the certificates in the audit session and discovery of mis-issuance
17:00 UTC Post of this Bugzilla
2024-04-29
17:30 UTC (10:30 PDT) comment in Bugzilla form Amir pointing to another affected certificate
2024-04-30
06:00 UTC start of an further internal Analysis and check why this certificate was missed in initial report
2024-05-01
Public Holiday
2024-05-03
14:00 UTC investigation shows further already revoked certs
18:30 UTC update of this Bugzilla
Root Cause Analysis
The additional Certificate from Amir's input was created during last years Audit in the same process and not discovered by us.
The additional 4 certificates were revoked by another Bugzilla as stated above and are still being investigated.
|
||
Comment 4•1 month ago
|
||
(In reply to Sandy Balzer from comment #3)
Impact
We still assign a low risk to the ecosystem by these mis-issued certificates because the certificates are already revoked.
The impact section is meant to provide readers info on the number and type of things affected. It's not for the CA to try to downplay the incident. Instead, you should answer the questions in the CCADB incident report template.
The Impact section should contain a short description of the size and nature of the incident. For example: how many certificates, OCSP responses, or CRLs were affected; whether the affected objects share features (such as issuance time, signature algorithm, or validation type); and whether the CA Owner had to cease issuance during the incident.
You said that these were additional and you listed 9 serial numbers. Does that mean there was 9 leaf certificates affected in total? Did you cease issuance during the incident?
|
||
Comment 5•1 month ago
|
||
(In reply to Mathew Hodson from comment #4)
Dear Mathew,
Thanks for your input and questions!
This post contains both answers to your questions as well as updated sections reflecting our current status in investigating this error.
(In reply to Sandy Balzer from comment #3)
Impact
We still assign a low risk to the ecosystem by these mis-issued certificates because the certificates are already revoked.
The impact section is meant to provide readers info on the number and type of things affected. It's not for the CA to try to downplay the incident. Instead, you should answer the questions in the CCADB incident report template.
We appologize for not following the incident template correctly. Below an improved version of the section in question:
Impact
We initially identified 8 certificates (4 pre and 4 leaf) that were mis-issued. As it was immediately clear that they were issued during the review of the new step-up process which is not in productive use, we did not stop certificate issuance. We requested our operations team to check for other certificates with this error. They did not identify any.
After Amir reported one more mis-issued certificate we investigated why we didn't find this.
We realized that our operations team misinterpreted our query and only checked for valid (not expired and not revoked) certificate with this error.
After checking with the correct parameters, we can now confirm mis-issuance of:
- 8 certificates (4 pre and 4 leaf) as initially reported
- 2 certificates (1 pre and 1 leaf) reported by Amir
- 8 certificates (4 pre and 4 leaf) additionally found by correct internal checking
This gives a total of 18 mis-issued certificates. All of which are already revoked.
All the crt.sh links and serial numbers have been listed in the previous post and for brevity sake, we're not duplicating them here.
You said that these were additional and you listed 9 serial numbers. Does that mean there was 9 leaf certificates affected in total? Did you cease issuance during the incident?
Correct. There were 9 leaf certificates mis-issued. We did not cease issuance as the process that produced them was not cleared for productive use.
Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Correction of error in the step-up process | Prevent | to be defined |
| Review test-coverage to include all test-cases of existing code parts if they are re-used in new process | Prevent | to be defined |
| Review of our internal bugzilla template to better align with the desired content of the required sections | Mitigate | done |
| Clarification of work-instruction from Compliance team to Operations team to explicitly include expired and revoked certificates when searching for other possible mis-issued certificates | Mitigate | 10.05.2024 |
|
|
||
Comment 6•1 month ago
|
||
Update 2024-05-10
Our analysis of the incident is now complete. The results of the analysis impact the timeline, the root cause analysis and the action items. The impact is unchanged and remains as stated in Comment 5.
Timeline
All times are UTC.
2022-08-05
Our new CA platform is approved by audit body to go into productive use. This marks the start of the migration of existing customers from the legacy platform to the new platform.
2023-05-12
Customer X is migrated. JOILocality field is erroneously set.
2023-07-05
In preparation for S/MIME BR 1.0 a change is implemented to set a correct Organization Identifier for all customers who get S/MIME certificates from us.
2023-09-01
S/MIME BR 1.0 becomes effective
2024-03-04
Guidelines for the Issuance and Management of Extended Validation Certificates Version 1.8.1 have been released
2024-04-24
Walkthrough of the MPKI step-up process and issuance and revocation of the EV certificates
08:58 UTC First mis-issuance
2024-04-26
09:46 UTC Last mis-issuance during issuance and revocations of EV certificates
2024-04-29
10:50 UTC Review of the certificates in the audit session and discovery of mis-issuance. Decision to not cease issuing certificates as the step-up proecess was not cleared for productive use.
17:00 UTC Post of this Bugzilla
17:30 UTC Amir reports additional mis-issued certificates (1 pre, 1 leaf)
2024-05-03
18:30 UTC SwissSign reports an additional 10 mis-issued certificates (5 pre, 5 leaf)
2024-05-10
Update of the Bugzilla with extended timeline, root cause analysis and action items
Root Cause Analysis
We have now identified all the affected certificates. The root cause for the mis-issuances can be broken down as follows:
- 8 initially reported mis-issued certificates: see Root Cause Analysis at initial posting above.
- 2 reported by Amir mis-issued certificates: see Comment 3 above.
- 8 additionally found mis-issued certificates:
During the migration from the legacy CA platform in Q1-Q3 2023, a semi-automated process was used. The process contained review of migrated customers by RAO 1 and RAO 2. It was probably due to high workload that both RAO did not spot the wrongly set JOILocality field (all other migrations were done properly). After the customer was migrated, they issued EV certificates in their regular operation of the MPKI.
On 2023-07-05 we implemented a change in preparation for the S/MIME BR 1.0 regarding organization identifier. This change accidentally corrected the JOILocality field for the customer. At this point, we should have had a process to catch these "accidental" changes. EV certificates issued by the customer after this date do not contain the JOILocality field.
Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Correction of error in the step-up process | Prevent | 2024-05-24 |
| Review test-coverage to include all test-cases of existing code parts if they are re-used in new process | Prevent | to be defined |
| Review of our internal Bugzilla template to better align with the desired content of the required sections | Mitigate | done |
| Clarification of work-instruction from Compliance team to Operations team to explicitly include expired and revoked certificates when searching for other possible mis-issued certificates | Mitigate | done |
| Enhance procedure for automated changes to customer data: Always do a dry-run and review all changed fields to detect "accidental" changes | Prevent | 2024-06-15 |
Kind regards
Roman
|
|
Assignee | |
Comment 7•1 month ago
|
||
Update 2024-05-17
The correction of the error in the step-up process has been completed. The process will not be handed over to production before it has been approved by our Audit body.
Timeline
All times are UTC.
2022-08-05
Our new CA platform is approved by audit body to go into productive use. This marks the start of the migration of existing customers from the legacy platform to the new platform.
2023-05-12
Customer X is migrated. JOILocality field is erroneously set.
2023-07-05
In preparation for S/MIME BR 1.0 a change is implemented to set a correct Organization Identifier for all customers who get S/MIME certificates from us.
2023-09-01
S/MIME BR 1.0 becomes effective
2024-03-04
Guidelines for the Issuance and Management of Extended Validation Certificates Version 1.8.1 have been released
2024-04-24
Walkthrough of the MPKI step-up process and issuance and revocation of the EV certificates
08:58 UTC First mis-issuance
2024-04-26
09:46 UTC Last mis-issuance during issuance and revocations of EV certificates
2024-04-29
10:50 UTC Review of the certificates in the audit session and discovery of mis-issuance. Decision to not cease issuing certificates as the step-up process was not cleared for productive use.
17:00 UTC Post of this Bugzilla
17:30 UTC Amir reports additional mis-issued certificates (1 pre, 1 leaf)
2024-05-03
18:30 UTC SwissSign reports an additional 10 mis-issued certificates (5 pre, 5 leaf)
2024-05-10
Update of the Bugzilla with extended timeline, root cause analysis and action items
2024-05-17
Implementation of correction of error in the step-up process
2024-05-17
Update of Bugzilla with info on implemented prevention
Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Correction of error in the step-up process | Prevent | done |
| Review test-coverage to include all test-cases of existing code parts if they are re-used in new process | Prevent | to be defined |
| Review of our internal Bugzilla template to better align with the desired content of the required sections | Mitigate | done |
| Clarification of work-instruction from Compliance team to Operations team to explicitly include expired and revoked certificates when searching for other possible mis-issued certificates | Mitigate | done |
| Enhance procedure for automated change to customer data: Always do a dry-run and review all changed fields to detect "accidental" changes | Prevent | 2024-06-15 |
Kind regards
Sandy
|
|
Assignee | |
Comment 8•23 days ago
|
||
Update 2024-05-24
No Update this week.
Kind regards
Sandy
|
|
||
Updated•23 days ago
|
|
|
Assignee | |
Comment 9•9 days ago
|
||
Update
We have enhanced our procedure for automated changes to customer data. Now, we always conduct a dry-run and review all changed fields to detect any accidental changes.
This includes reviewing the test coverage to ensure all test cases of existing code parts are included if they are reused in the new process.
Action Items
| Action Item | Kind | Due Date |
|---|---|---|
| Correction of error in the step-up process | Prevent | done |
| Review test-coverage to include all test-cases of existing code parts if they are re-used in new process | Prevent | done |
| Review of our internal Bugzilla template to better align with the desired content of the required sections | Mitigate | done |
| Clarification of work-instruction from Compliance team to Operations team to explicitly include expired and revoked certificates when searching for other possible mis-issued certificates | Mitigate | done |
| Enhance procedure for automated change to customer data: Always do a dry-run and review all changed fields to detect "accidental" changes | Prevent | done |
All action items have been implemented. We are monitoring this Bugzilla for any further community feedback or questions.
Kind regards
Sandy
|
|
Assignee | |
Comment 10•2 days ago
|
||
Update 2024-06-14
With the last posting, all open Action Items are done.
Unless there are further questions, we kindly request this Bugzilla to be closed.
Description
•