1608333 - CFCA: Wrong OrganizationName

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Oliver Bi]

Attachment: Bugzilla Report - wrong organizationName.pdf

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0

Steps to reproduce:

CFCA issued two wrong certificates ( for testing purposes ) with the OrganizationName.
During the test procedure, our test engineer issued two wrong certificates with the OrganizationName , the first one issued on Jan 9 06:56:00 (UTC) 2020 and the second one issued on Jan 9 06:56:59 (UTC) 2020,the two certificates has been revoked on 07:19:53 (UTC) and 08:50:22 (UTC).

(1) We have revoked the wrong certificates immediately.
(2) Test engineer permissions had been restricted to the minimum scope after this incident.
(3) More professional auditors will join in the next testing process.

Actual results:

The wrong certificates are issued by our test engineer, he issued two error certificates inadvertently while on a test mission(the mission includes certificate application, revocation and update ). According to the test case, he revoked the second certificate by himself. About one hour later, our auditor found the problem, then we checked the error scope and found that only two wrong certificates had been issued, one of them has been revoked, then we revoked the other certificate immediately, the whole process took less than two hours.

Comment 1

[Ryan Sleevi]

Pasting in from the PDF to enable easier searching:

1. Problem Report:

CFCA issued two wrong certificates ( for testing purposes ) with the OrganizationName

2. Timeline:

During the test procedure, our test engineer issued two wrong certificates with the OrganizationName , the first one issued on Jan 9 06:56:00 (UTC) 2020 and the second one issued on Jan 9 06:56:59 (UTC) 2020,the two certificates has been revoked on 07:19:53 (UTC) and 08:50:22 (UTC).

3. Statement

The wrong certificates are issued by our test engineer, he issued two error certificates inadvertently while on a test mission (the mission includes certificate application, revocation and update). According to the test case, he revoked the second certificate by himself. About one hour later, our auditor found the problem, then we checked the error scope and found that only two wrong certificates had been issued, one of them has been revoked, then we revoked the other certificate immediately, the whole process took less than two hours.

4. Summary

We issued two wrong certificates (for testing purposes), and we have revoked them less than two hours when it happen, this incident didn't affect anybody.

5. Certificate Data:

Please visit https://crt.sh/?id=2308076814 and https://crt.sh/?id=2308079369 to check the data.

6. Explanation:

Last year, we start the automatic ordering system development work. The system has been developed (Level 1) and is currently in acceptance testing, during the testing, the test engineer didn’t follow the test rules, the issuance and verification are all done by himself (this would not happen in practice, because our auditors, certificate issuers were distributed in two different departments, permissions are separate).

7. Steps:

1. We have revoked the wrong certificates immediately.
2. Test engineer permissions had been restricted to the minimum scope after this incident.
3. More professional auditors will join in the next testing process.

Comment 2

[Ryan Sleevi]

Oliver,

Thanks for reporting this. I'm glad to hear this was quickly caught, but I think there are a number of concerns that still remain.

Root Cause

I'm not sure sufficient root cause analysis has been done here. At a minimum, based on this report, we can see that users had more privileges than was necessary for their job, testing procedures were not followed nor technically enforced, automatic monitoring was not sufficient and required human review (the auditors). The goal isn't to identify that human error happened, but to figure out what enabled that human error happened and how to mitigate it.

Follow-up Questions

1. Have you reviewed all users permissions, to determine who can cause issuance and under what situations?
2. Have you reviewed how you can ensure testing procedures are followed, such as making it technically impossible to issue a certificate without multi-party review?
3. What caused the auditor to notice this certificate? For example, were you monitoring for revocation events and reviewing what caused them (e.g. as has been suggested to and implemented by other CAs)? Or was it just luck?
   • If it was luck, have you thought about implementing such alerts for revocation?
4. What is meant by "More professional auditors will join in the next testing process"?

Comment 3

[Oliver Bi]

(In reply to Ryan Sleevi from comment #2)

Oliver,

Thanks for reporting this. I'm glad to hear this was quickly caught, but I think there are a number of concerns that still remain.

Root Cause

I'm not sure sufficient root cause analysis has been done here. At a minimum, based on this report, we can see that users had more privileges than was necessary for their job, testing procedures were not followed nor technically enforced, automatic monitoring was not sufficient and required human review (the auditors). The goal isn't to identify that human error happened, but to figure out what enabled that human error happened and how to mitigate it.

Follow-up Questions

1. Have you reviewed all users permissions, to determine who can cause issuance and under what situations?
2. Have you reviewed how you can ensure testing procedures are followed, such as making it technically impossible to issue a certificate without multi-party review?
3. What caused the auditor to notice this certificate? For example, were you monitoring for revocation events and reviewing what caused them (e.g. as has been suggested to and implemented by other CAs)? Or was it just luck?
   • If it was luck, have you thought about implementing such alerts for revocation?
4. What is meant by "More professional auditors will join in the next testing process"?

Ryan,

Thanks for your reply, I understand your concern, the reply below may dispel some concern.

About Root Cause

The Root Cause is the tester had more privileges than was necessary for their job, I agree with that. You know, the permissions of the system are strictly differentiated, there is only one kind of permissions for each role (except super administrator), but during the test process, testers need to test each role and permissions to confirm whether the function achieves the expected effect. That's not too much problem, right? But in this case, the tester need to know the rules very clearly. Obviously, he didn't do it well enough on this job, he should know enough about the requirements(like BR)，I think the two main reasons led to the mistake;

About Follow-up Questions

1. Yes, we've reviewed the user permissions.
2. Yes, we've reviewed the testing procedures, the certificate will not be issued without an audit, but this issue warning me, I will add technical restrictions, such as the same IP prevents two or more different roles to log in, prevent abuse of permissions and manage vulnerabilities.
3. During the testing process, both product manager and compliance manager will join, we will verify the test process and result. During this review process, we discovered the wrong behavior and dealt it promptly.
4. As you said, the problem is the tester have too much privileges than was necessary, and we think the other one is the tester are not familiar with the requirements(like BR), that very important too, We should expect more from them, in the follow-up test, we will arrange professional auditors to train them, more frequent, more professional. This part of the testing will be done by a professional auditor until the tester meets the expectations.

Comment 4

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.