1714439 - DigiCert: Incorrect RegNumber-Org Type combination

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Brenda Bernal]

We are filing the following incident report:

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

DigiCert’s compliance analytics team was scanning for potential certificate issues and found certificates issued where the organizational type failed to match the registration number convention. This prompted an internal investigation where we found 39 certs with this condition with our certificates. We have also detected similar issues with other CAs. We will be reporting the non-DigiCert certificates identified in the same scan to the appropriate CAs.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

05/29/2021: Compliance team completed a new scan for information related to mis-matched registration numbers to their org type. This list was provided to the internal audit team for review and identification of false positives.
6/2/2021: Internal audit team completed review and sent the identified issues to validation for review. The positive list was sent to revoke alias and customer notifications started. We also ran another comprehensive system sweep to double check the corpus of impacted certificates and found no other issues.
The Issue was identified as a missing system logic check to make sure the type of organization matched the registration number.
6/3/2021: A system fix was deployed to DigiCert systems to ensure integrity between the type of organization and registration field. A fix associated with QuoVadis certificate issuance is being evaluated.
6/07/2020: The problem certificates will be revoked; see full list in section 5.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

New code was released on June 3, 2021 that prevents agents from selecting: Private Organization, Business Entity, Non-Commercial Entity, or Person and entering the JOI registration number field as “Government Entity”. The condition to ensure consistency in the org type and registration number was enforced with this fix.

One of the listed certificated pertains to the QuoVadis CA. Ongoing integration of systems is in flight, and we are still evaluating a systematic fix on the QV system for this particular issue. We will provide an update.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

The first certificate was issued on 07/31/2019.
The last certificate was issued on 02/10/2021.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
   https://crt.sh/?id=1543129538
   https://crt.sh/?id=1902158497
   https://crt.sh/?id=1925725068
   https://crt.sh/?id=2967297771
   https://crt.sh/?id=2977113959
   https://crt.sh/?id=3111553326
   https://crt.sh/?id=3125061413
   https://crt.sh/?id=1772415778
   https://crt.sh/?id=1754294184
   https://crt.sh/?id=3461708568
   https://crt.sh/?id=3390156301
   https://crt.sh/?id=3331865744
   https://crt.sh/?id=3505837348
   https://crt.sh/?id=2011031570
   https://crt.sh/?id=2020608683
   https://crt.sh/?id=2052358635
   https://crt.sh/?id=2176689822
   https://crt.sh/?id=3520646900
   https://crt.sh/?id=3601373973
   https://crt.sh/?id=3658688679
   https://crt.sh/?id=3600985742
   https://crt.sh/?id=3712826540
   https://crt.sh/?id=2100494937
   https://crt.sh/?id=3542466195
   https://crt.sh/?id=3659281718
   https://crt.sh/?id=3704451727
   https://crt.sh/?id=1971161180
   https://crt.sh/?id=3784014506
   https://crt.sh/?id=2327100299
   https://crt.sh/?id=2075033537
   https://crt.sh/?id=2481836874
   https://crt.sh/?id=2602266083
   https://crt.sh/?id=2705872666
   https://crt.sh/?id=4060571885
   https://crt.sh/?id=3067267574
   https://crt.sh/?id=2744009204
   https://crt.sh/?id=3253678755
   https://crt.sh/?id=3268377318
   https://crt.sh/?id=3233360466

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

For each impacted certificate, the validation agents selected the incorrect org type. The two fields (org type and registration number) were not systematically checked for consistency. We implemented a fix on June 3, 2021 to tie the org type and registration number together.

7. List steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We have implemented a system block on organizational type and org. registration number mis-match. The fix was implemented in production on June 3, 2021. We are currently evaluating the other interconnections between certificate fields to see what other improvements can be made.

Comment 1

[Ryan Sleevi]

05/29/2021: Compliance team completed a new scan for information related to mis-matched registration numbers to their org type. This list was provided to the internal audit team for review and identification of false positives.

What prompted this review, and can you speak more about what sort of analysis was performed? That is, it sounds like "Well, we looked through serialNumber for the string 'Government Entity'", and I'm hoping you can speak more to what prompted this or how it was conducted.

Comment 2

[Ryan Sleevi]

We have implemented a system block on organizational type and org. registration number mis-match.

Also, can you provide more detail on what precisely this means (e.g. to allow other CAs to replicate the same solution)

It also sounds like, based on the answer to Question 2, either a new incident for QuoVadis is expected, or the answer to Question 7 should include a binding timeline for resolving the QuoVadis issues.

Comment 3

[Jeremy Rowley]

Hey Ryan,

The Investigation:
The analytics team noticed a validation approval request in the system with a mixed registration type and org type. That particular validation was rejected as being incorrect. Based on the rejected certificate request, the analytics team kicked off a query to see if any CA had a similar problem. The answer turned out to be yes. The JOI linter we deployed previously only looked at the registration number. What we needed is to bridge the JOI and Org type info as they aren't necessarily independent fields. The change ties the org validation to the JOI linter to ensure consistency between the org type and registration number used.

Fixing the Issue:
The fix we put in place is to make sure an entity verified as a private org cannot be selected as a government entity type. The entity type was a dropdown box that was selected by the validation staff. It's still a drop down box, but the two are now tied together.

Quovadis:
We're working with the QV engineering team to get a fix in place on their system. We can either file a new bug for Quovadis or report on progress here by updating part 7 of the bug report, whichever is more convenient for you. The fix will be similar (making sure that government entity can't be selected in the org type where the verification was done as a private org). We expect the Quovadis fix to deploy in three days.

Comment 4

[Ryan Sleevi]

(In reply to Jeremy Rowley from comment #3)

Quovadis:
We're working with the QV engineering team to get a fix in place on their system. We can either file a new bug for Quovadis or report on progress here by updating part 7 of the bug report, whichever is more convenient for you. The fix will be similar (making sure that government entity can't be selected in the org type where the verification was done as a private org). We expect the Quovadis fix to deploy in three days.

I think it would make sense to treat all DigiCert-operated CAs as DigiCert (i.e. including QuoVadis). I know historically there's been separate bugs for incidents discovered immediately after M&A closing (e.g. Verizon, Symantec), but from a community perspective, we're dealing with DigiCert and DigiCert's management. The time to integrate (or not integrate) an acquisition is mostly a reflection of the compliance priorities of the acquirer, and thus seems relevant.

So yes, if you can make sure that all DigiCert operated CAs are in scope of this (and subsequent) incident reports, as well as timelines for resolution and fixing, that'd be great. That's no different than some CAs that operate multiple platforms (e.g. GlobalSign's Atlas & GCC, GTS's in-house and PrimeKey based CAs) are treated as one logical incident.

Comment 5

[Brenda Bernal]

Here is the consolidated update for this issue for all DIgiCert-operated CAs:

• All certificates identified (noted in 5 above) were revoked.
• The system fix for QuoVadis was applied that would block the org type and registration number inconsistency.

Comment 6

[Brenda Bernal]

Ben, Is there anything else needed for this bug? Otherwise, we request for closure.

Comment 7

[Ben Wilson]

Unless there are comments or objections, I will queue this up to be closed on Wed. 23-June-2021.