1853987 - Sectigo: S/MIME certificates with (null) string value in subject attributes

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Martijn Katerbarg]

Attachment: SMIME certificates with (null) surname or givenName.csv

Comment 1

[Martijn Katerbarg]

1. How your CA first became aware of the problem

During a manual review of our S/MIME certificate issuance, we noticed a number of certificates issued with “(null)” in the subject:givenName and subject:surname attributes. Due to certificate profile settings in some cases, these subject:givenName and subject:surname values were automatically included in the subject:commonName field as well.

2. Timeline

September 11, 2023 – 07:49 UTC
We create an internal ticket for our Database Administrator (DBA) team requesting a list of all issued S/MIME certificates going back to September 1, 2023.

September 12, 2023 – 14:41 UTC
Our DBA team provides a query for review and execution.

September 12, 2023 – 20:49 UTC
Our DBA team runs the first version of the query and provide a limited set of results for review.

September 13, 2023 – 07:36 UTC
We request a minor change to the query.

September 13, 2023 – 08:45 UTC
We start investigating the first set of results and start a routine check for any anomalies. Within minutes we notice records showing “(null)” in either the subject:givenName attribute, the subject:surname attribute, or both.

September 13, 2023 – 09:01 UTC
We reach out to our development team with our findings and request them to investigate further.

September 13, 2023 - 11:58 UTC
Our development team confirm they are investigating.

September 14, 2023 – 11:51 UTC
Our development team report back that we have found an external Identity Provider (IdP) that sends a literal string value of “(null)” in cases where an attribute value is missing, causing the data to be included in certificates issued for one of our Enterprise RAs.

September 14, 2023 – 11:48 UTC
We decide to develop a patch for our own systems to block issuance of any S/MIME certificate for which subject:givenName and/or subject:surname attributes are requested to be “(null)”.

September 14, 2023 – 13:14 UTC
Our development team requests approval from relevant stakeholders to perform a deployment.

September 14, 2023 – 14:42 UTC
We initiate customer notifications and schedule a revocation event for the certificates identified so far. Revocation is scheduled for September 18, 2023 at 07:30 UTC.

September 14, 2023 – 18:49 UTC
We deploy our patch blocking the “(null)” string, thereby remediating this issue.

September 18, 2023 – 07:20 UTC
We receive an updated list of all issued S/MIME certificates from our DBA team.

September 18, 2023 – 07:56 UTC
We revoke the initial 8 discovered certificates.

September 18, 2023 – 08:03 UTC
We complete the review of all issued certificates and identify a total of 126 certificates containing “(null)” in the subject:givenName and/or subject:surname attributes.

September 18, 2023 – 14:20 UTC
We initiate further customer notifications and schedule a second revocation event for September 22, 2023 at 22:15 UTC.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem.

We have deployed a patch that blocks further issuance of certificates with this problem.

4. Summary of the problematic certificates

126 S/MIME certificates issued between September 5, 2023 and September 14, 2023.

5. Affected certificates

The serial number and SHA256 of all affected certificates are listed in attachment 9353944 [details].

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now

All affected certificates were issued with the involvement of a single Enterprise RA customer, as permitted by subsection 5 of Section 3.2.4.1 of the SBRs.

Missing data in an IdP managed by this Enterprise RA, combined with the default behavior of the external IdP connector being to send a literal string value of “(null)”, led to our system receiving non-empty, supposedly valid strings in Enterprise RA records during the certificate request process and then issuing the certificates.

Prior to these events, we already blocked actual empty values for the affected Subject fields, but we were unaware that there was an IdP connector that would submit bogus “(null)” literal strings.

We decided to revoke these certificates as we believe they do not contain valid information. Additionally, we decided to open a Bugzilla bug to inform the wider community, and especially other CAs, that there exists at least one IdP connector that signifies empty values by submitting "(null)” literal strings.

As we do not have any direct relation to the IdP in question, we will refrain from mentioning its name. We believe the events described are detailed enough to allow other CAs to take appropriate action. For us, this issue brought to light the possibility of IdPs sending default values instead of empty values when data is missing. It also raises the suspicion that there may be other IdPs out there that we have not yet encountered following a similar practice. As there is no way to predict all possible combinations that any vendor may use, we believe co-operation and communication between CAs when they encounter such a case is essential, and we call upon all other CAs to publicly report these encounters.

Whilst preparing this incident report, we had to consider whether or not these events constitute an “incident.” SBR section 3.2.4.1 subsection 5 says that ”records maintained by the Enterprise RA SHALL be accepted as evidence of Individual identity,” which seems to imply that Sectigo should not be regarded as at fault and that this bug should perhaps be resolved as INVALID. Regardless, in the interest of informing the community (and especially other CAs) about potential pitfalls in executing the new S/MIME BRs, we feel it is valuable to provide relevant information in this incident report. We welcome further community discussion on this point.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future

As described above, we have put a block in place to prevent any further misissuance of certificates.

Comment 2

[Martijn Katerbarg]

All remaining certificates were revoked by September 22, 2023 – 20:26 UTC.

The revocation of the remaining certificates concludes our investigation and remediation of this bug. We are monitoring this bug for any questions and/or comments.

Comment 3

[Martijn Katerbarg]

There appear to be no further questions and/or comments.

Ben, we are still keen to hear opinions from the community and yourself regarding whether this bug should ultimately be RESOLVED FIXED or RESOLVED INVALID.

Comment 4

[Ben Wilson]

I believe that this bug should be closed as RESOLVED FIXED, rather than as INVALID. Full compliance with the S/MIME BRs is ultimately the CA's responsibility, which includes an ability to provide well-formed certificates. Subsection 5 of Section 3.2.4.1 doesn't seem to be clear enough to absolve the CA in this case.