1551375 - certSIGN: "Some-State" in stateOrProvinceName

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Wayne Thayer]

One certSIGN certificate containing a stateOrProvinceName of "Some-State" were published at https://misissued.com/batch/53/ This is apparently the default placed in OpenSSL CSRs, indicating that this field was not validated. BR section 7.1.4.2.2(f) states: If present, the subject:stateOrProvinceName field MUST contain the Subject’s state or province information as verified under Section 3.2.2.1. The EVGLs reference the BRs.

The certificate was revoked the day after being published.

Please provide an incident report, as described at https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report

Comment 1

[Cristian Garabet]

Incident Report

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

We became aware of the problem on Sat 5/11/2019 via our problem reporting mechanism, that is by receiving an email on [e-mail].

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

Sat 5/11/2019 8:36 PM received mail from Alex Cohn [e-mail]
„Inspired by Nick Lamb's comment a week or so ago on m.d.s.p about "Default City" being an OpenSSL default value in CSRs, I ran some more searches on the OpenSSL defaults and found almost 100 certificates with a stateOrProvinceName of "Some-State". BR section 7.1.4.2.2(f) requires this field to be verified if present in a certificate. “
Sat 5/11/2019 8:36 PM – 9:30 PM On https://misissued.com/batch/53/ we identified only one certificate issued by us with the mentioned problem.
Sat 5/11/2019 10:14 PM the certificate was revoked.
Monday 5/13/2019 we started to investigate the cause of the problem.
Tue 5/14/2019 3:54 AM – a bug with the problem was raised on bugzilla. mozilla.org
Friday 5/17/2019 6:00 PM – we have finalised the investigation and identified the cause of the issue as human error + insufficient technical controls regarding certificate subject fields data validation.
During Monday 5/20/2019 and Thursday 5/23/2019 we decided the changes to be made and implemented and communicated those changes to all those concerned.
3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

There was only one certificate issued with the mentioned problem. See next the measures we have taken.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

There was only one certificate issued with the mentioned problem. See next.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

There was only one certificate issued crt.sh ID 1012576297

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

We have an issuance procedure is in place at RA. The RA Officer checked the CSR she received from the client. When checking the CSR she failed to verify the field stateOrProvinceName. The second RA Officer performed the cross-check verification and also failed to verify the filed stateOrProvinceName. The RA interface visually distinguish DN subject fields provided by the customer CSR and validation errors (e.g. missing mandatory DN fields, invalid field length, invalid country, invalid characters in CN) are shown in the interface.
The CSR provided by the client was created using OpenSSL with the default value for stateOrProvinceName ‘Some-State’. During the checks performed by our Registration Authority Officers this was not detected which led to the issue of the certificate mentioned. One reason for this issue was the fact that there was no technical control in place for the content of field stateOrProvinceName. Also, the second reason is the fact that both RA officers failed to verify subject information. RA officers reported that sometimes they are disturbed by other colleagues while performing the verifications and they may have lost focus and missed to observe the wrong information.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We already implemented technical controls to check the default value in stateOrProvinceName=”Some-State” and L="Default City". Also we decided that the RA Officers that check SSL certificates information before issuance, while doing this, will concentrate solely on this type of activity and will not allow to be disturbed by other activities. We will provide trainings to our RA staff twice a year as opposed to once per year as the situation is currently . Also, we will increase the percentage of the sample certificates verified during the internal audit from 3% to 10%.

Comment 2

[Ryan Sleevi]

Cristian: Thanks for providing the update. I think it'd be useful to examine why an initial incident report wasn't filed (between the 2019-05-11 and 2019-05-14 timeline), and why it wasn't acknowledged until 2019-05-27. While certSIGN just barely squeaked under the upper-bound for incident reports ("certainly within two weeks of the initial incident report"), the delay in response didn't provide transparency into certSIGN's awareness of the issue or the steps being taken to resolve it.

Within your incident report, you note:

One reason for this issue was the fact that there was no technical control in place for the content of field stateOrProvinceName

However, the proposed technical control:

We already implemented technical controls to check the default value in stateOrProvinceName=”Some-State” and L="Default City"

Seems to address the symptom, rather than the cause. What other technical controls were examined, and why were they not implemented? Can you confirm that you've reviewed other CAs' responses, such as:

• Bug 1552562
• Bug 1551372
• Bug 1551375
• Bug 1548714
• Bug 1551371
• Bug 1551369
• Bug 1551363
• Bug 1551374
• Bug 1551364

For other issues and mitigations? Can you include your analysis about why each of those other CAs' specific mitigations may or may not be appropriate?

Comment 3

[Cristian Garabet]

Dear Ryan,
We acknowledged the incident on May 11, about 8 hours after its reporting of Alex Cohn on [e-mail]. In the same mail we informed the community that the certificate was revoked.

Besides human validation, we have several technical controls already in place, however none of these took into account the default contents for Some-State and Default City.
First, we have a CSR checker where the operator will upload and verify that the fields from the request are compliant and do not generate any errors. Second, after the request was submitted to the RA, before the pre-certificate is issued, we are running the cab certificate linter and if any errors are detected, the certificate is not issued.
Unfortunately, at the moment we have no authoritative source to check the provided StateOrProvice and Locality against, so rejecting the defaults automatically is our solution for the moment.

We are performing analysis on all the reasons and mitigations in the bugs referred but we do not have a conclusion yet. However, if we find something to improve on technical controls, or someone can suggest us something feasible, we are more than glad to do it.

Cristian

Comment 4

[Ryan Sleevi]

Christian: Do you have an update of your analysis of the controls from the related bugs, mentioned in Comment #3? For example, they highlight authoritative stateOrProvice sources.

Comment 5

[Cristian Garabet]

Hi Ryan,
We have already started to develop a new technical control for the CSR checker application based on ISO 3166-2. When the operator will validate the CSR, if stateOrProvice name is incorrect an error will be shown in the interface. We have planned to deploy the update in production until July 25th.

Cristian

Comment 6

[Cristian Garabet]

Today we have deployed the update in the production environment.

Comment 7

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.