Closed Bug 1763173 Opened 2 years ago Closed 2 years ago

certSIGN: Incorrect data in stateOrProvinceName

Categories

(CA Program :: CA Certificate Compliance, task)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: gabriel.petcu, Assigned: gabriel.petcu)

Details

(Whiteboard: [ca-compliance] [ov-misissuance])

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36

Steps to reproduce:

Following an internal request for an OV SSL certificate our RA operators started the issuance procedure by creating a Pre-certificate. Due to a bug on the update of the CA software recently applied, the automatic verification of the certificate was skipped due to a technical issue with the linter warnings allowing bypassing the technical control and the pre-certificate was issued. The second verification post issuance revealed the error, the stateOrProvinceName was incorrect, and the pre-certificate was revoked.

Actual results:

Actual results:
This bug filing encompasses CAs managed by Certsign S.A., which includes the Intermediate CA: certSIGN Enterprise CA Class 3 G2

  1. How your CA first became aware of the problem, and the time and date:
    The problem had been identified by certSIGN operators on April 4th, 2022, immediately after the issuance of the certificate
  2. A timeline of the actions your CA took in response:
    Note: Times are listed in EEST time zone
    • April 4th, 2022 16:31 – the certificate for OV SSL was issued with error
    • April 4th, 2022 18:00 – Update rollback to the previous configuration
    • April 5th, 2022 09:00 - Internal incident created
    • April 5th, 2022 17:00 – the certificate was revoked
    • April 5th, 2022 17:30 - analysis on the improvement of the process started
    • April 5th, 2022 18:50 – Issue reported by Gabriel Petcu to Bugzilla
  3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation:
    Our initial investigation confirmed that the root cause was a bug in the CA update and was fixed through update rollback to the previous configuration. There was only one certificate issued with the mentioned problem. See next.
  4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued:
    There was only one certificate issued with the mentioned problem. See next.
  5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem:
    The known impacted certificates were already provided as crt.sh link with ID 6476032747 for the pre-certificate (https://crt.sh/?id=6476032747).
  6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now:
    An update was applied on March 30, 2022 to the CA software application in order to improve the response time for certificate issuance (see Bugzilla bug 1762707 - certSIGN: Subscriber precertificate without Certificate Policies). We have determined that there was a technical issue with the linter warnings allowing bypassing the technical control and relying on the operator for the final decision.

Expected results:

List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things:
All warnings will be treated as exceptions. We will have this update in production until 31st of May 2022, taking enough time to test thoroughly a new set of test cases which arise from this incident.

Assignee: bwilson → gabriel.petcu
Whiteboard: [ca-compliance]

Could you describe a little more about how the system allowed the invalid stateOrProvinceName in the first place, as well as how the operator decided to override the warning?

That is, it sounds great that pre-issuance linters are in place, but it’s useful to understand not only how the linter ended up being temporarily misconfigured, but what processes were involved in the validation.

For example, does the operator have full discretion? Is there a training/knowledge issue? Do other controls besides linters exist for these fields?

Flags: needinfo?(gabriel.petcu)
Type: defect → task

The stateOrProvinceName was filled wrong on the OV SSL CSR request. The bug on the previous update on the CA software, that was rollback after this incident, did not present the warnings to the operator.

Following previous years issues regarding mis-issue of certificates, we have developed an Automatic Validation process through a Linter app that checks the validity rules per profile and fields and if warnings are returned from the Linter, specific messages are presented to the operator. On any error the process is stopped automatically and the request is rejected. The Linter raises a warning for instance if it sees “SomeState” in the State or Province field, as it is checked against ISO 3166-2.

When a warning is raised, the operator is notified that an action must be taken and he might decide to accept the request after manually checking the CSR. In this specific case we have discovered that the update caused the warning not to be presented to the operator, and he assumed that the request has passed the linting process successfully and it is safe to be issued.

With this occasion, we have identified a weakness, our operators rely heavily on the implemented technical controls (they caught almost all potential problems until now) and if the technical control fails silently like this, they assume that everything is fine. We will have to make sure that they are following the procedures as laid out.

Flags: needinfo?(gabriel.petcu)

We continue to monitor all the issuing of the production certificates, and we will post another status update on April 29, 2022

We continue to monitor all the issuing of the production certificates, and we will post another status update until May 31, 2022

Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true

We have deployed the planned update in the production environment.

I will close this on or about Friday, 3-June-2022.

Flags: needinfo?(bwilson)
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] → [ca-compliance] [ov-misissuance]
You need to log in before you can comment on or make changes to this bug.