KIR S.A.: Certificates issued with multiple BR violations

ASSIGNED
Assigned to

Status

task
ASSIGNED
11 months ago
Last month

People

(Reporter: wayne, Assigned: piotr.grabowski)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [ca-compliance])

CABLint, X509Lint, and ZLint have identified multiple errors with certificates issued from the SZAFIR ROOT CA2, including postalAddress containing invalid information, SANs containing email adresses, and organizationName without State or Locality fields.

https://crt.sh/?caid=15985&opt=cablint,zlint,x509lint&minNotBefore=2017-01-01

Please provide an incident report for this problem, as described here:
https://wiki.mozilla.org/CA/Responding_To_A_Misissuance#Incident_Report
The incident report should be posted to the mozilla.dev.security.policy forum and added to this bug.
Hello, 
Thank you for your email. We started preparing the incident report and when it is finished we will post it to the mozilla.dev.security.policy.
Assignee: wthayer → piotr.grabowski
Here is the incident report that was posted to mozilla.dev.security.policy:

1.    How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, via a

discussion in mozilla.dev.security.policy, or via a Bugzilla bug), and the date.

Email from Wayne Thayer Oct 1, 2018

2.    A timeline of the actions your CA took in response.

A. Oct 2, 2018 - Investigation began.
B. Oct 4, 2018 - Found impacted certificate policy templates.
C  Oct 4, 2018 - All the certificates owners were contacted and agreed on issuance new BR compliant certificates in time convenient for them,                  preferably not later than by the end of this year and revocation current ones.
D. Oct 8, 2018 - Fixed impacted certificate policy templates.
E. Oct 8, 2018 - This disclosure.

Ongoing:
F. Replacement of impacted certificates
G. Training of periodic certificate policy templates validation.

3.    Confirmation that your CA has stopped issuing TLS/SSL certificates with the problem.

Confirmed.

4.    A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem

were issued.

There are 46 certificates.  The certificates were issued between Feb 20, 2017 and Sep 25, 2018.

5.    The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is

logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct
problem.

Added as attachment
https://crt.sh/?caid=15985&opt=cablint,zlint,x509lint&minNotBefore=2017-01-01

6.    Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The the incident concerns 46 certificates in the vast majority issued on KIR S.A. internal system purposes.
The root cause of this issue was human error in certificate policy templates.


Remediation items:

1. Reviewed all certificate policy templates for ensuring that all of them are BR comliant.
2. All the certificates owners were contacted and agreed on issuance new BR compliant certificates in time convenient for them, preferably not later than by the end of this year and revocation current ones.
3. Added procedural step for periodic certificate policy templates validation.
Follow-up questions are being answered on the mozilla.dev.security.policy list: https://groups.google.com/d/msg/mozilla.dev.security.policy/6MSwBBikf10/q-vyCtr_AwAJ

Piotr: please update this bug as pre- and post-issuance linting, as well as any other additional remediation steps, are scheduled and completed.
Poitr: Please provide an update on the status of pre- and post-issuance linting.
Flags: needinfo?(piotr.grabowski)

Hello,

  1. With regard to the pre-issuance linting we have requested such feature from our PKI software vendor. Now they are considering
    feasibility of such change request. It the meantime we've received some more guidance concerning extended policy templates
    validation. We have implemented them.

  2. Post-issuance linting is already implemented as part of SSL certificate issuance proccess.

Flags: needinfo?(piotr.grabowski)

Meanwhile, KIR has misissued another certificate: https://crt.sh/?id=1036631881&opt=zlint

Please respond to this bug with another incident report including an explanation of why this certificate was not detected, reported, and revoked, and clear timelines for preventing any future occurrences.

Flags: needinfo?(piotr.grabowski)
Status: NEW → ASSIGNED

Piotr Grabowski posted the following response in the mozilla.dev.security.policy thread (https://groups.google.com/d/msg/mozilla.dev.security.policy/6MSwBBikf10/fQDHubsRDQAJ):

I think it is still the same incident. In my opinion there is no need to create another thread. Let me explain how it happened.

As I said in previous emails we have requested urgent need for pre-linting feature from Verizon just after the incident was reported. I have sent a few reminders

and finally on 18th of December 2018 got a response :

"The feature request of implementing pre-issuance linting was discussed with PM and development today. Our feeling is that UniCERT is policy-centric software and therefore the burden of ensuring that “misissuance” does not happen is on the implementer of the policies. To help with this, we provide registration policy templates in the Registration Policy Wizard and they ensure most of x509lint test. ", but the keyword here is most.

We can not for example restrict the length of the field like organizationName to have no more than 64 characters. We can set it as mandatory or not.

It is the only gap we have now in our policy templates. We tried to mitigate this risk putting the informational text on operators website and in the dedicated

procedure to double check the orgranizationName text size when operator is filling up the form of certificate request but but this time he did not perform this check. The post-linting procedure officially came into a force on the 1st of January 2019 as a procedural step to check the certificate that was just issued so unfortunately this certificate was not encompassed by the procedure. The OCSP status is 'unknown' because the customer have not picked up the certificate yet.

Anyway, I sent another email to Verizon that we need specific x509lint or zlint feature with fileld length validation so the case is in progress.

Please belive me, we try to practice due care as much as we can but we are just one among many clients of Verizon and we don't have such a clout to

force them to implement new feature in the timeline we propose. I have described in details our need and belive they will deliver the feature as soon as they can.

We have already reinforced and made live our post-linting procedure to check not only certificate that was just issued but check wider range like this

https://crt.sh/?caid=15985&opt=cablint,zlint,x509lint&minNotBefore=2019-01-01

TODO:

  •   Keep exerting pressure on Verizon to deliver:
    

o Policy field size validation – in our opinion it is simple change request and should be delivered ASAP.

o native x509lint or zlint feature

Responded to the mozilla.dev.security.policy thread with a request for answers to the questions that had been posted.

Unfortunately I have to report problem with one more certificate
https://crt.sh/?id=1120102462&opt=cablint
I will write more detailed description on Monday but for this time it is software bug.

Flags: needinfo?(piotr.grabowski)

Piotr: this sounds like a significantly different issue. Please create a new bug when submitting your incident report.

ok

We have revoked this certificate, identified and removed from usage possible but not yet confirmed source of the problem in our software, raised the issue to Verizon (Ticket 395958) but we need couple of days for them to confirm our suspicions. Then I will have sufficient amount of data to prepare the incident report.

Duplicate of this bug: 1532112

When can we expect https://crt.sh/?id=1036631881&opt=cablint,x509lint,zlint to get revoked? OCSP status is Unknown.

Piotr: Any updates from Comment #12? That was 5 months ago.

Flags: needinfo?(piotr.grabowski)

Hello all,
The problem was identified and resolved.
Actually it was configuration issue and after detailed explanation by vendor the problematic setting was fixed.
In the meantime we have also deployed basic prelinting patch but still waiting for further prelinting features.
Now we are issuing much more certificates and I think our validation processes look well.
https://crt.sh/?caid=15985&opt=cablint,zlint,x509lint&minNotBefore=2019-02-01

Flags: needinfo?(piotr.grabowski)

With respect to Comment #16, that doesn't really provide the detail I was anticipating from Comment #12. Could you provide a more thorough description about the problem and the configuration issue?

If you want examples of how other CAs have provided detailed reports, consider Bug 1551374 or Bug 1556948 as examples, or further back, Bug 1390978.

Flags: needinfo?(piotr.grabowski)

Emailed POCs on 2019-07-04 regarding this issue, highlighting https://wiki.mozilla.org/CA/Responding_To_An_Incident#Keeping_Us_Informed

Flags: needinfo?(piotr.grabowski)

Piotr: Thank you for reporting this.

However, I do want to highlight a few details:

The incident reported in Comment #19 doesn't seem to respond to address any of the concerns highlighted in Comment #13/Comment #14. I can understand if they're being treated separate - in which case, we should undupe the bug. If Comment #16 was meant to encompass Comment #13 / Comment #14, then it doesn't contain sufficient information - hence, Comment #17. If it wasn't, then we still need details about what went wrong in Comment #13 / Comment #14.

Flags: needinfo?(piotr.grabowski)

Ryan:
Comment #16 does contain sufficient information because it stated that: "In the meantime we have also deployed basic prelinting patch ..." but it can be extended by information that this patch validates size, among others, OU field.
What went wrong in Comment #13 / Comment #14 are the result of lack of the pre-linting feature from https://bugzilla.mozilla.org/show_bug.cgi?id=1495497. As we disccussed https://bugzilla.mozilla.org/show_bug.cgi?id=1523186 the patch is already deployed.

Flags: needinfo?(piotr.grabowski)

A lack of pre-issuance linting is not really a root cause. That is, it describes a lack of a mitigation, but doesn't explain why the system lacked the mitigation. With respect to linting, it should not be seen as a substitute for effective controls; that is, linting is the last check in a process to try and make sure all of your controls succeeded properly. If linting rejects something, it means that there is an underlying systemic issue that needs to be corrected and fixed. It is complementary to controls, not a replacement for them.

Comment #16 simply states "problematic setting", and the further response, in was suggested that the issue was a lack of precertificates, which is not a BR violation. However, what is (or may be) a BR violation is omitting the AIA extension with OCSP responder (per BRs 7.1.2.3 (c)). However, nothing of the incident report seems to support why that was reported in #9, nor do the details in Comment #9, Comment #12, or Comment #19 really identify the BR violation, how it happened, or how it's been fixed.

Since you mentioned the CA software in question in Comment #12, it sounds like the Verizon software does not properly implement RFC 5280. Implementing pre-issuance linting is thus arguably an insufficient mitigation; while it detects problems, CAs should treat any failure of any linter that prevents a certificate just as seriously as if it weren't there, because it represents a systemic control failure. The CA is ultimately responsible for knowing and ensuring those controls are in place, and relying on linting can be seen as a way to avoid having to understand that role.

Flags: needinfo?(piotr.grabowski)

Maybe I will try to summarize:

Comment #12 was a response to Comment #10, which itself was about Comment #9.
Comment #9 disclosed https://crt.sh/?id=1120102462&opt=cablint

I agree - and I have created separate report for this https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/ndKdwaCqWb8 and explained in p. 6 how it happened. We have intentionally created registration policy with following "settings":

  • SCT enabled
  • not listed as policy enabled for CT logging
    We were sure that other fields such as AIA extension are irrelevant because we were hoping that the result of this test will be negative.

The problem with this bug was imprecise documentation which was clarified further with vendor and fixed.

Flags: needinfo?(piotr.grabowski)
You need to log in before you can comment on or make changes to this bug.