Closed Bug 1398242 Opened 2 years ago Closed Last year

Disig: Non-BR-Compliant OCSP Responders

Categories

(NSS :: CA Certificate Compliance, task)

task
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: kwilson, Assigned: peter.miskovic)

References

Details

(Whiteboard: [ca-compliance])

Problems have been found with OCSP responders for this CA, and reported in the mozilla.dev.security.policy forum here:

https://groups.google.com/d/msg/mozilla.dev.security.policy/o1MX07iWDco/RuM1NK_0AQAJ

As per section 4.9.10 of the BRs, OCSP responders MUST NOT respond with a “good” status for unissued certificates. The effective date for this requirement was 2013-08-01.

Please provide an incident report in this bug, as described here:
https://wiki.mozilla.org/CA/Responding_To_A_Misissuance#Incident_Report


Note that the problem has been fixed as of 2017-08-31, so the purpose of this bug is to record the CA's incident report.
1)How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, via a discussion in mozilla.dev.security.policy, or via a Bugzilla bug), and the date.

Disig: Via a problem report submitted to your Problem Reporting Mechanism 

2) A timeline of the actions your CA took in response.
Disig: 

August 29, 2017 2:48 PM we obtained  e-mail from paul.l.kehrer@gmail.com about Violations of Baseline Requirements 4.9.10 - August 29, 2017 was a public holiday at Slovakia - no action was taken.

August 30, 2017 8:03 CET (UTC+2)we starting the investigation the problem and inform about starting of investigation Paul Kehrer via e-mail to paul.l.kehrer@gmail.com with copy to mozilla-dev-security-policy@lists.mozilla.org

August 30, 2017 8:03 to 16:30 - investigation of problem and fixing it at first for OCSP SubCAR2I2 Disig, OCSP SubCAR1I1 Disig.

August 31, 2017 7:24 CET - Send information to paul.l.kehrer@gmail.com about fixing problem for SubCA's and continuing fixing OCSP problem for CA Disig Root R1 and CA Disig Root R2 with copy to mozilla-dev-security-policy@lists.mozilla.org

August 31, 2017 7:30 to 16:45 CET - fixing OCSP problem for Root CA R1 and Root CA R2

August 31, 2017 16:45 OCSP problem fixed for Root CA R1 and Root CA R2 

September 4, 2017 11:42 CET Information about fixig a OCSP problem for root CA's via e-mail to paul.l.kehrer@gmail.com with copy to mozilla-dev-security-policy@lists.mozilla.org


3) Confirmation that your CA has stopped issuing TLS/SSL certificates with the problem.

Disig: Not applicable - problem was with OCSP response

4) A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

Disig: Not applicable - problem was with OCSP response

5) The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

Disig: Not applicable - problem was with OCSP response

6) Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

Disig:
OCSP problem that SubCA's were responding on OCSP request with bogus SN:0xdeadbeefdeadbeefdeadbeefdeadbeef with a "GOOD" response was in misconfiguration at the SubCA's responders

OCSP problem that ROOT CA's were responding on OCSP request with bogus SN:0xdeadbeefdeadbeefdeadbeefdeadbeef with a "GOOD" response was, that base of OCSP response was CRL's where response was in conformance wit RFC 6960 and our wrong decision that if the Root CA are off-line, this configuration is O.K.

7) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Disig:
 SubCA's OCSP responder's configuration was set-up such way, that the all OCSP response for SubCA's are in compliance with Baseline Requirements 4.9.10.
 Root CA's OCSP responder source was reconfigured such way that all OCSP response are in compliance with Baseline Requirements 4.9.10.
The explanation about why they were made or how they avoided detection (#6 in Comment #1) doesn't provide sufficient assurance that the CA has analyzed the factors involved or provided sufficient mitigations (#7 in Comment #1)

Given the BRs have long had this provision, and this provision was directly in response to the failure of DigiNotar, it's important to understand why Disig failed to implement these controls. It also raises concern as to whether or not Disig had a process in place to examine OCSP responses for non-issued certificates and alert on them.

Please provide greater detail about why these misconfigurations were made (or the decision to violate the BRs), as well as what steps systemically are being taken to address these issues.
Flags: needinfo?(peter.miskovic)
(In reply to Ryan Sleevi from comment #2)
> The explanation about why they were made or how they avoided detection (#6
> in Comment #1) doesn't provide sufficient assurance that the CA has analyzed
> the factors involved or provided sufficient mitigations (#7 in Comment #1)
> 
> Given the BRs have long had this provision, and this provision was directly
> in response to the failure of DigiNotar, it's important to understand why
> Disig failed to implement these controls. It also raises concern as to
> whether or not Disig had a process in place to examine OCSP responses for
> non-issued certificates and alert on them.
> 
> Please provide greater detail about why these misconfigurations were made
> (or the decision to violate the BRs), as well as what steps systemically are
> being taken to address these issues.

Hi Ryan, I'm out off my office now with the limited possibility to prepare qualified answer to your concerns. I think that we did all necessary mitigation action to resolve problem with our OCSP response. I'll be back from my holiday next Monday, so you have to wait for details. Thank you for understanding. 
Peter
Flags: needinfo?(peter.miskovic)
(In reply to Ryan Sleevi from comment #2)
> The explanation about why they were made or how they avoided detection (#6
> in Comment #1) doesn't provide sufficient assurance that the CA has analyzed
> the factors involved or provided sufficient mitigations (#7 in Comment #1)
> 
> Given the BRs have long had this provision, and this provision was directly
> in response to the failure of DigiNotar, it's important to understand why
> Disig failed to implement these controls. It also raises concern as to
> whether or not Disig had a process in place to examine OCSP responses for
> non-issued certificates and alert on them.
> 
> Please provide greater detail about why these misconfigurations were made
> (or the decision to violate the BRs), as well as what steps systemically are
> being taken to address these issues.

Hi Ryan, I'm out off my office now with the limited possibility to prepare qualified answer to your concerns. I think that we did all necessary mitigation action to resolve problem with our OCSP response. I'll be back from my holiday next Monday, so you have to wait for details. Thank you for understanding. 
Peter
(In reply to Ryan Sleevi from comment #2)
> The explanation about why they were made or how they avoided detection (#6
> in Comment #1) doesn't provide sufficient assurance that the CA has analyzed
> the factors involved or provided sufficient mitigations (#7 in Comment #1)
> 
> Given the BRs have long had this provision, and this provision was directly
> in response to the failure of DigiNotar, it's important to understand why
> Disig failed to implement these controls. It also raises concern as to
> whether or not Disig had a process in place to examine OCSP responses for
> non-issued certificates and alert on them.
> 
> Please provide greater detail about why these misconfigurations were made
> (or the decision to violate the BRs), as well as what steps systemically are
> being taken to address these issues.

Disig: Our certification authority uses custom OSCP software where is the possibility to configure OCSP response in accordance with different requirements.
Here is part of configuration file:
--------------------------------
Status that should be returned as SingleResponse.certStatus when requested certificate is not present in the database.
Allowed values:
good - According to RFC 6960 this positive response indicates that the certificate is not revoked, but does not necessarily mean that the certificate was ever issued or that the time at which the response was produced is within the certificate’s validity interval.
unknown - According to RFC 6960 this status indicates that the responder doesn’t know about the certificate being requested. This value is required by CA/Browser forum also for a certificate that has not been issued.
<undetermined_status>good</undetermined_status>
---------------------------------------
Default OCSP response configuration is according RFC 6960 where in paragraph 2.2. is written:
"The "good" state indicates a positive response to the status inquiry. At a minimum, this positive response indicates that no certificate with the requested certificate serial number currently within its validity interval is revoked. This state does not necessarily mean that the certificate was ever issued or that the time at which the response was produced is within the certificate’s validity interval."

Misconfiguration was made by using a default value in OCSP configuration file during the OCSP responder migration and software update.

What we did after receiving information from Paul Kehrer was:
SubCA’s:
1)	First we investigated why the problem occurred for issuing SubCA (we issued SSL only by CA Disig R2I2 Certification Service)
2)	Finding that there was misconfiguration in config file.
3)	Fixing the problem via changing configuration file.
4)	Checking if OCSP responder answer after configuration changing is conforming with BR requirements by sending different bogus OCSP requests – all OCSP response was with the status “unknown”.
5)	Check all our OCSP logs (we have all logs from beginning of OCSP service providing)  if there were similar OCSP requests for non-existing certificate issued by our CA Disig R2I2 Certification Service or OCSP requests for certificate we didn’t issued. We found that only record of OCSP request to our OCSP responder for non-existing certificate were from Aug 28, 2017 (Four (4) request for status of Serial="DEADBEEFDEADBEEFDEADBEEFDEADBEEF") – we identified that all came from Paul Kehrer.
6)	At that time we deciding that all OCSP response from CA Disig R2I2 are conforming wit BR requirement and in the past there was no real attempt to misuse OCSP service.
7)	We did the same for CA Disig R1I1 Certification Service – this CA is capable to issue SSL certificate, but we are not used it anymore due to sha1RSA signature is no accepted for issuing SSL certificates.
8)	After confirmation from Paul Kehrer that we fixed the problem (August 31, 2017) we again check our logs and found only one record of OCSP request for status of Serial="DEADBEEFDEADBEEFDEADBEEFDEADBEEF" we didn’t issue with the conforming OCSP response as „Status="unknown”” – we suppose that it was Paul Kehrer verification if we really fixed the problem.
9)	From this time we regularly monitor our OCSP responder logs for suspicious OCSP requests – till now we found another six (6) requests from September 5, 2017 and September 7, 2017 where three (3) of them requested for status of status of Serial="DEADBEEFDEADBEEFDEADBEEFDEADBEEF" which used Paul Kehrer and another three used three (3) different serials. We suppose that it was from somebody who also need to check if the response will be conforming to BR requirements – all six (6) our responses were with Status="unknown".

Root CA’s:
1)	After fixing the problem with SubCA’s (August 31, 2017) we investigate also root CA’s non-conformance with BR requirements. Our Root CA Disig R2 is in off-line mode so there be problem has a continuously synchronized database for OCSP response. So the configuration for the root CA’s was setup according to RFC 6960 and based on CRL, so the answers by OCSP responder for the root CA’s were conform to RFC 6960 but no to the BR requirement.
2)	We fixed the problem with Root CA R2 (August 31. 2017) by creation new OCSP database based on certificates issued by this root, where OSCP response is created according the information included in it. For all request on certificates status for serials which are not in this database is OCSP response conform to BR requirement with Status="unknown".
3)	We did the same for CA Disig Root R1 – this root CA is part of Mozilla store but in reality we don’t use it and in near future we will ask all root store program to remove it from root stores.
4)	We also investigate out OCSP logs for CA Disig Root R2 with the same result as for CA Disig R2I2 Certification Service – only record of OCSP request for status of Serial="DEADBEEFDEADBEEFDEADBEEFDEADBEEF") from Paul Kehrer testing.
5)	After fixing OCSP response for CA Disig Root R2 we found only four (4) requests from September 5, 2017 and September 7, 2017 where one (1) of them requested for status of status of Serial="DEADBEEFDEADBEEFDEADBEEFDEADBEEF" which used Paul Kehrer and two used serial very similar to Paul Kehrer (changing only last for character) and another two used two (2) different serials. We suppose that all requests were send for testing if response will be conforming to BR requirements – all five (5) our responses were with Status="unknown".
(In reply to Peter Miskovic from comment #5)
> Disig: Our certification authority uses custom OSCP software where is the
> possibility to configure OCSP response in accordance with different
> requirements.
> Here is part of configuration file:
> --------------------------------
> Status that should be returned as SingleResponse.certStatus when requested
> certificate is not present in the database.
> Allowed values:
> good - According to RFC 6960 this positive response indicates that the
> certificate is not revoked, but does not necessarily mean that the
> certificate was ever issued or that the time at which the response was
> produced is within the certificate’s validity interval.
> unknown - According to RFC 6960 this status indicates that the responder
> doesn’t know about the certificate being requested. This value is required
> by CA/Browser forum also for a certificate that has not been issued.
> <undetermined_status>good</undetermined_status>
> ---------------------------------------
> Default OCSP response configuration is according RFC 6960 where in paragraph
> 2.2. is written:
> "The "good" state indicates a positive response to the status inquiry. At a
> minimum, this positive response indicates that no certificate with the
> requested certificate serial number currently within its validity interval
> is revoked. This state does not necessarily mean that the certificate was
> ever issued or that the time at which the response was produced is within
> the certificate’s validity interval."
> 
> Misconfiguration was made by using a default value in OCSP configuration
> file during the OCSP responder migration and software update.

Peter,

First, let me say, thank you for the tremendous level of detail in your follow-up. This is exactly the kind of investigation we hope for all CAs to do - to maintain thorough logs and provide ways to automatically examine them. Despite the configuration issue, this speaks positively towards Disigs operations in the robustness of the response.

With that said, I think it's worthwhile to continue to understand more about the nature of the misconfiguration, and what steps can be taken to prevent future misconfigurations.

It would appear as if there was a process failure - as you note, it was a default configuration during your responder migration and software update. Understanding why this process failed, and how this process is being improved, is an important part of preventing future incidents.

Can you speak to what controls you had in place to ensure the configuration was correct, what may have caused those controls to fail, and what steps are being taken to improve those controls in the future? I have some suggestions to offer, but I think it'd be useful to allow you a chance to respond first.

Thanks
Flags: needinfo?(peter.miskovic)
Peter Miskovic: your input is needed here to respond to comment #6.

Gerv
Peter: the lack of response to the questions posed in comment 6 reflect poorly on Disig as a CA worthy of public trust. Do you intend to respond, or shall I close this bug on that note?
(In reply to Ryan Sleevi from comment #6)
> (In reply to Peter Miskovic from comment #5)
> > Disig: Our certification authority uses custom OSCP software where is the
> > possibility to configure OCSP response in accordance with different
> > requirements.
> > Here is part of configuration file:
> > --------------------------------
> > Status that should be returned as SingleResponse.certStatus when requested
> > certificate is not present in the database.
> > Allowed values:
> > good - According to RFC 6960 this positive response indicates that the
> > certificate is not revoked, but does not necessarily mean that the
> > certificate was ever issued or that the time at which the response was
> > produced is within the certificate’s validity interval.
> > unknown - According to RFC 6960 this status indicates that the responder
> > doesn’t know about the certificate being requested. This value is required
> > by CA/Browser forum also for a certificate that has not been issued.
> > <undetermined_status>good</undetermined_status>
> > ---------------------------------------
> > Default OCSP response configuration is according RFC 6960 where in paragraph
> > 2.2. is written:
> > "The "good" state indicates a positive response to the status inquiry. At a
> > minimum, this positive response indicates that no certificate with the
> > requested certificate serial number currently within its validity interval
> > is revoked. This state does not necessarily mean that the certificate was
> > ever issued or that the time at which the response was produced is within
> > the certificate’s validity interval."
> > 
> > Misconfiguration was made by using a default value in OCSP configuration
> > file during the OCSP responder migration and software update.
> 
> Peter,
> 
> First, let me say, thank you for the tremendous level of detail in your
> follow-up. This is exactly the kind of investigation we hope for all CAs to
> do - to maintain thorough logs and provide ways to automatically examine
> them. Despite the configuration issue, this speaks positively towards Disigs
> operations in the robustness of the response.
> 
> With that said, I think it's worthwhile to continue to understand more about
> the nature of the misconfiguration, and what steps can be taken to prevent
> future misconfigurations.
> 
> It would appear as if there was a process failure - as you note, it was a
> default configuration during your responder migration and software update.
> Understanding why this process failed, and how this process is being
> improved, is an important part of preventing future incidents.
> 
> Can you speak to what controls you had in place to ensure the configuration
> was correct, what may have caused those controls to fail, and what steps are
> being taken to improve those controls in the future? I have some suggestions
> to offer, but I think it'd be useful to allow you a chance to respond first.
> 
> Thanks

Ryan,

he problem with using OCSP's default configuration was that we simultaneously migrated multiple OCSP respondents and not all belonged to the CA / Browser Forum rules. The person who performed the migration forgot to change the original configuration of the affected OCSP respondents, which eventually caused the incident. At that time, we did not introduce any measures to detect this deficiency.

Here are our correction action we did to prevent future incidents.

After finding the problem in the configuration file and its settings
we changed immediately the default settings in the source files of our OCSP so that after the installation, the OCSP response is set in accordance with the requirements of the CA / Browser Forum.

The OCSP installation and startup procedures, including the configuration, are described in an internal "OCSP Description Service" document. There are written procedures for setting up the OCSP service to ensure that a similar problem is no longer occurring, including the migration of OCSP responders..

We also have internal monitoring (Nagios) for the relevant OCSP responders where we constantly monitor the response to a randomly generated SN that was not issued by our certification authority.
Flags: needinfo?(peter.miskovic)
Changing QA contact per https://bugzilla.mozilla.org/show_bug.cgi?id=1438254
QA Contact: gerv → wthayer
Status: NEW → RESOLVED
Closed: Last year
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.