Closed Bug 1398246 Opened 2 years ago Closed 11 months ago

Consorci AOC: Non-BR-Compliant OCSP Responders

Categories

(NSS :: CA Certificate Compliance, task)

task
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: kwilson, Assigned: fferre)

References

Details

(Whiteboard: [ca-compliance])

Attachments

(1 file)

35.15 KB, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Details
Problems have been found with OCSP responders for this CA, and reported in the mozilla.dev.security.policy forum here:

https://groups.google.com/d/msg/mozilla.dev.security.policy/o1MX07iWDco/RuM1NK_0AQAJ

As per section 4.9.10 of the BRs, OCSP responders MUST NOT respond with a “good” status for unissued certificates. The effective date for this requirement was 2013-08-01.

Please provide an incident report in this bug, as described here:
https://wiki.mozilla.org/CA/Responding_To_A_Misissuance#Incident_Report
Francesc: it has now been 13 days. Please respond ASAP to tell us how you plan to address this issue.

Gerv
Dear all,

Finally we provide the answer to this bug:

1.- How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, via a discussion in mozilla.dev.security.policy, or via a Bugzilla bug), and the date.

We were aware of the issue the 29th of August via an email from Paul Kehrer via dev-security-policy

2.- A timeline of the actions your CA took in response.

In fact,as far as we see it, it is not a problem. The reported CAs are no longer issuing SSL/EV certificates.
All SSL/EV certificates issued by Consorci AOC are issued from CA commonName = EC-SECTORPUBLIC, Serial Number: 71b065397c8e07d2541a967f75593792, whose OCSP responder is working pursuant current Baseline Requirements

3.- Confirmation that your CA has stopped issuing TLS/SSL certificates with the problem.

Yes we do confirm that the affected CAs are no longer issuing SSL nor EV certificates.

4.- A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

DN: C=ES, O=Agencia Catalana de Certificacio (NIF Q-0801176-I), L=Passatge de la Concepcio 11 08008 Barcelona, OU=Serveis Publics de Certificacio ECV-2, OU=Vegeu https://www.catcert.net/verCIC-2  (c)03, OU=Administracions Locals de Catalunya, CN=EC-AL Example cert:
https://crt.sh/?q=88f6298c5a8cc66cefb8ea214a528c3efce36a26213fe4fd260613967d39e7d4
OCSP URI: http://ocsp.catcert.net

Last certificate issued : 2015-09-29, only 4 still valids at this moment expiring 2017-09-29 (tomorrow)

DN: C=ES, O=Agencia Catalana de Certificacio (NIF Q-0801176-I), L=Passatge de la Concepcio 11 08008 Barcelona, OU=Serveis Publics de Certificacio ECV-2, OU=Vegeu https://www.catcert.net/verCIC-2  (c)03, OU=Administracions Locals de Catalunya, CN=EC-AL Example cert:
https://crt.sh/?q=1869a83f83b8f034336ab09fe52563c00c80c4b45897b3ea15e658fd14306208
OCSP URI: http://ocsp.catcert.net

same CA as above. 

DN: C=ES, O=Agencia Catalana de Certificacio (NIF Q-0801176-I), L=Passatge de la Concepcio 11 08008 Barcelona, OU=Serveis Publics de Certificacio
ECV-2, OU=Vegeu https://www.catcert.net/verCIC-2   (c)03, OU=Secretaria
d'Administracio i Funcio Publica, CN=EC-SAFP Example cert:
https://crt.sh/?q=15d3c7463f477e2627c4c9a158e429abd6bfe63101d6745560a36d1c03363d30
OCSP URI: http://ocsp.catcert.net

Last certificate issued : 2015-09-29, only 4 still valids at this moment expiring 2017-09-29 (tomorrow)

DN: C=ES, O=Agencia Catalana de Certificacio (NIF Q-0801176-I), L=Passatge de la Concepcio 11 08008 Barcelona, OU=Serveis Publics de Certificacio ECV-2, OU=Vegeu https://www.catcert.net/verCIC-2 (c)03, OU=Universitats i Recerca, CN=EC-UR Example cert:
https://crt.sh/?q=7432b4c29e1360668814ec282ad78208cd521e62b8d8d60d5084fdf8daad57cb
OCSP URI: http://ocsp.catcert.net

Last certificate issued : 2015-10-07, only one still valid at this moment expiring 2017-10-07 

DN: C=ES, O=Agencia Catalana de Certificacio (NIF Q-0801176-I), L=Passatge de la Concepcio 11 08008 Barcelona, OU=Serveis Publics de Certificacio ECV-2, OU=Vegeu https://www.catcert.net/verCIC-2 (c)03, OU=Universitats i Recerca, CN=EC-UR Example cert:
https://crt.sh/?q=3148d57a495fd7bdf4653dfdd3d3c9d186547df42e296c4e1b6a7c679179d03f
OCSP URI: http://ocsp.catcert.net

same CA as above. 

DN: C=ES, O=Agencia Catalana de Certificacio (NIF Q-0801176-I), L=Passatge de la Concepcio 11 08008 Barcelona, OU=Serveis Publics de Certificacio, OU=Vegeu https://www.catcert.net/verCIC-3 (c)05, OU=Universitat Rovira i Virgili, CN=EC-URV Example cert:
https://crt.sh/?q=caa2a1fe7756bd5e227add40c5e06808dc0a79f1e8c93e4bf982df4893b284e4
OCSP URI: http://ocsp.catcert.net

Last certificate issued : 2014-05-09, all expired already

DN: C=ES, O=Agencia Catalana de Certificacio (NIF Q-0801176-I), OU=Serveis Publics de Certificacio, OU=Vegeu https://www.catcert.net/verarrel (c)03, OU=Jerarquia Entitats de Certificacio Catalanes, CN=EC-ACC Example cert:
https://crt.sh/?q=356a5f4d994e9efa7caefc491768911d65ec25977465b610e2f29aa4472631c3
OCSP URI: http://ocsp.catcert.net

Last certificate issued : 2008-07-23, all expired already.   

DN: C=ES, O=Agencia Catalana de Certificacio (NIF Q-0801176-I), OU=Serveis Publics de Certificacio, OU=Vegeu https://www.catcert.net/verarrel (c)03, OU=Jerarquia Entitats de Certificacio Catalanes, CN=EC-ACC Example cert:
https://crt.sh/?q=20d082b1f53252e33cee5991be8650b414f11f3af16a14295c2fee0c9ab558c2
OCSP URI: http://ocsp.catcert.net

Same as above

5.- The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

These CAs stopped issuing certificates the 2015-09-29, at the latest. 

6.- Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

It is not a problem of the CA certificates. We decided to group all certificates policies issuing SSL or EV certificates under a common CA (commonName = EC-SECTORPUBLIC), whose OCSP responder is working pursuant current Baseline Requirements 

7.- List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Our intention is to wait until they all expire (most of them tomorrow and only one in a week aprox.). If we are not wrong, no further steps are needed.
Please let us know if you need further details.

Thank you very much and once again, sorry for the late reply,

Francesc,
Paul Kehrer: the CA asserts this is working as it should. If it's not, can you provide more details steps to reproduce the problem?

Thanks,

Gerv
If the CAs reported are no longer issuing certificates and all have expired then okay, although I'd like to note that this requirement was in place long before they stopped issuing off the old subordinates and they've been out of compliance for years on these CAs.

I decided to check the new intermediate (https://crt.sh/?id=9975892) provided, but ran into one interesting issue immediately. The OCSP responder returns 403 when queried from Hong Kong (and presumably some other geographic areas). Instead of an OCSP response it returns a 403 + HTML page that says the site has been blocked by the network administrator (with a Dell SonicWALL appliance logo). That seems extremely undesirable, although I'm not aware of a rule prohibiting it.

Switching my source IP to the US I was able to confirm that it is properly returning unknown status for invalid serials.
Francesc: do you know why your OCSP responder is not available from IP addresses in Hong Kong?

Gerv
Dear Gervase,

This is an expected behaviour. In our aim to increase security we, by default, block traffic from high risk areas or regions if we do not have any customer on them.

This does not mean that we can not open traffic from such areas, if needed.

Thank you,
(In reply to Francesc Ferrer from comment #7)
> This is an expected behaviour. In our aim to increase security we, by
> default, block traffic from high risk areas or regions if we do not have any
> customer on them.

Francesc: I can see that makes sense for e.g. your main site webservers, but the people using your OCSP and CRL servers are not _your_ customers, they are your customers' customers. And they could be located anywhere in the world. There is no rule which says that to visit a website using a Consorci certificate, I have to be in e.g. Europe.

So please can you open access to your revocation servers worldwide?

Gerv
Francesc: comment #8 is waiting for your response.

Gerv
Flags: needinfo?(fferre)
(In reply to Gervase Markham [:gerv] from comment #9)
> Francesc: comment #8 is waiting for your response.
> 
> Gerv

Dear Gervase, 

We opened the access to our revocation servers worldwide. 

Kind regards,
The following report shows 7 Consorci CAs that still exhibit the problematic behavior:

https://crt.sh/ocsp-responders?randomserial=Good&trustedBy=Mozilla&trustedFor=Server%20Authentication&trustedExclude=constrained,expired,onecrl&randomserial=Good&sort=2&dir=v

One of them is the root, so I don't see how Consorci can claim BR compliance. Francesc, please explain why you believe that the EC-ACC root is exempt from this requirement.

For the sub-CAs, I propose that we add them all to OneCRL.
Dear Mr. Thayer, 

We claim for BR compliance on our only subCA issuing SSL certificates to end-entities that is EC-SECTORPUBLIC. All the rest of subCAs and
Flags: needinfo?(fferre)
Dear Mr. Thayer, 

We claim for BR compliance given that our only subCA issuing SSL certificates to end-entities (EC-SECTORPUBLIC) has its OCSP server responding accordingly to BR. As stated in comment#3 in this bug, all the rest of subCAs and Root CA are either not issuing SSL certificates to end entities or have all them expired already. Do we have implemented the system correctly according to BR compliance, then, in this case?. 

Thank you,
No, this is not BR compliant because your root must also comply with the BRs. In addition, Mozilla policy requires any subordinate CA that is capable of issuing SSL certificates (and it appears that all of your subordinates CAs are technically capable of issuing SSL certificates) to comply with the BRs. It does not matter that you are no longer issuing SSL certificates from these subordinate CAs - only that it is possible to do so and that the root is trusted for websites in Mozilla's program (it is). You will need to:
1. Fix the OCSP responder for the EC-ACC root
2. Either fix the OCSP responders for all of the subordinate CAs, or request that the non-BR compliant subordinate CAs be added to OneCRL so that they are not trusted by Mozilla products for SSL.

Please respond with a date by which you will have this problem corrected.
Flags: needinfo?(fferre)
Changing QA contact per https://bugzilla.mozilla.org/show_bug.cgi?id=1438254
QA Contact: gerv → wthayer
Francesc: please respond to comment #14
Wayne: CAs requested to be added to OneCRL: https://bugzilla.mozilla.org/show_bug.cgi?id=1450805
OSCP to be fixed:

- The Root's OCSP, that will be whitelisted by mid April:

EC-ACC: https://crt.sh/?caID=77 

- The EC-CIUTADANIA's CA issues smime certs and we aim to request smime flag activation soon. Although not technically constrained and not issuing ssl certs, will be whitelisted on a date to be stated that will be reported soon. 

EC-CIUTADANIA: https://crt.sh/?caID=13344 

QUESTION: related to the configuration of EC-CIUTADANIA's OCSP in whitelist mode and if it took longer to configure: is it possible to add the ec-ciutadania CA (or any other CA) for a period of time and remove it later on?. In this case OneCRL would not be working as a normal CRL as we know it..
(In reply to Francesc Ferrer from comment #17)
> Wayne: CAs requested to be added to OneCRL:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1450805

Thank you.

> OSCP to be fixed:
> 
> - The Root's OCSP, that will be whitelisted by mid April:
> 
When you say it will be "whitelisted", do you mean that it will stop responding "good" for unknown certificates?

> EC-ACC: https://crt.sh/?caID=77 
> 
> - The EC-CIUTADANIA's CA issues smime certs and we aim to request smime flag
> activation soon. Although not technically constrained and not issuing ssl
> certs, will be whitelisted on a date to be stated that will be reported
> soon. 
> 
> EC-CIUTADANIA: https://crt.sh/?caID=13344 
> 
> QUESTION: related to the configuration of EC-CIUTADANIA's OCSP in whitelist
> mode and if it took longer to configure: is it possible to add the
> ec-ciutadania CA (or any other CA) for a period of time and remove it later
> on?. In this case OneCRL would not be working as a normal CRL as we know it..

No, we do not remove OneCRL entries until after the certificate has expired. However, OneCRL only affects certificates used in Firefox, so EC-CIUTADANIA can still be used for email certificates even if it is added to OneCRL. If this is acceptable, please add EC-CIUTADANIA to the OneCRL bug.
Flags: needinfo?(fferre)
>> OSCP to be fixed:
>> 
>> - The Root's OCSP, that will be whitelisted by mid April:
>> 
> When you say it will be "whitelisted", do you mean that it will stop responding "good" for unknown certificates? 

Yes. Scheduled to be in production by 20th of April. 

>> EC-ACC: https://crt.sh/?caID=77 
>> 
>> - The EC-CIUTADANIA's CA issues smime certs and we aim to request smime flag
>> activation soon. Although not technically constrained and not issuing ssl
>> certs, will be whitelisted on a date to be stated that will be reported
>> soon. 
>> 
>> EC-CIUTADANIA: https://crt.sh/?caID=13344 
>> 
>> QUESTION: related to the configuration of EC-CIUTADANIA's OCSP in whitelist
>> mode and if it took longer to configure: is it possible to add the
>> ec-ciutadania CA (or any other CA) for a period of time and remove it later
>> on?. In this case OneCRL would not be working as a normal CRL as we know it..

> No, we do not remove OneCRL entries until after the certificate has expired. However, OneCRL only affects certificates used in 
> Firefox, so EC-CIUTADANIA can still be used for email certificates even if it is added to OneCRL. If this is acceptable, 
> please add EC-CIUTADANIA to the OneCRL bug.

Finally Ec-ciutadania has been requested to be added into oneCRL : https://bugzilla.mozilla.org/show_bug.cgi?id=1450805#c2
EC-ACC OCSP responder is working in conformity to BR since today as you can see here:

https://crt.sh/ocsp-responders

What would be the next steps to take in order to solve this issue?. Would it be sufficient in adding the rest of subCAs into OneCRL as we are requesting in the other tiquet:

https://bugzilla.mozilla.org/show_bug.cgi?id=1450805

Thank you,
Flags: needinfo?(wthayer)
(In reply to Francesc Ferrer from comment #20)
> EC-ACC OCSP responder is working in conformity to BR since today as you can
> see here:
> 
> https://crt.sh/ocsp-responders
> 
> What would be the next steps to take in order to solve this issue?. Would it
> be sufficient in adding the rest of subCAs into OneCRL as we are requesting
> in the other tiquet:
> 
> https://bugzilla.mozilla.org/show_bug.cgi?id=1450805
> 
> Thank you,

I've asked a question in that bug. Once answered, we will work on these OneCRL additions.
Flags: needinfo?(wthayer)
(In reply to Wayne Thayer [:wayne] from comment #21)
> (In reply to Francesc Ferrer from comment #20)
> > EC-ACC OCSP responder is working in conformity to BR since today as you can
> > see here:
> > 
> > https://crt.sh/ocsp-responders
> > 
> > What would be the next steps to take in order to solve this issue?. Would it
> > be sufficient in adding the rest of subCAs into OneCRL as we are requesting
> > in the other tiquet:
> > 
> > https://bugzilla.mozilla.org/show_bug.cgi?id=1450805
> > 
> > Thank you,
> 
> I've asked a question in that bug. Once answered, we will work on these
> OneCRL additions.

Answered: https://bugzilla.mozilla.org/show_bug.cgi?id=1450805#c4

Thank you,
The non-compliant CA certificates have been added to OneCRL. Marking this resolved.
Status: NEW → RESOLVED
Closed: Last year
Resolution: --- → FIXED
Francesc: The ACC root and SectorPublic intermediate are again exhibiting this problem: https://crt.sh/ocsp-responders?randomserial=Good&trustedBy=Mozilla&trustedFor=Server%20Authentication&trustedExclude=constrained,expired,onecrl&randomserial=Good&sort=2&dir=v

Please file an incident report explaining why this continues to occur and what will be done to prevent it.
Status: RESOLVED → REOPENED
Flags: needinfo?(fferre)
Resolution: FIXED → ---
We were aware of the issue through our monitoring system.
It took more than expected to solve it.
It was solved finally today June the 1st at 8:00 CET.
The service was up and running although responding "Good" for non existing certs.
We will provide more details in an Incident Report
Flags: needinfo?(fferre)
Flags: needinfo?(fferre)
Dear Kathleen, 

A bug has been filled for this issue: https://bugzilla.mozilla.org/show_bug.cgi?id=1467110

Looking forward for your comments. 

Thank you,
Flags: needinfo?(fferre)
Duplicate of this bug: 1467110
The following incident report was posted in bug 1467110:

Incident report:

1.- How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Our monitoring system at 1:30 AM 1st of June.

2.- A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

From our point of view this is a severity 3 issue, since the service is available, giving the right answer except when it is asked about a non existing certificate.
Severity 3 definition: Failure of one or several functions of the service without presenting an immediate significant effect on the quality of service, or affecting a very limited number of users and not having a global significance: absence or presentation of misleading data, problems in the design of the pages , etc.
Severity 3 SLA:
•	Time to answer: 8 hours
•	Time to diagnose: 2 working days
•	Time to solve: 2 working days
Our SysOp started to work on it at 6:00 AM CET, 1st of June
It was solved at 8:00 AM CET

3.- Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

It was not a problem of issuance neither a security issue.
Our service is configured so if the OCSP responses become slow or stop due to DB issues, automatically changes to a less demanding system, reading the Certificate Status from the CRL. Despite this means that in the meantime when we solve the problem, at least all customers have the service available.

4.- A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

There were no certificates involved

5.- The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

There were no certificates involved

6.- Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

It was not a mistake but a problem of syncing between the master and slave DB since we have added a redundant SAN.

7.- List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a time-line of when your CA expects to accomplish these things.

The addition of the redundant SAN is a measure to add HA even if the hardware fails, but the syncing was an issue already solved and correctly configured.
Francesc: your response leads me to the conclusion that your OCSP system will violate the BRs whenever the primary system becomes slow or unavailable. Given the well-documented risk of responding "good" to unknown certificates, do you have specific plans and dates to upgrade your backup system to become compliant?
Flags: needinfo?(fferre)
(In reply to Wayne Thayer [:wayne] from comment #29)
> Francesc: your response leads me to the conclusion that your OCSP system
> will violate the BRs whenever the primary system becomes slow or
> unavailable. Given the well-documented risk of responding "good" to unknown
> certificates, do you have specific plans and dates to upgrade your backup
> system to become compliant?

Dear Wayne, 

The redundant SAN mentioned in "https://bugzilla.mozilla.org/show_bug.cgi?id=1467110#c1" point #7, was deployed and is working 100x100. Therefore, since then, we have hardware based high availability.

Additionally, the backup system it's been configued in order to respond properly. That is, responding “Unknown” for not issued certificates, according with the requirements.

Thank you,
Flags: needinfo?(fferre)
Status: REOPENED → RESOLVED
Closed: Last year11 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.