1731887 - Entrust: Test Website Certificates Expired

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Bruce Morton]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36

Steps to reproduce:

Entrust has reviewed recent incident reports https://bugzilla.mozilla.org/show_bug.cgi?id=1730291 and https://bugzilla.mozilla.org/show_bug.cgi?id=1726333. These reports provide incidents where test website certificates have expired.

Entrust has also had similar incidents but did not understand that a report was required. We understand that was an error and will update our policy to address this problem.

Entrust will provide a full incident report by 28 September 2021.

Comment 1

[Matthias]

Entrust has also had similar incidents but did not understand that a report was required.

The MRSP state that any BR non-compliance is an incident, and that incidents should be filed in Bugzilla (MRSP s2.4 "Incidents"). Those requirements do not seem too difficult to understand / seem quite difficult to misunderstand, so it would be helpful to understand how this misunderstanding came to be and how this was not detected earlier. Could you share that information in the incident report or in a seperate comment?

Comment 2

[Bruce Morton]

Agreed, it is not too hard to understand. Please note incident reports have moved from a practice to a policy over the years. The practices which we originally put into place were focused around miss-issued certificates. We implemented pre-issue and post-issuance linting to prevent/monitor miss-issued certificates. We did learn over the years to update our practices for other reasons such as late audit reports and failure to revoke. Unfortunately, we did miss that expiring test website certificates would be cause for an incident. However, we did recognize by monitoring the Incident Reports that expired test website certificates were a reason for an incident. We will ensure that our updated practices will cover all compliance requirements to the BRs, EVGs, Mozilla Policy and other root store policies.

Comment 3

[Bruce Morton]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

Entrust monitors the test websites and was aware of the expired certificates on the day they expired. Entrust was also advised through a Chromium bug https://bugs.chromium.org/p/chromium/issues/detail?id=1061530 that a certificate https://crt.sh/?id=1653291880 on a root test web page had expired.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

7 September 2019: Entrust was aware that 8 certificates supporting AffirmTrust root test sites had expired.
25 March 2020: Entrust was aware that 2 certificates supporting Entrust root test sites had expired.
13 May 2020, 20:15 EDT: Entrust was advised through a Chromium bug that a test certificate had expired.
29 May 2020: Entrust completed a project to migrate all test CAs to a new software architecture with an updated HSM and in a new data center.
29 May 2020: All expired test certificates were reissued.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

This incident did not affect certificate issuance.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

Ten test website certificates as listed below were expired.

5. The complete certificate data for the problematic certificates.

The following certificates for Entrust CA’s test websites expires without being renewed:

https://crt.sh/?id=1653291880
https://crt.sh/?id=1653292455
https://crt.sh/?id=270700971
https://crt.sh/?id=924603491
https://crt.sh/?id=924603465
https://crt.sh/?id=924603481
https://crt.sh/?id=924603432
https://crt.sh/?id=924603471
https://crt.sh/?id=924603436
https://crt.sh/?id=924603472

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The problem was known at time of expiration. The priority to resolve the problem in September 2019 was low as the root certificates had been embedded and there were no Application Software Suppliers requested to test their software with the subscriber certificates. The problem was further extended as Entrust was in a process of migrating the CAs to a new software platform with the CA keys also migrated to a later version of the HSM product. The old CA configuration required manual test certificate issuance which had caused previous miss-issuances, so was not used to correct the issue. Due to the complexity of the change, the priority of test CAs, restrictions due to COVID-19, 2 more certificates also expired in March 2020 before the CA project was complete.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

The test CAs are now operated on the same software platform as the customer CAs. The certificates are also issued from an enterprise account, where 60-, 30- and 10-day expiry notifications are provided to the web server administrators. In addition, the operations team has implemented monitoring for test website certificates. We believe that we have implemented controls to ensure test website certificates will not expire.

Comment 4

[Ben Wilson]

I appreciate the insight that this incident report provides into the issue of expired certificates for test websites. (One website should have an expired certificate, one should have a valid certificate and another website should have an unexpired, but revoked certificate installed.) A useful resource is https://crt.sh/test-websites. I doubt that all CAs have expiration alerts in place - hopefully those that read this report will implement something similar (and ensure that such alerts are not inadvertently disabled, as reported in Bug #1730291). It seems that this incident occurred and was remedied in 2019-2020 and that this bug can be closed. However, are there additional questions, comments, suggestions, or lessons that can be learned from a postmortem of this incident?

Comment 5

[Ben Wilson]

I'll close this on next Wed. 20-Oct-2021.