1581183 - Google Trust Services: CRL handling of expired certificates not fully compliant with RFC 5280 Section 3.3

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Wayne Thayer]

Andy Warner posted the following incident report to m.d.s.p.:

This is an initial report and we expect to provide some additional details and the completion timeline after a bit more verification and full deployment of in-flight mitigations. We are posting the most complete information we have currently to comply with Mozilla reporting timelines and will follow-up with additional details soon.

1. How your CA first became aware of the problem and the time and date.

While performing an internal review and assessment of the CRL generation system for Google Trust Services' GTS CA 1O1 on August 16, 2019, it was discovered that the CRL generation service did not include CRL entries of expired certificates. The periodic job only considered certificates with valid lifetimes. This does not conform to RFC 5280 Section 3.3 which states that “An entry MUST NOT be removed from the CRL until it appears on one regularly scheduled CRL issued beyond the revoked certificate's validity period.” We expect that few, if any, clients have been impacted. For a client to be impacted they would have to: clock skewed to a time before the not-after field of the certificate; and have a CRL published after expiration dropping the revoked certificate.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

August 16, 2019 15:00 UTC - Reviewer realizes that CRL will not publish for one update past expiration
August 16, 2019 16:00 UTC - Reviewer checks for other issues & talks to peers to confirm problem
August 16, 2019 17:00 UTC - Bug is filed to fix the issue with a proposed design fix
August 16, 2019 23:30 UTC - Fix is sent for review
August 20, 2019 16:00 UTC - Remediation work is discussed & assigned
August 20, 2019 18:00 UTC - Query to inspect revoked certificates is created and sent to be run by production team for initial analysis.
August 21, 2019 10:40 UTC - Production team runs query and returns result
August 21, 2019 15:00 UTC - Reviewer analyzes data
August 21, 2019 20:30 UTC - Reviewer asks for a follow up query to ascertain if any certificates did not make it onto the CRL
August 22, 2019 07:00 UTC - Initial attempt at updating test systems with fix.
August 22, 2019 09:00 UTC - Updating of test systems aborted due to (unrelated) issues.
August 22, 2019 07:00 UTC - Production team runs query for CRLs that may have missed a certificate
August 22, 2019 15:00 UTC - Reviewer ascertains that certificates under question were on a CRL
August 26, 2019 11:00 UTC - Second attempt at updating test systems with fix.
August 26, 2019 13:00 UTC - Test systems updated, confirmed integrity of fixed software.
August 27, 2019 09:00 UTC - Confirmed fix is effective on test systems.
August 27, 2019 10:00 UTC - present: Ongoing staged deployment to production systems. Should complete fully by September 3, 2019 17:00 UTC (slightly extended window due to push policies around holiday weekends. The rollout was staged in accordance with Google's standard rollout procedures.)

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem.

The affected CA software has been patched. It now populates expired certificates in the CRL for 7 days after their expiration to ensure they appear in at least one regularly issued CRL update. Automated testing was added as part of the same patch to check that revoked certificates are kept in the CRL. The patch was developed, tested, reviewed and landed within the codebase by August 19, 2019. The CRL entry removal bug has been fully remediated.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

Investigation began on August 20, 2019 to discover the potential impact of the logic bug. The CRL generation had contained the bug since its inception, affecting all issuance under GTS 1O1 since March 2018. There were 200,263 revoked certificates during that time window. Almost all certificates were for internal monitoring specific to checking revocation. The few non-monitoring certificates were all revocations by clients following rotation of certificates and not due to compromises.

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

crt.sh IDs to follow, waiting on confirmation that the 2 test certificates mentioned below are the only cases where the issue was surfaced.

The team looked for revoked certificates from first issuance that never appeared within a published CRL from operation of CA until August 21, 2019. It was detected that 2 test certificates which were revoked within 2 standard CRL update windows; but both were present in at least one CRL before expiration.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

It is believed that this went unnoticed for so long due to the majority of requirements being located in CA/B BR 4.10.1 & RFC 5280 section 5. The extra requirements inside RFC 5280 3.3 were most likely overlooked due to the explicit wording of the BR - Revocation entries on a CRL or OCSP Response MUST NOT be removed until after the Expiry Date of the revoked Certificate - during initial development and subsequent reviews.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

Our existing internal review program has proven effective in discovering issues related to certificates. The program is already in place and will continue.

Comment 1

[Wayne Thayer]

The original report was posted on 30-August 2019. Today the following update was posted to m.d.s.p.:

A quick follow-up to close this out.

The push to fully address the issue was completed globally shortly before 16:00 UTC on 2019-09-02.

After additional review, we're confident the only certificates affected were these two:
https://crt.sh/?id=760396354
https://crt.sh/?id=759833603

Google Trust Services considers this matter fully addressed. We will of course continue our ongoing internal review program, but no other work or information is outstanding at this point.

--
Andy Warner
Google Trust Services

Comment 2

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.