1731164 - Google Trust Services: CRL validity period set to expected value plus one second

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Cade Cairns]

This is a preliminary incident report.

On 2021-09-10 at 16:04 UTC, one of our Compliance Engineers was conducting a periodic review of our CA systems and identified an issue with the validity period of CRLs used by our secondary CA system running EJBCA.

The engineer identified that in our configuration the validity period was set to be equal to 10 days, and due to how this is configured in EJBCA, the actual validity period of CRLs from the affected systems may be 10 days plus one second.

RFC5280 section 6.3.3 states that a CRL is retrieved if the current time is after the CRL next update field. Therefore the validity period is inclusive of the final second. Although the Baseline Requirements do not explicitly state the validity period duration to be inclusive of the last second in the same way they do for OCSP and certificates, we believe this is against the spirit of the BRs.

We have deployed a fix as of 2021-09-10 at 21:53 UTC and the fix was deployed to all systems as of 2021-09-16 at 18:32 UTC. A full incident report is being worked on and will be provided by 2021-09-24.

Comment 1

[Cade Cairns]

1. How your CA first became aware of the problem

On 2021-09-10 at 16:04, one of our Compliance Engineers was conducting a periodic review of our CA systems and identified an issue with the validity period of CRLs used by our secondary CA system running EJBCA.

2. A timeline of the actions your CA took in response.

YYYY-MM-DD (UTC)	Description
2020-07-16 17:00	Voting for CABF ballot SC31 (Browser Alignment ballot) ends.
2020-07-30 16:00	Initial review of SC31 was conducted during a weekly GTS compliance review meeting.
2020-09-01 00:00	Version 1.7.1 of the Baseline Requirements brings a requirement into effect that the validity period of certificates and the validity interval of OCSP are inclusive of the final second.
2020-09-03 22:20	IdenTrust files Bug 1663080 on violation of the certificate lifetime limits by one second
2020-09-28 09:56	Dhimyotis/Certigna files Bug 1667744 on violation of the certificate lifetime limits by one second
2020-10-28 10:55	EJBCA developer PrimeKey sent an email to the dev-security-policy mailing list to alert the community that EJBCA does not calculate certificate and OCSP durations in conformance with RFC 5280 and BRs 1.7.1.
2020-11-04 15:30	The EJBCA issue report was reviewed during a weekly GTS compliance review meeting where the determination was that our practices aligned to the BRs.
2021-05-02 08:39	KIR SA files Bug 1708965 on violation of the certificate lifetime limits by one second
2021-06-09 07:15	Let’s Encrypt files Bug 1715455 on violation of the certificate lifetime limits by one second
2021-09-10 16:04	A Compliance Engineer identifies that the CRL validity period is set to 10 days plus one second and creates a tracking bug to investigate the issue.
2012-09-10 19:47	A patch rollout to our secondary CA system is initiated.
2021-09-10 20:45	New CRLs were published.
2021-09-16 18:32	The patch rollout is deployed to all systems.

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident.

A fix was applied to correct the CRL validity period on our secondary CA system and new CRLs were published on 2021-09-10 at 20:45 UTC. The issue did not result in misissuance of certificates, so certificate issuance was not stopped. Between the time of discovery and the patch, no additional CRLs were issued.

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.)

N/A - the issue did not result in misissuance of certificates.

5. In a case involving certificates, the complete certificate data for the problematic certificates.

N/A - the issue did not result in misissuance of certificates.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

When version 1.7.1 of the BRs was published on 2020-08-20, it clarified that the validity period of a certificate and the validity interval of an OCSP response were inclusive of the last second. While this clarification aligned with RFC 5280 section 4.1.2.5 for certificates and RFC 6960 section 2.4 for OCSP, this method of calculating duration was not well-understood by issuers and resulted in more than one public incident report after the update to the BRs was published.

On 2020-07-30 we began discussing CABF ballot SC31, which introduced these changes, during our weekly compliance review meeting. Since the ballot bundled a large number of changes, discussion and review took place over several weeks and concluded at a meeting on 2020-08-26.

At the time we began our investigation, we reviewed the certificate profiles and OCSP settings for both our primary CA platform, which runs our custom ACME implementation, and our secondary CA platform, which runs EJBCA and serves as a backup. The maximum lifetime of certificates issued by our primary and secondary CA platforms was 90 days and 364 days, respectively, well under the BRs limit of 398 days. The validity interval of our OCSP responses was also determined to be compliant with the BRs. Despite the focus on validity periods, we did not thoroughly review the CRL validity period on the secondary platform.

On 2020-09-03, Bug 1663080 was created. We again assessed our certificate profiles and determined that our maximum lifetimes mitigated the issue. Similar assessments were made for Bug 1667744, Bug 1708965, and Bug 1715455. When the developers of EJBCA posted an email to the dev-security-policy list on 2020-10-28, we reviewed the issue report during our weekly compliance review meeting on 2021-11-04 and came to the same conclusion as our past investigation, without reviewing the CRL validity period.

On 2021-09-10, one of our compliance engineers was conducting a periodic review of our CA systems and identified that the validity period of the CRLs used by our secondary CA platform may be incorrectly configured. The engineer identified that in our configuration the validity period was set to be equal to 10 days, and due to how this was configured in EJBCA, the actual validity period of CRLs from the affected systems may be 10 days plus one second.

RFC5280 section 6.3.3 states that a CRL is retrieved if the current time is after the CRL next update field. Therefore the validity period should be considered to be inclusive of the final second. Although neither RFC5280 nor the BRs explicitly state that the next update field is inclusive of the next second, we believe that the setting resulted in it being equal to 10 days plus one second and needed to be corrected.

The limited review conducted for the CRL validity period of our secondary CA platform was an oversight due in part to the fact the original bugs did not reference this as a consideration, coupled with systemic issues described in Bug 1708516 Comment 44. As described in that bug, we have made many changes to increase engineering participation in compliance, which among other things has resulted in a more rigorous review process and the identification of this issue.

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

We believe that this should be clarified in future versions of the BRs. We plan to engage in a discussion at the CA/B Forum in order to determine how to amend them.

We believe based on our recent changes to our compliance program, specifically the creation of the Compliance Engineering function described in Bug 1708516, that we would have caught this issue when investigating the other validity periods. We are continuing to build on these changes and have expanded this role to include additional people.

Comment 2

[Cade Cairns]

Google Trust Services is monitoring this bug for any additional updates or questions.

Comment 3

[Cade Cairns]

Google Trust Services is monitoring this bug for any additional updates or questions.

Comment 4

[Cade Cairns]

Google Trust Services is monitoring this bug for any additional updates or questions.

Comment 5

[Cade Cairns]

Dear Ben,

We have completed all remediation steps, and it appears that there are no concerns raised by the community. We believe that we can move forward by marking the bug as Fixed. Can you please advise? We are making this request following your guidance at https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/BOwcbWbZTg0/m/ceLXGwgEBwAJ

We had an action item to engage in a discussion about this issue with the CA/B Forum, but a draft ballot has already been proposed by other community members to clarify the maximum permitted validity interval of a CRL in the BRs: https://lists.cabforum.org/pipermail/validation/2021-October/001705.html

Thank you

Comment 6

[Ben Wilson]

I'll schedule to close this next Wed. 20-Oct-2021.