Open Bug 1577652 Opened 23 days ago Updated Yesterday

Let's Encrypt: OCSP Responder Returned "Unauthorized" for Some Precertificates

Categories

(NSS :: CA Certificate Compliance, defect)

defect
Not set

Tracking

(Not tracked)

ASSIGNED

People

(Reporter: jsha, Assigned: jsha)

Details

(Whiteboard: [ca-compliance])

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36

Steps to reproduce:

On 2019.08.28 we (Let's Encrypt) read Apple’s bug report at https://bugzilla.mozilla.org/show_bug.cgi?id=1577014 about DigiCert’s OCSP responder returning incorrect results for a precertificate. This prompted us to run our own investigation. We found in an initial review that for 35 of our precertificates, we were serving incorrect OCSP results (“unauthorized” instead of “good”). Like DigiCert, this happened when a precertificate was issued, but the corresponding certificate was not issued due to an error.

We’re taking these additional steps to ensure a robust fix:

  • For each precertificate issued according to our audit logs, verify that we are serving a corresponding OCSP response (if the precertificate is currently valid).
  • Configure alerting for the conditions that create this problem, so we can fix any instances that arise in the short term.
  • Deploy a code change to Boulder to ensure that we serve OCSP even if an error occurs after precertificate issuance.
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true

Jacob,

Thanks for the quick investigation and response. Was this something in the open-source Boulder code? Could you perhaps provide pointers to it?

In particular, it'd be interesting to understand the flow and sequencing of a pre-certificate being signed, an OCSP response being generated, the pre-certificate being submitted to logs, and the actual final certificate being issued. While normally this would be a question for the CA to report back, the open-source nature hopefully makes it easier to find answers in the source to the questions and any follow-ons?

Assignee: wthayer → jsha
Flags: needinfo?(jsha)
Whiteboard: [ca-compliance]

Yes, this was in the open-source Boulder. These were the specific error cases that could result in a precertificate not having OCSP available:

https://github.com/letsencrypt/boulder/blob/fe23dabd69ce5f07abb901faa63b5a0959978753/ra/ra.go#L1275-L1282

However, it's more informative to think of this as a design flaw than to look at specific error lines. Boulder treats OCSP responses as a property of the final certificate. Instead, we will probably need to move things around to treat OCSP responses as a property of a precertificate.

Flags: needinfo?(jsha)

Right. I was wanting to understand the overall design, and thought it might be useful to look at the source for how the design worked, than to try to ask an increasing amount of questions that has you describe the design in psuedo-code.

I think your summary of the design flaw sounds spot on. Perhaps naively, I would have assumed that the process was:

  • Assign a serial number and a certificate "to be issued"
  • Confirm it's recorded in durable storage (i.e. you haven't forgot the cert)
  • Actually sign it (as a pre-certificate). This is done after database recording, so that you know what you "were going to sign" rather than only what you actually signed, in case there's any issues in recording that you signed something
  • As a pre-certificate, produce and distribute status information for this certificate
  • Once the status information is distributed, log the certificate to one or logs
  • Once sufficient logs have returned, record and issue the final certificate
  • Release the final certificate to the Subscriber

This flow is what I assumed the 'legacy' flow for CAs in a pre-CT world did, namely:

  • Assign a serial number and a certificate "to be issued"
  • Confirm it's recorded in durable storage (i.e. you haven't forgot the cert)
  • Actually sign it (as a certificate)
  • Produce and distribute status information for this certificate
  • Release the certificate to the Subscriber

Part of this sequence is to try to avoid issues like Bug 1524815, Bug 1462844, or Bug 1526154 - where things fall between the cracks and lead to non-compliance.

Of course, if I've misunderstood considerations, that'd be useful to know. As I said, it was a somewhat naive assumption I'm making as to how things work.

Hi Ryan,

I have a variation:

  • Assign a serial number and a certificate "to be issued"
  • Sign it as a pre-certificate
  • As a pre-certificate, do not distribute status information for this certificate, leave it as "unknown" as the final state of the transaction is not yet known (BR 4.9.10)
  • Log to CT logs, get SCTs and audit log everything
  • Issue the final certificate
  • Choice, if:
    ** the final certificate should not be distributed (not enough SCTs, failed linting, or other cause), set status to a revoked state and do not release to subscriber
    ** the final certificate should be distributed, set status to good state and release to subscriber

A benefit of not distributing status for the pre-certificate is that is saves on latency. It also keeps the issuer/serial unusable (anything-but-good) for practical matters as the cert can/should not yet be used (it has not been issued in real bytes yet). CAs publishing time to OCSP may span from milliseconds, to higher values depending on volume, load and technology used.
Latency is a concern, and flipping statuses when there is no need to takes unnecessary time, and also introduces additional possible error paths.

For the non-TLS use-case there are other processes as well (issuing in on-hold state), but here we limit ourselves to the TLS use case of course.
Even though I have no reference, I can envision TLS use cases where you want to make sure CDN caches are populated with OCSP responses before releasing the certificate to the subscriber making the "distribute status" step a process in it's own.
(validation/linting can be done at any of the stages).

The reason for the suggested ordering is exactly as you mentioned: to ensure clients that use OCSP stapling can obtain a fresh, usable status once they receive the certificate. The presumption here is that there is both latency, and more importantly for CAs (and related to this bug), complexity in getting the HTTP caching later correctly configured.

Adopting a flow as you suggest is understandably easier for the issuance software, but it moves the complexity to the CA interacting with the CDN and the Subscriber tasked with caching. This is because if the unknown status is not evicted, or has an unduly long TTL, then Subscribers will need to continually refresh and not productionize the certificate until the newly signed response is available.

Admittedly, a common path I’ve seen deployed is to simply set a default handler at the CDN to map any “unknown to the CDN” serials into a default state (Unknown or simply 404 are both valid and popular). In that scenario, there is only one publication event, and thus fewer state transitions. However, because CAs must monitor such requests, I suspect it’s easier to publish precisely to reduce the false negative.

Note that, for clarity sake, “failed linting” is hopefully an edge case, as any linting should have been done and successful well before the serial and TBSCertificate were assigned. It should only refer to linting the SCT extension OID, as linting that late is still misissuance.

Let me number the point in order to make my point clearer.

  1. Assign a serial number and a certificate "to be issued"
  2. Sign it as a pre-certificate
  3. As a pre-certificate, do not distribute status information for this certificate, leave it as "unknown" as the final state of the transaction is not yet known (BR 4.9.10)
  4. Log to CT logs, get SCTs and audit log everything
  5. Issue the final certificate
  6. Choice, if:
    ** the final certificate should not be distributed (not enough SCTs, failed linting, or other cause), set status to a revoked state and do not release to subscriber
    ** the final certificate should be distributed, set status to good state and release to subscriber

The time period between 1 and 6 is measured in milliseconds, or seconds if CT logs are slow. This is far lower than the merge delay of CT logs and will likely not help clients in obtaining fresh statuses. To be fair, this bug (Let's encrypt) did not affect any clients right, as no certificate was distributed to clients? Therefore are we not looking at different issues, where CT may not necessarily be the right tool?

  1. Yes, ensuring the intent to issue is fulfilled is a tool for CT
  2. No, ensuring that clients can use OCSP efficiently is not a tool for CT

The issue with OCSP in this bug looks like a symptom of the underlying issue (not issuing a certificate even though a pre-certificates was issued), and not the issue itself.
The process for ensuring fresh OCSP responses directly when they need it is definitely important. I see the need to avoid things falling through the cracks for sure (the bugs you linked), but we don't want to open up new cracks.

Recall that the original proposed mitigation is:

  • For each precertificate issued according to our audit logs, verify that we are serving a corresponding OCSP response (if the precertificate is currently valid).

This is very reasonable, and it seems we disagree on whether the Precertificate should return something like 404, or "unknown" - both of which may be cached by clients and intermediaries without the ability to flush - or whether it's desirable to have the 'correct' response being served.

Attempting the scenario as you've described requires that the CA, whether Let's Encrypt or other, ensure that they can reliably evict such responses from caches. One way is to guarantee that the response cannot be cached; but that, of course, increases load on the CA. Thus, any increase in the cache time increases the risk that, by the time #6 happens (using your numbering), the certificate cannot be safely used / does not have a correct response.

CT is entirely orthogonal to this. No one is proposing that CT be used to ensure clients can use OCSP efficiently. Rather, it's about understanding what commitments the CA is making when they log a precertificate. There are, for example, some CAs that delay generating an OCSP response until it's actually queried for (even for TLS cases), as a means of trying to reduce load on their signing servers; however, with CT, any client can monitor CT logs and query such certificates, thus defeating that load reduction.

I do not view not issuing a certificate, even though a pre-certificate was issued, to inherently be a bug. However, once a precertificate has been issued, all the necessary services for that response "should" be provisioned. This complexity is precisely why relying on Precertificates in OCSP is so fraught with peril; specifically, it requires the CA have a reliable mechanism to evict any caches that exist between them, the Subscriber, and the Client, as they transition from the "pre-precertificate" provisioned response to the "post-precertificate" provisioned response.

Again, #3 in Comment #6, places significant burden on CAs with respect to monitoring if #6 is failing, which is why the flow in Comment #3 was my naive understanding. I would normally think CAs would want to reduce risk.

(In reply to Ryan Sleevi from comment #7)

Recall that the original proposed mitigation is:

  • For each precertificate issued according to our audit logs, verify that we are serving a corresponding OCSP response (if the precertificate is currently valid).

This is very reasonable, and it seems we disagree on whether the Precertificate should return something like 404, or "unknown" - both of which may be cached by clients and intermediaries without the ability to flush - or whether it's desirable to have the 'correct' response being served.

I agree that this is reasonable. My assumption here was that monitoring the audit logs for pre-certificates and verifying against OCSP is something typically done after step 6 completes (1-6 completes in ~1 second). I know that CAs do monitor audit logs to find and mitigate any issues in the process, including with pre-certificates. The assumption is absolutely that the verification using audit logs results in a proper ok/revoked response of that issuerDN/serialNumber. Correct me if I'm wrong, but the verification is not done within steps 1-6?

Attempting the scenario as you've described requires that the CA, whether Let's Encrypt or other, ensure that they can reliably evict such responses from caches. One way is to guarantee that the response cannot be cached; but that, of course, increases load on the CA. Thus, any increase in the cache time increases the risk that, by the time #6 happens (using your numbering), the certificate cannot be safely used / does not have a correct response.

We're still talking about the time gap of a few seconds right?

CT is entirely orthogonal to this. No one is proposing that CT be used to ensure clients can use OCSP efficiently. Rather, it's about understanding what commitments the CA is making when they log a precertificate. There are, for example, some CAs that delay generating an OCSP response until it's actually queried for (even for TLS cases), as a means of trying to reduce load on their signing servers; however, with CT, any client can monitor CT logs and query such certificates, thus defeating that load reduction.

I do not view not issuing a certificate, even though a pre-certificate was issued, to inherently be a bug. However, once a precertificate has been issued, all the necessary services for that response "should" be provisioned. This complexity is precisely why relying on Precertificates in OCSP is so fraught with peril; specifically, it requires the CA have a reliable mechanism to evict any caches that exist between them, the Subscriber, and the Client, as they transition from the "pre-precertificate" provisioned response to the "post-precertificate" provisioned response.

Thanks for this clarification. I had, perhaps erroneously, though that the "intent to issue" mandated by CT really meant "issue" and I know others have made the same interpretation. Meaning that if not enough SCTs are received the certificate MUST be issued anyhow, and preferably revoked as it can not be used. If this is not the case that can affect implementation quite some.
The more details such as this included in the process description the better, it gives CAs better understanding what is ok and not. Not actually issuing can be considered a short cut, allowed or not, but it may actually save from some obscure corner case errors.

Again, #3 in Comment #6, places significant burden on CAs with respect to monitoring if #6 is failing, which is why the flow in Comment #3 was my naive understanding. I would normally think CAs would want to reduce risk.

I'm not working for a CA, but I'm sure they want to reduce risk.
I am aware of real issues of pre-certificates existing without certificates, but I have not heard about problems with OCSP cache poisoning due to pre-certificate logging and OCSP requests occurring before the certificate was issued. That doesn't mean it doesn't happen of course, I would like to hear about such cases in order to learn.

(In reply to Ryan Sleevi from comment #7)

I do not view not issuing a certificate, even though a pre-certificate was issued, to inherently be a bug. However, once a precertificate has been issued, all the necessary services for that response "should" be provisioned. This complexity is precisely why relying on Precertificates in OCSP is so fraught with peril; specifically, it requires the CA have a reliable mechanism to evict any caches that exist between them, the Subscriber, and the Client, as they transition from the "pre-precertificate" provisioned response to the "post-precertificate" provisioned response.

I didn't see any process suggestions suggesting "pre-precertificate" provisioned response. Was that theoretical, or did I missunderstand something?
Your process suggestion has "post-precertificate" provisioned response if I understand correctly, and mine I would call "post-certificate" provisioned response.

Just chiming in to say we've deployed the code change as of 2019-09-12 17:44 UTC. I'll be following up next week with a full incident report.

Summary: Let's Encrypt OCSP Responder Returned "Unauthorized" for Some Precertificates → Let's Encrypt: OCSP Responder Returned "Unauthorized" for Some Precertificates

We've reverted the flag controlling the code change mentioned above. We have a component in our signing service that handles timeouts from our database, locally queuing certificates that received a timeout and retrying until they are successfully written. Currently that component only handles certificates. When we made our fix for this issue, we decided to add precertificate handling for this queue as a follow-on task, rather than block the fix on it. In the meantime we would monitor our audit logs and ensure that any precertificates that failed initial insertion to the database would be manually inserted.

However, what we found in practice is that the original issue (precertificates lacking OCSP due to an error before final issuance) was significantly less common than the timed-out inserts, so we've reverted the flag. This both reduces the amount of manual work required and reduces the number of precertificates affected. We were seeing about 80 precertificates per day affected by timeouts, versus 0 affected by the original issue on most days.

Given that we've reverted the flag, we plan to continue our original interim remediation plan: Regularly check our audit logs against our available OCSP data, and fix any instances where a precertificate is missing an OCSP response.

Still working on the incident report for later this week.

Here’s our incident report per https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report.

  1. How your CA first became aware of the problem

(previously reported) On 2019-08-28 we (Let's Encrypt) read Apple’s bug report at https://bugzilla.mozilla.org/show_bug.cgi?id=1577014.

  1. A timeline of the actions your CA took in response.

    2019-08-28 21:44 (UTC) Incident began
    2019-08-29 02:15 Generated and stored OCSP for first batch of precertificates (based on errors in logs)
    2019-08-30 23:10 Generated and stored OCSP for second batch of precertificates (based on analysis of all issuance logs)
    2019-08-31 01:59 Generated and stored OCSP for third batch of precertificates (based on analysis of all issuance logs since the start of the incident)
    2019-09-05 and 2019-09-10: Generated and stored OCSP for additional batches of precertificates identified by our monitoring processes while we worked on deploying a code change.
    2019-09-12 17:44: Deployed code change to Boulder.
    2019-09-17 23:56: Reverted code change in Boulder (see comment above for details).

  2. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem.

Our fix is not currently live. On most days we are issuing zero certificates with this problem. On peak traffic days we may issue some certificates with this problem, but we have processes in place to ensure we manually generate OCSP for them until the fix is live again.

  1. A summary of the problematic certificates.

217 currently-valid precertificates were missing OCSP responses until we remediated them. The problem began 2018-03-29 when we first started issuing precertificates.

  1. The complete certificate data for the problematic certificates.

https://crt.sh/?serial=0317cc5db8f191d750c37303811f7b024836
https://crt.sh/?serial=031941c5f44189f5cb03b6433216f97aef9c
https://crt.sh/?serial=031962f7a1e8d7975212e43dbf2094b40639
https://crt.sh/?serial=031c12c9dbee223cd7fd8f8b98c570ba6443
https://crt.sh/?serial=0337467c94f32d679b730b3d34a695970c39
https://crt.sh/?serial=034840e5290b274fb7e52807648fdcb1f885
https://crt.sh/?serial=034a43354f89abc6eb1770f7b41b849af3d6
https://crt.sh/?serial=0361e48465403341964b1f8493789dd142fb
https://crt.sh/?serial=0368db78fab4bb497ab40beb292f40aaa41e
https://crt.sh/?serial=036cd5e8043199443711334bf8dbac2dae79
https://crt.sh/?serial=03721abb0e17b85e6b118a314e92e45af9a4
https://crt.sh/?serial=038304b353e468809409ccc6afe2093ecf5e
https://crt.sh/?serial=038ca3fa5781b441c9744dac343905290685
https://crt.sh/?serial=03922fdaf2516c3d6e530c8fc31c509b5eb2
https://crt.sh/?serial=0396894b3d0cf6eed3ba312cb1ec365321f1
https://crt.sh/?serial=039878458446d8dcb3d3dee0480d3e773db8
https://crt.sh/?serial=039d8f27af17aab5df4d165cb2213744b6b5
https://crt.sh/?serial=03a7ca0aaa5ec8c11d14c1f62ef0132ae941
https://crt.sh/?serial=03a93b93194e3ac914877c9173f1383a5816
https://crt.sh/?serial=03b2e747418e83bd344ee8ff04fbdcf0d69b
https://crt.sh/?serial=03b4d42368b81e257898d2172d7f73d9654a
https://crt.sh/?serial=03b81b4402da5678c93ffe9a5929b39e3012
https://crt.sh/?serial=03c414e39f7efb08e31cb6f04df4fd0c6a92
https://crt.sh/?serial=03c5fc914a877018afec29e5b52a31ceed7b
https://crt.sh/?serial=03cac9187d8c18b558fee6e73a68fff195c0
https://crt.sh/?serial=03cbc5e3a6d2333e22519186e818df6eec73
https://crt.sh/?serial=03cd0e72996d88922dbae36f65f056f94a91
https://crt.sh/?serial=03e3e39ddd133410a1f90a3d2507f914cb60
https://crt.sh/?serial=03eb23b6eb74a1949d0d0fe1886613c45ea0
https://crt.sh/?serial=046f1d31b717f364f85fefc235b9a625aab6
https://crt.sh/?serial=04746a637a525b847b009ee5fe4d147cc232
https://crt.sh/?serial=048030fd6ab64a487bd14f7b08cbf42c0387
https://crt.sh/?serial=048fc504b4f894ddff7ad3098a6ca8041524
https://crt.sh/?serial=0490739e4099d8954380d7df82afb16ce4a3
https://crt.sh/?serial=04a8b470007ae5b3f6a51272da9dbac1312b
https://crt.sh/?serial=03066440c75e8d20dd1a327e12611ed30433
https://crt.sh/?serial=041cbba9c8be11eba4735707648bff93c21f
https://crt.sh/?serial=0333334603a068b48b188da8c1c36fcfba2c
https://crt.sh/?serial=03569ba4488491645899b2bd1d0b66261a63
https://crt.sh/?serial=0334d963d3c6759a941bdf47b3037ce18531
https://crt.sh/?serial=03c44076a3652abbd73510bb30754c337935
https://crt.sh/?serial=03a68b1e05769defa8fb9a6c5180264236c7
https://crt.sh/?serial=035db7e037a2dfaaaf22cd0337891970ec0d
https://crt.sh/?serial=03201c96814f5bbf1b0c4797c47f8b880269
https://crt.sh/?serial=0358b53c4b65be945eaf6c10568da22a0d90
https://crt.sh/?serial=03f9e3496e72a73906a43ada842b5d36720f
https://crt.sh/?serial=03cf9cc3af0eb2cb0463bd192c22be4c03d1
https://crt.sh/?serial=0339d8081715d8207822fa1b93707916b82a
https://crt.sh/?serial=03dca4db71fd8b50874c90a9d883945bfeaf
https://crt.sh/?serial=0337b923a3fb71ffd73863f5265aaf56f16b
https://crt.sh/?serial=0320cf30235b827c62aa8de9bd88c718bb4a
https://crt.sh/?serial=0304672b2743e97d927c3d0b40d93d70caac
https://crt.sh/?serial=03eb674507f5da22bc53cf050dd9bd96bf06
https://crt.sh/?serial=0377ac7d5889a58a3258e64a2e9882cf2bd7
https://crt.sh/?serial=034d6f3d57f4394499c8779a2ee6acd7fd8c
https://crt.sh/?serial=0329a3cdeeca0a93e13da13eb0029b66374e
https://crt.sh/?serial=0376e523450e538ebc5f7686a6ada5247a12
https://crt.sh/?serial=03598b488e82bbf3ec501738dd3b74fc08d0
https://crt.sh/?serial=036a07887f5472780050d1c854343f48e65a
https://crt.sh/?serial=035161af938d7af26e9c51060f4813797702
https://crt.sh/?serial=03f5730f2d19cc09b8d685053f4a8a6cf701
https://crt.sh/?serial=03ec300a65a57dd882a137819b773b8cdb07
https://crt.sh/?serial=03813d039d96ee9b2b8aa0e6bb1db3f9bcb8
https://crt.sh/?serial=03e19f1af5c97b7460fa5c0571ec82e6889c
https://crt.sh/?serial=03a2a9165c7c75d5842e70e03ebca8fcc8fc
https://crt.sh/?serial=03c2238cb58522799353b4672caa391e4116
https://crt.sh/?serial=038efdfbfdee43b586084715f943d62ce650
https://crt.sh/?serial=03006a3e8e62eceb650bd2e107d9034d16b6
https://crt.sh/?serial=038d42c377991fd17e312584143f82bf26a1
https://crt.sh/?serial=03f4817eca0bc6b3ab0e297c426d2c4b13f7
https://crt.sh/?serial=03c61d06b216e24b93d53947ccbc8e3befbe
https://crt.sh/?serial=03f56c2ad01f2e3b8fb28bae377399803550
https://crt.sh/?serial=0364312378bf80d0e4e9bc1a77ff88f43189
https://crt.sh/?serial=03887914f8cf339dbb96521548c432c5f628
https://crt.sh/?serial=03145a256afc3689a4caa2fb81f0fc23e95d
https://crt.sh/?serial=038dd759ee3c888c000dab329be7f2e870af
https://crt.sh/?serial=038dfae80bb37121abb65ec2674701e0aa66
https://crt.sh/?serial=03ff599744dfb6c157742c3ad4d51863b7a9
https://crt.sh/?serial=03352bc411f6c6fda8d330e5a1f37cc75b78
https://crt.sh/?serial=030be29c6dbecfe5e1c3596a5a7c873970cd
https://crt.sh/?serial=03eb29e94fcc1aa5a2706c50c93481c1a595
https://crt.sh/?serial=037df32be6e668a38e564d0250cde3fc76f6
https://crt.sh/?serial=0388b8125b75b8d1f3144a6e69ee5a66a86f
https://crt.sh/?serial=03d0cf8eda551659c6f8ef81e44f749b5229
https://crt.sh/?serial=03d7fed2b194be808e7351c495075c8c6866
https://crt.sh/?serial=03bc27172e5be6ae63384f4542194da98dea
https://crt.sh/?serial=035526be03eac5a229f423a1f83bab4d8dd9
https://crt.sh/?serial=03b6e2b14e6fd76d5a0a0758233975454116
https://crt.sh/?serial=03b012c294d44a77e9a4371f4755d7487be4
https://crt.sh/?serial=03e9d56a10e04583ea9eb71d6d408a39ac32
https://crt.sh/?serial=0380fa95f0444e640f092fdea28254eff3a7
https://crt.sh/?serial=032f5fd04a4679cce80643f9d2dbf576eabd
https://crt.sh/?serial=0393b8c5ed45fe5ca9e066cbce49e0d70adc
https://crt.sh/?serial=0368485efcd7c12a4028bb9af70c4bd10216
https://crt.sh/?serial=03c8e406d4a6df0d2dc579a7c5ab17682c4e
https://crt.sh/?serial=034b9377107d560e92b69e6f4bd3a5d9fca9
https://crt.sh/?serial=03256c03218904f0a723434cbc6726132af0
https://crt.sh/?serial=0316d7706db23c29fef12eae7aa3e2679137
https://crt.sh/?serial=034661e60b747412a3d2c8f289f325e88777
https://crt.sh/?serial=0379927be5897bc860b367f6b2c4bc7401e4
https://crt.sh/?serial=031ecad74b05e14bbbd4958a1733f8bab993
https://crt.sh/?serial=03bcfa0da644d5ab9f707df532d9b184f410
https://crt.sh/?serial=031d93e1bf12f40ed026607237b4cd6325f1
https://crt.sh/?serial=030362b16151069609b0ccaa2462be623968
https://crt.sh/?serial=0378c429f552eb296c292f5f46241fa26938
https://crt.sh/?serial=035c6f38df925e75791f72da86cb0b09bc6a
https://crt.sh/?serial=0301d22c1d5d1f2dc790207469f4001bb02d
https://crt.sh/?serial=03388dbba5fe6da423edbbfa417d9dcbc7d9
https://crt.sh/?serial=0303bdd821362c623895d0c96d28f463cc27
https://crt.sh/?serial=0302ac60db35fea283378d9cc4cddd7166a4
https://crt.sh/?serial=0304c13af74fc231fd25958c8bd5193850db
https://crt.sh/?serial=030ceac6d44498b6a9a1ffc9e0c90d12bf06
https://crt.sh/?serial=030d52f8b70c0fdb2710d8824ade63b80453
https://crt.sh/?serial=031148672f3489c32217ad783208d4873902
https://crt.sh/?serial=0317d4641abebb35eb71dfc40b81a2e46e60
https://crt.sh/?serial=031ab1c6d743dd34fe9440cb405537f37b8d
https://crt.sh/?serial=03338443e41f9cb25ea3edf6f899470bc36b
https://crt.sh/?serial=0339e855ba98fa46691c134811d3eed1921d
https://crt.sh/?serial=033a74a2504d675d67a9b5922eb3fd375c81
https://crt.sh/?serial=034aa359fcf9faaf6e828953c7158775f61e
https://crt.sh/?serial=034b3d715746f5546b2a47ffd1000eb34b40
https://crt.sh/?serial=034bb460a0c87e796e1e7d290b4f8db7092c
https://crt.sh/?serial=034e8a44af20580618baadb5448462751d15
https://crt.sh/?serial=0353aa9bcd502ee0493024d835187108c7d7
https://crt.sh/?serial=035912e0f51e83f96c15638b1cf87a45f2aa
https://crt.sh/?serial=035c7e940b92bb4a121118d76ca0d85b4097
https://crt.sh/?serial=035f1cb7cc4dd8968ba67e3bac8059cf7db0
https://crt.sh/?serial=035f27a33ddc28fa5f4f1b9b27c2f1a57081
https://crt.sh/?serial=0361160cccf0f10487afbba4a1367a12d5bc
https://crt.sh/?serial=0362d1ff4557dd2fe8d6f3e7f0ec140e00c2
https://crt.sh/?serial=036b6f1b2ebb89bcbdfac41cc1d239cb2c87
https://crt.sh/?serial=036efd2fcc3bdb099a4fbab270a3b12cd702
https://crt.sh/?serial=03764e04da604db00ea5c2e58f49982543bb
https://crt.sh/?serial=0379dc355bcfd8a20732ccbfd60c4c5bb854
https://crt.sh/?serial=037ae644e628ecf5b034c9d699813c75c340
https://crt.sh/?serial=037b515d63dde559078e0cba0f8a09a743ed
https://crt.sh/?serial=037d97b768dae97f364f42a7544863d49291
https://crt.sh/?serial=03851c81fd4a95d1cbdb6fee92b18d867962
https://crt.sh/?serial=03889034493d9932a3ac67953efdbe23552f
https://crt.sh/?serial=03a63b5971fa2a74c187d00d14c4ab00e0c8
https://crt.sh/?serial=03aec8463e27f4256d6b0156f744a32dc787
https://crt.sh/?serial=03b15848dab1fa41f721b9c5f7e8ae502f44
https://crt.sh/?serial=03b1a66c53f37d9111794cbb52aceb3f190a
https://crt.sh/?serial=03b3bd98cbac2ee2b6d5dd46abf5a547dc9d
https://crt.sh/?serial=03b99d1befcb919a430d5810e7e2cd5eeadc
https://crt.sh/?serial=03bddd10051da45f340bd407880de98247b6
https://crt.sh/?serial=03c406195954f9023f9b3a65fd7204f1caef
https://crt.sh/?serial=03c5534bb1fc96afddacfb0be3f9e0e6512a
https://crt.sh/?serial=03c90315679b0e07f5e90b820115cb519618
https://crt.sh/?serial=03caa6e01adcb2424be724c7b11d47cd980a
https://crt.sh/?serial=03cde9f7478c5ce78a49bee70ad8c33ef2d3
https://crt.sh/?serial=03ce00c59d8f19d114079707783a7dab0755
https://crt.sh/?serial=03ce609a5b74a907b99903c866ad10c7a96e
https://crt.sh/?serial=03cfdd86865325e7f85dd52c9dd7a0c743d5
https://crt.sh/?serial=03d1253ab987b7658e7ab933007ba5d032f4
https://crt.sh/?serial=03d3e15c9c001cd662fef0e9d7970a0d6444
https://crt.sh/?serial=03d3e67c347576ef8359ef93c2621402db78
https://crt.sh/?serial=03d571934b3bccf9438b57b0fd1d04b81737
https://crt.sh/?serial=03d5f129507d7b8d032ad634d4744663249e
https://crt.sh/?serial=03d664a1353236abd7df23a0da8839942534
https://crt.sh/?serial=03d89a74a4a96dd9861779fe831ee3e8287d
https://crt.sh/?serial=03d9de3b41376f6308ff7742e2ac2c7a4bf1
https://crt.sh/?serial=03dfa1af74a97981db81d5dac8411b345351
https://crt.sh/?serial=03e1abc47395ba42aec811109412c9d549a3
https://crt.sh/?serial=03e4b5d5c28eaface1096f1bf436cf8cb9f5
https://crt.sh/?serial=03f254176d1a305a5890f7f4f477f19c03ec
https://crt.sh/?serial=03f9cf40b86ba22c497b5ea71ed6aa0798c2
https://crt.sh/?serial=03fda47e2a505be899f510f12bb7bebd5baf
https://crt.sh/?serial=041682a14e6f293811b14798957d89e4d17d
https://crt.sh/?serial=042352263acacda11b1fe930e704ef4123b7
https://crt.sh/?serial=04387df43301837aaa6ff5068fdb551a3c8d
https://crt.sh/?serial=04646f9bd564d3164d3b0f801ddcb4d3f988
https://crt.sh/?serial=046d13d2bc2a20fa81b9828731bd495bb284
https://crt.sh/?serial=048683a0c9d78c5c09514125ebfcef9c8c59
https://crt.sh/?serial=0498d2a905dff66fb25552cecc985907e82b
https://crt.sh/?serial=04c3266f69730d02a0a0a69855f52cc95387
https://crt.sh/?serial=04c8677144c4001eb0a2ab06ec19f098c952
https://crt.sh/?serial=04d55ec60596f6245052095e1ef70da705ac
https://crt.sh/?serial=036c96d523f0a62f9e3ffd5ef7398dbf21f0
https://crt.sh/?serial=0375d0c226922fc2e883a48d119b6a89a268
https://crt.sh/?serial=037b7d9baf658d5175bdf7b640a2e17f29cb
https://crt.sh/?serial=037da25348f551cccc3d22d2ecc451adb41f
https://crt.sh/?serial=037da4500e5285ccaa8e9725f5bf8e6a3bf9
https://crt.sh/?serial=038e70090712d18ce397320acabd26a92f68
https://crt.sh/?serial=03ce074727871c6cee0a90a6b48280fb8335
https://crt.sh/?serial=03db3013d708877d5ed60d0a331cca98197a
https://crt.sh/?serial=03e21b2135156e52ec8b0df1f6307159173a
https://crt.sh/?serial=0303e332559d3c29f996cb47c423ffc6bcbd
https://crt.sh/?serial=030d3f59bf08392ea7fd030db38d822693bd
https://crt.sh/?serial=032048149d5ced1ccb68e31d5d60c13cec04
https://crt.sh/?serial=03226f3c560673a51b217ba537f41cafbcc1
https://crt.sh/?serial=033d878861a94235e16bb04b6dc215aef730
https://crt.sh/?serial=0351a82525c108fb606071493daa469680c1
https://crt.sh/?serial=0354ec8fb31d7f0813b02b118398ce4699b9
https://crt.sh/?serial=0383ea3c1ce07365585b0bd76c6b7656ef5b
https://crt.sh/?serial=03985c4cc590fa39444b280ba0c8247b6a60
https://crt.sh/?serial=03b0ef735816f3155af5730f4440bce9fd78
https://crt.sh/?serial=03d19475bf148a5c8c2b66358bad4608afed
https://crt.sh/?serial=0303ab6eafa4e584aba3f6c6008d89fbcc49
https://crt.sh/?serial=0327844d69c8c46d3c682d608cc4be7b7d5f
https://crt.sh/?serial=032e4109d62f481a97cb9ea527f6ee09bbcc
https://crt.sh/?serial=0337251f51560879f5daf27cc8c5e89f8dd0
https://crt.sh/?serial=034a6224f6ebb63f43b7e452c6551de7ae7e
https://crt.sh/?serial=0381d813579900245a595b10fadc07be121d
https://crt.sh/?serial=03902018d03a17ac1c3e13c36c81422fdd55
https://crt.sh/?serial=0393dcc7c34124fac238b9994aa92207141e
https://crt.sh/?serial=0396d94f8063e06fc14c2677ca53367b64e0
https://crt.sh/?serial=03bca5c23b35a4024cdb8fc1b78cbb039b3b
https://crt.sh/?serial=03e36fcc0196f623749a516b3b921b913999
https://crt.sh/?serial=03ec5d98cffa20d7eaccfe48956a4cf996b4
https://crt.sh/?serial=048a3856e5d824c4910652e8d7af5859f528
https://crt.sh/?serial=04a86dea5b8419c9e36750882c3152e4c500
https://crt.sh/?serial=04f5ab68223e71d82380d3b41a1e302ccbba
https://crt.sh/?serial=03e9dcbcf33a12f3b71ad6c3a06a73639348
https://crt.sh/?serial=0326349e4f912cedeaab1b0c0a74e9a3343e
https://crt.sh/?serial=03bb3dd8a7c63b76750380d3b1a271df15da
https://crt.sh/?serial=03ddb0aac8bbbedbbebbb70499c52a8a0689

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

    This was a conceptual problem in Boulder, where we considered OCSP responses to be a property of certificates rather than of precertificates, and only generated OCSP when a final certificate was issued. We did not detect the problem until now because, while we have tests for OCSP generation and tests for SCT submission failures, we don’t have tests that combine the two. Part of our pending fix is to explicitly test that combination, and expect valid OCSP responses when SCT submission fails (which is a subset of the possible “failed to issue final certificate” cases).

  2. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

    We intend to redeploy the fix to Boulder on 2019-10-03.

You need to log in before you can comment on or make changes to this bug.