Closed Bug 1759959 Opened 2 years ago Closed 2 years ago

GoDaddy: OV Documentation Reuse

Categories

(CA Program :: CA Certificate Compliance, task)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: brittany, Assigned: brittany)

Details

(Whiteboard: [ca-compliance] [ov-misissuance])

User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36

Problem Summary:

We issued 33 Organization Validated (OV) certificates which used documentation for organization validation that was greater than 825 days old which is a violation of section 4.2.1 Performing identification and authentication functions of the Baseline Requirements for Publicly Trusted SSLs which states "The CA MAY use the documents and data provided in Section 3.2 to verify certificate information, or may reuse previous validations themselves, provided that the CA obtained the data or document from a source specified under Section 3.2 or completed the validation itself no more than 825 days prior to issuing the Certificate." Note: only organization validation was impacted.

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in the MDSP mailing list, a Bugzilla bug, or internal self-audit), and the time and date.

During the completion of a 3% certificate audit, the RA team noticed an anomaly related to audit notes for organization validation within the vetting system for one certificate and sent this information to the development team for review. On 3/8/2022, at 15:00 MST the PKI development team confirmed a bug within code which would allow a certificate to pass organization validation checks with data that was greater than 825 days.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

DD/MM/YYYY HH:MM (Times are all MST)

  • 03/08/2022 09:41 RA team send question to development team about a certificate that was reviewed as part of the 3% audit process
  • 03/08/2022 15:00 PKI development confirms a bug in the code that allowed a subset of OV certificate requests to be issued on reused organization validation from previous certificate requests older than the allowed amount of time (825 days)
  • 03/08/2022 15:05 PKI development updates an application configuration to disallow the re-use of data for any organization documentation in the OV certificate issuance process
  • 03/08/2022 15:30 PKI development estimates impact of less than 50 certificates.
  • 03/09/2022 02:30 PKI development identifies full list of impacted certificates (33 - see list below)
  • 03/09/2022 14:29 All impacted certificates are revoked
  • 03/09/2022 14:30 PKI Development verifies that all the OCSP responders are showing certificates as revoked

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

As of 03/08/2022 15:05 MST, we updated the application configuration to disallow the re-use of any organization documentation data in the (OV) certificate issuance process.

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

33 Organization Validated (OV) certificates were impacted. The oldest certificate was issued with a valid start date of 8/2/2020 2:48:46 PM MST and the newest certificate was issued with a valid start date of 3/7/2022 7:01:09 AM MST.

Certificate type breakdown below

  • 1 Multi-domain (UCC)
  • 23 Single-domain (standard)
  • 9 Wildcard

As of 03/09/2022 at 14:29 MST, all impacted certificates have been revoked.

5. In a case involving TLS server certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. It is also recommended that you use this form in your list "https://crt.sh/?sha256=[sha256-hash]", unless circumstances dictate otherwise. When the incident being reported involves an SMIME certificate, if disclosure of personally identifiable information in the certificate may be contrary to applicable law, please provide at least the certificate serial number and SHA256 hash of the certificate. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

https://crt.sh?sha256=31a2df2828a4fb79063d99de353a10e704c3e153df2ba1eeacdf38cba962067d
https://crt.sh?sha256=9b21e286e87c8c4705897175c41f1fbe3f1b5fd8e9c509cabadb45ecc9a03739
https://crt.sh?sha256=6fb2489ae309a564abbb2e8b00445ab2d9536d5b7f90e0f0a9a79639ae3f9ec1
https://crt.sh?sha256=5f72667b3500c3be0a724eff008ae648f88448d6e63ed35dfe815b6c4c49e88e
https://crt.sh?sha256=6f5da01ae3126a3e0eaafcbc19d8b4cc84a5eb75c683cf26f332401f786dfc20
https://crt.sh?sha256=9e5a4e29c0847b4395fb6bfb296c4f622e88af7f7ee38c4993f29186d1483450
https://crt.sh?sha256=7fa72d2edbd2dc7d9796e9b711b53de06d0543bd5bd7ece15e75b7a7046b054e
https://crt.sh?sha256=4f80232c9f3ac86d4d81ff278bf4a5df8379a02a96a0098f5c12826cfa83dceb
https://crt.sh?sha256=e81a35ff956bf8e7307d57e79b2f3a647044e3c1fafbb822ff6a57a00380ca1e
https://crt.sh?sha256=0e9d30f2566ead030ce64caf53f9d8351aaa06b345e79e7a33129109d8d04555
https://crt.sh?sha256=5cbbed67ea16740cbc9ec083c7a5a44b5bd180dbd42e8cf8f61790d4d2ab32ff
https://crt.sh?sha256=1e69ab6fd5f6cae415ec9c8b0ea481b897394d54acd8c54a28d17311d7fc6712
https://crt.sh?sha256=384ff7cd3d99f32a3ea8ac713404d4b6bc9bbaa923e93aa075397377b1a1bf76
https://crt.sh?sha256=b1bf1d16cfe0dfe6161192d15ef5052cf311962554a481b76a39ee705d75beb1
https://crt.sh?sha256=60aa192734abf814cb5949acf8ad03d14e9b7c1f633ec1e9b1fedb2ca1f91b15
https://crt.sh?sha256=7e29a4db9dd1107c5270ac08c5d8c5ffdc4e14914d230f868245ae9c2e1c283e
https://crt.sh?sha256=43fd75a49cdc45f0a94446941c11cc1d5c32d280c693e266f406a8b746136dbd
https://crt.sh?sha256=1bc45daee599b8e37be70e28e6ef697f70743e3cbbf0f44302a1bcbb283af310
https://crt.sh?sha256=7f124d4a4238591c5d2644056855e5c2d87dbcb0b2ba1e165df21195d4a84f12
https://crt.sh?sha256=8c7da93b14c4d3057b6bb29b17a4586881c74b33a30d06dd218e3c2fe16d961e
https://crt.sh?sha256=4a1b7eef4ff877d6393a6e249239d7947905625fe7a6d344e3525f9558dd163b
https://crt.sh?sha256=e66bc4c76fd80d0b701ad68c1784b50282663782e85d98a2b8ed4e9d2b4be10d
https://crt.sh?sha256=93ed7947cbd65eaf7755571416ef4d504a6db07d10877af2e975b346898e1dc1
https://crt.sh?sha256=eb058141a1223a7a7fa294315800c945c8c3ed58e2caf966b77dfcc5eb5d1a28
https://crt.sh?sha256=b30006dfd810022441bb78a909fbcfcfe27a7bc10b35526921016e55b2b4b820
https://crt.sh?sha256=95d3739baecef4eee941a2df0de1bb2566fdaeb44dcd7dd7ae202c06346ab0a1
https://crt.sh?sha256=1ae6604ca2464c7f94b34e4eec43b0c27b8d3c9df62d25514493e5095af8ab1e
https://crt.sh?sha256=f1fa3540f6347cfbfe4b1eaef9354c0bc5646aa660148f2f611d27b23673c639
https://crt.sh?sha256=67354854810983a14e85983ac2b1d234a85ec0097c65b4070b6049a09b3922c6
https://crt.sh?sha256=a935e956eedd44c82670a80b1de586b1cf3a418a0192c6db8a5828542ddbc1d5
https://crt.sh?sha256=14c39b593e0835e42b09d0c2db39d0a4df5f4f2b456970dbd71e7b2f003be537
https://crt.sh?sha256=33ad2d03f1e8ee6484f25fc6909bc14603aa56a4630503b070d07a02855f7a85
https://crt.sh?sha256=09405955c3056950ec33a038046df8bae63812b0815beb2e3ff9209377487f04

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

We identified a bug was introduced when an update to the code targeted at efficiency used a new query with an existing query and did not properly handle the combined results in rare cases. Due to the small volume of impacted OV certificates (33 certificates), the bug was not detected until identified during a recent 3% certificate audit.

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

Below is a list of specific remediation actions and the expected completion time:

  • Systematically stop issuance of OV certificates with re-use of organization validation data (Completed). As of 03/08/2022 15:05 MST, we updated the application configuration to disallow the re-use of data any organization documentation in the (OV) certificate issuance process. Transparently, once the root cause of the issue within the code is addressed (see below), this feature will be re-enabled.
  • Revoke all the certificates associated with this issue (Completed): As of 03/09/2022 at 14:29, all impacted certificates have been revoked.
  • Address issue within the code (Target date: 06/30/2022). We are reviewing the code which contained the bug that allowed the issue. Fixes to the code will address the bug and allow documentation re-use to be re-enabled for OV certificates.
  • Validation linter review (Target date: 06/30/2022). We are reviewing our validation linter to see where additional checks could be added or current checks could be enhanced to prevent this type of issue in the future.

We identified a bug was introduced when an update to the code targeted at efficiency used a new query with an existing query and did not properly handle the combined results in rare cases. Due to the small volume of impacted OV certificates (33 certificates), the bug was not detected until identified during a recent 3% certificate audit.

Could you provide a more detailed explanation?

Flags: needinfo?(brittany)

Also, it’s unclear where that timeline fits in response to question 2, which seems relevant to this incident report. The timeline is trying to determine the overall system issue, not from detection.

Thanks for your questions, Ryan. Just wanted to acknowledge that we've seen them and we are working on responses.

Flags: needinfo?(brittany)
Assignee: bwilson → brittany
Status: UNCONFIRMED → ASSIGNED
Type: defect → task
Ever confirmed: true
Whiteboard: [ca-compliance]
Flags: needinfo?(brittany)

Hey Ryan. Just wanted to provide an update. We will post a response to your questions this Monday, 3/21/2022. Have a great weekend!

Flags: needinfo?(brittany)
Flags: needinfo?(brittany)

(In reply to Ryan Sleevi from comment #2)

We identified a bug was introduced when an update to the code targeted at efficiency used a new query with an existing query and did not properly handle the combined results in rare cases. Due to the small volume of impacted OV certificates (33 certificates), the bug was not detected until identified during a recent 3% certificate audit.

Could you provide a more detailed explanation?

On 06/15/2020 17:17 MST, PKI development introduced a feature related to document validity that reuses organization validation (OV) from previous validated certificates. This feature used a new query, alongside an existing query, to find previous OV certificates that included organization validation information. The results of the two queries were merged to find the most recent OV certificate containing the vetted documentation. In rare cases, this merged logic returned certificates that did not contain vetting documentation because the returned OV certificate itself referred back to an older certificate for its documentation re-use. The validation logic for the documents' dates being within the reuse period did not accommodate for this edge case of a certificate with no organization validation documents, allowing that certificate to be incorrectly reused to support the organization validation.

Flags: needinfo?(brittany)

(In reply to Ryan Sleevi from comment #3)

Also, it’s unclear where that timeline fits in response to question 2, which seems relevant to this incident report. The timeline is trying to determine the overall system issue, not from detection.

Thanks for calling this out! Below is the timeline entry related to the intro of the system bug:

DD/MM/YYYY HH:MM (Times are all MST)

  • 06/15/2020 17:17 - Code and bug described above was deployed to production.

Incident Report Update 3/21/2022:

Summary: We have identified three additional certificates that were issued related to this bug.

On 03/08/2022 15:05, 5 minutes after confirming the bug and compliance issue engineers turned off organization validation reuse in the PKI system, which meant all new OV certificate requests thereafter required new organization vetting. While the functionality was turned off at that time, there were pending certificate requests that were already set to reuse organization validation. As an example of a cert that could end up in this use case, would be one where the system had already completed the OV validation, but was still completing the domain and CAA validation.

See below for additions/amendments to the incident report from the addition of these certificates.

  1. Update 33 certificates issued to 36 certificates issued

  2. Addition of the following items into the timeline
    DD/MM/YYYY HH:MM (Times are all MST)

  • 3/17/2022 07:06 and 07:07 Ryan Sleevi posts questions to Bug 1759959
  • 3/17/2022 07:45 PKI Development begins investigating answers to questions posted to the Bug 1759959 by Ryan Sleevi.
  • 3/17/2022 11:52 PKI Development reruns the query that identified the original list of 33 impacted certificates which identifies the 3 new certificates out of compliance and surfaces the items in pending issuance. PKI Development then queries for the remaining pending items that could have been impacted by this bug
  • 3/17/2022 14:55 PKI Development denies the 159 pending requests
  • 3/17/2022 16:37 PKI Development revokes the 3 non-compliant certificates
  1. As of 03/18/2022 14:55 MST, all pending certificate requests are denied meaning no additional certificates should be issued as a result of this bug.

  2. Addition of 3 additional certificates which results in the following summary updates:

  • the newest certificate was issued with a valid start date of 3/16/2022 19:38:27 MST
  • Addition of 2 Single-domain (standard) and 1 wildcard certificate
  • As of 3/17/2022 16:37 MST, these remaining certificates were revoked
  1. Addition of the following:
    https://crt.sh?sha256=8dbd3b8560fd148ce08f083bddd320ecaf390c6105cdb9e08a439c01abfdcc9e
    https://crt.sh?sha256=1ee37a668aaca653a8d10b4f06beff99b71a503bfa5cb469bf9c9be74bec0a6b
    https://crt.sh?sha256=76b89c626e8f5728b981281dde0649741f1b57d00069186bf375a3cedcd588fa

  2. Note: The systematic root cause for these three certificates is the same and relates to the change made on 06/15/2020 17:17.

  3. No update.

No formal updates at this time. We are still monitoring for questions and will be providing updates on remediation activities as applicable.

No formal updates at this time. We are continuing to monitor for questions.

Whiteboard: [ca-compliance] → [ca-compliance] Next update 2022-06-30

How is your progress coming on the following?

  • Address issue within the code (Target date: 06/30/2022). We are reviewing the code which contained the bug that allowed the issue. Fixes to the code will address the bug and allow documentation re-use to be re-enabled for OV certificates.
  • Validation linter review (Target date: 06/30/2022). We are reviewing our validation linter to see where additional checks could be added or current checks could be enhanced to prevent this type of issue in the future.
Flags: needinfo?(brittany)

(In reply to Ben Wilson from comment #11)

How is your progress coming on the following?

  • Address issue within the code (Target date: 06/30/2022). We are reviewing the code which contained the bug that allowed the issue. Fixes to the code will address the bug and allow documentation re-use to be re-enabled for OV certificates.
  • Validation linter review (Target date: 06/30/2022). We are reviewing our validation linter to see where additional checks could be added or current checks could be enhanced to prevent this type of issue in the future.

Hi Ben - Thanks for reaching out on these. We have completed both of these items. I'll circle back with our PKI development team to get some details of the dates for when they were actually completed and report back.

Flags: needinfo?(brittany)

Official update. All action items have been completed. Notes added below:

  • Address issue within the code (Completed). We deployed a fix as of 5/2/2022 and re-enabled documentation reuse at that time.
  • Validation linter review (Completed). In conjunction with the code updates completed above (deployed on 5/2/2022), we improved our linter rules around document reuse.

All action plan items have been completed. Please let us know if you have any additional questions as this time.

I'm closing this bug as complete.

Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] Next update 2022-06-30 → [ca-compliance] [ov-misissuance]
You need to log in before you can comment on or make changes to this bug.