1646226 - GoDaddy: Document Reuse Issue

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Daniela Hood]

Attachment: Document Reuse - Problematic Certificates.xlsx

1- How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
After walking our external auditor through our internal self-audit and a certificate that had what looked to be a manual documentation issue that was corrected, we noticed a potential bug. We investigated further and determined on Tuesday, June 2 around 4:30 PM that in certain relatively rare cases the system was misidentifying document validity dates.​​​​

2- A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

​Tue 6/2 4:30 PM Identified the combination of activities that was leading to the system misidentifying document validity dates
Wed 6/3 6:38 PM Derived the list of potentially affected certificates and began efforts to verify whether certificates were actually affected
Wed 6/4 ​​3:30 PM Finalized the list of certificates for revocation
Fri 6/5 7:00 PM Revoked 471 certificates and instituted an interim process to revoke new certificates with this issue daily until the fix was implemented
Mon 6/15 5:17 PM Implemented system fix and verified all affected certificates were revoked, no new certificates issued with this bug​

3- Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
We stopped issuing certificates with this problem as of Monday 6/15/2020 5:15 PM. An additional 24 problem certificates were issued between the mass revocation on 6/5/2020 at 7:00 PM and the system fix on 6/15/2020 at 5:17 PM. These certificates revoked within 5 days of being issued.

4- A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
Of the total 495 certificates with this issue, 398 were OV certificates and 97 were code signing.

5- The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
List included separately.

6- Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.​
The bug has to do with a combination of manual procedures and system functionality. Each component operating independently functions properly within standards, but when performed in combination results in the system misidentifying document validity dates. The sequence must contain:

• An agent manually validates a certificate with documentation and issues a certificate. This is original system functionality.
• An agent manually validates a renewal or rekey of the certificate by pulling forward documentation from the original certificate that is within the reuse period. This feature was implemented around 8/21/2017 and worked as expected.
• The certificate is then renewed or rekeyed and the system identifies that the certificate was previously vetted within the appropriate period of time. This feature was implemented in 2011 and works as expected when the prior certificate uses original documents.

The automated portion of the verification looks to see that the prior certificate was previously vetted within the appropriate period. If the prior certificate used documents copied from an older certificate, this introduced the potential for the system to misidentify the validity date of the documents. Further, this situation only results in an issue when activities are performed on certificates renewed over a period of years. ​Only one certificate in the revocation list dates to 2018, 174 in 2019 and 197 in 2020. With the increase in instances in 2020, we identified the issue in the 3% audits. However, the issue was not sourced as a bug until we analyzed further.

7- List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.​
We implemented a technical fix to to ensure the system always evaluates reuse timing from the document upload date. We are planning an internal audit of our end-to-end flows against a matrix showing all the time-based baseline requirements, and using the matrix to help ensure changes are well evaluated, understood, and tested.

Comment 1

[paul.leo.steinberg]

(In reply to Daniela Hood from comment #0)

4- A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
Of the total 495 certificates with this issue, 398 were OV certificates and 97 were code signing.

5- The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
List included separately.

The attached file contains 400 rows for OV. A very simple sort | uniq -c | sort -n shows that 4 of these entries are duplicates:

      2 https://crt.sh/?id=2581307073
      2 https://crt.sh/?id=2737482939
      2 https://crt.sh/?id=2749294993
      2 https://crt.sh/?id=2812157760

That is, your list only contains 396 certificates and not 398 as stated above.

Additionally, how can you be sure that your investigation is accurate at all if your attached file contains such very trivial mistakes (running sort | uniq -c | sort -n takes seconds!)?

Comment 2

[Daniela Hood]

Attachment: Document Reuse - Problematic Certificates.xlsx

Correct Upload

Comment 3

[Daniela Hood]

Hello Paul,

Thank you for your comment.
We had several people working on the list to capture the crt.sh IDs to provide to the forum, and that portion of collecting that information is a manual process, therefore some duplicates were inserted. Our research on the amount of certificates still correct. We performed a full reconciliation and re-verified the cert.sh links. After the reconciliation, we verified the total of 495 certificates is correct, however it contained 396 OV certificates and 99 code signing. I uploaded the correct list without the duplicates. ​

Comment 4

[Ben Wilson]

(In reply to Daniela Hood from comment #0)

7- List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.​
We implemented a technical fix to to ensure the system always evaluates reuse timing from the document upload date. We are planning an internal audit of our end-to-end flows against a matrix showing all the time-based baseline requirements, and using the matrix to help ensure changes are well evaluated, understood, and tested.

Do you have a schedule for when you plan to conduct that internal review of end-to-end flows and make changes? Thanks.

Comment 5

[Daniela Hood]

Hello Ben,

Thank you for your comment.
​​Our plan is to have a continuous internal review, to ensure that we are always current to changes and no changes were implemented incorrectly. The first end-to-end flow internal review should start on the beginning of Q3 and end by end of 2020.

Comment 6

[Ben Wilson]

Hi Daniela,
Could you provide an update on the relative success since you implemented changes to calculate document reuse from the document upload date? Then I intend to close this bug, unless there are other questions or comments.
Thanks,
Ben

Comment 7

[Daniela Hood]

Hello Ben,

Thank you for your comment.
​​We completed weekly audits and executed multiple queries throughout the issued certificates population that confirms the fixes implemented were successful.