GoDaddy: Insufficient serial number entropy

ASSIGNED
Assigned to

Status

ASSIGNED
14 days ago
7 days ago

People

(Reporter: wayne, Assigned: dreynolds)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [ca-compliance])

Attachments

(1 attachment)

(Reporter)

Description

14 days ago

Daymion Reynolds posted the following to the mozilla.dev.security.policy list:

As of 9pm AZ on 3/6/2019 GoDaddy started researching the 64bit certificate Serial Number issue. We have identified a significant quantity of certificates (> 1.8million) not meeting the 64bit serial number requirement. We are still performing accounting so certificate quantity is expected to change before we finalize the report.

  1.  How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
    

9pm 3/6/2019 AZ Time, due to reviewing a discussion in mozilla.dev.security.policy.

  1.  A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
    

9pm 3/6/2019 AZ Time, identified a hot issue with serial numbers in Mozilla group.
10am 3/7/2019 AZ Time, identified the issue was pervasive, and identified root cause.
6:30pm 3/7/2019 AZ Time, fix deployed to production to correct the serial number issue.
We are still quantifying and classifying the certificate scope of impact.

  1.  Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
    

We have deployed a fix to the issue, and are no longer issuing certificates with the defect.

  1.  A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
    

Issue was introduced with a change in 2016. Impacted certificates still being aggregated. Will update with information and timeline on issue closure.

  1.  The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
    

Still being aggregated. Will update with certificate information on issue closure.

  1.  Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
    

Ambiguity in language led to different interpretations of BR 7.1. It was believed a unsigned 64bit integer was sufficient to satisfy the new requirement. Additionally, industry tools like CABLint/ZLint were not catching this issue, which provided a false sense of compliance. We are submitting CABLint/Zlint updates as part of the fix.

  1.  List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
    

Defect has been resolved, we are also updating linting tools (CABLint/Zlint) and upstreaming to patch for other peoples usage.

Comment 1

11 days ago

Daymion,

Do you have an update on the list of impacted certificates? This is also missing a timeline about proposed next steps.

Flags: needinfo?(dreynolds)
(Assignee)

Comment 2

10 days ago

As of 9pm AZ on 3/6/2019 GoDaddy started researching the 64bit certificate Serial Number issue. Due to a m.d.s.p.[1] discussion validating an interpretation of BR 7.1 our revised count is approximately 12,152 live certificates not meeting the 64bit serial number requirement. Additionally, we have identified 273,784 “orphaned” certificates meeting the initial interpretation of BR 7.1. Orphaned certificates are certs, which were stopped mid-issuance due to a variety of reasons like requestor cancellation, system errors etc. These certs are most often pre-certificates, but some are leaf-certificates, which were logged to CT, but never received by the certificate requestor.

The initial report stated >1.8 million certificates were impacted. For our initial investigation we checked certs against the first bit being set, which seemed to be the industry interpretation at the time. This lead to more aggressive criteria than necessary. As we started revocations of orphan certificates we continued researching the criteria defined by BR7.1 After re-evaluating the criteria, per m.d.s.p.[1], we adjusted our certificate scope.

  1.    How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date. 
    

9pm 3/6/2019 AZ Time, due to reviewing a discussion in mozilla.dev.security.policy.

  1.    A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done. 
    

9pm 3/6/2019 AZ Time, identified a hot issue with serial numbers in Mozilla group.
10am 3/7/2019 AZ Time, identified the issue was pervasive, and identified root cause.
6:30pm 3/7/2019 AZ Time, fix deployed to production to correct the serial number issue.
2pm 3/9/2019 AZ Time, we revoked 273,784 orphaned, pre-certs and leaf certificates which met the initial criteria. These certificates were low\no risk as they were never distributed.
11pm 3/11/2019 AZ Time, defined resolution as stated in m.d.s.p., further research revised the quantity of impacted certificates.
3/12 – 3/16 – We will be revoking the before mentioned 12,152 live certificates.
Once the revocation is complete we will update this timeline.

  1.    Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation. 
    

We have deployed a fix to the issue on 3/7/2019, and are no longer issuing certificates with the defect.

  1.    A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued. 
    

12,134 certificates were affected.
Revoking as a precaution, certificates issued on 9/29/2016
9/29/2016 2:41 66269447367104810
9/29/2016 7:44 35092532380173749
9/29/2016 9:46 43324527254073466
9/29/2016 14:16 53640950198707040
9/29/2016 14:31 36562688867169546

The first affected certificate was issued on 9/30/2016 1:29 https://crt.sh/?id=290271291
The last affected certificate was issued on 3/7/2019 15:32 https://crt.sh/?id=1262876175

  1.    The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. 
    

Spreadsheet attached to defect. Older certificates may not yet be CT logged, as the majority of the certs are DV.

  1.    Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now. 
    

Ambiguity in language led to different interpretations of BR 7.1 in 2016. It was believed an unsigned 64bit integer was sufficient to satisfy the new requirement. Additionally, industry tools like CABLint/ZLint were not catching this issue, which provided a false sense of compliance. We are submitting CABLint/Zlint updates as part of the fix.

  1.    List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things. 
    

Defect has been resolved, we are also updating linting tools (CABLint/Zlint) and upstreaming to patch for other peoples usage.
Additionally, we are looking to scope and roadmap upgrading our certificate serial number to a minimum of 128-bit, or the max possible.
[1] https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/7WuWS_20758

Flags: needinfo?(dreynolds)
(Assignee)

Comment 3

10 days ago

Set of certs

(Assignee)

Comment 4

7 days ago

In accordance with our conversations to date, prior to 3/7 6:30pm AZ we utilized raw 64 bit output from CSPRING, with uniqueness and non zero checks. This new understanding of the rules calls for us to modify our original disclosure to 0 affected certificates.

For any new other serial number issues we will file a new incident.

You need to log in before you can comment on or make changes to this bug.