Scott Rea representing DarkMatter posted the following incident report to the mozilla.dev.security.policy mailing list:
- How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
A post by Corey Bonnell to the mozilla.dev.security.policy group on February 23, 2019 at 1:28am UAE time. Corey referenced Section 7.1 of the Baseline Requirements and the specific sentence he believed DM was violating: “Effective September 30, 2016, CAs SHALL generate non-sequential Certificate serial numbers greater than zero (0) containing at least 64 bits of output from a CSPRNG”. Corey also listed 23 certificates (CAs & EE’s) and their serialNumbers as evidence. Corey followed this up 2 days later with a more comprehensive list that included DMs pre-certs and final certs published to various CT Logs. Corey also pointed out a second issue in respect to nameConstrained intermediates issued by Quo Vadis to DM as also a potential violation, but this shall be dealt with in a separate incident report, since DM was not the issuer of these ICAs. Corey’s second post was received on the list at 12:50am UAE time February 25, 2019.
- A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
29-Apr-16: DM having previously engaged with QuoVadis for the establishment of a National PKI utilizing a platform and hardware that would initially be located in QV DCs, held key ceremonies for Private PKIs on EJBCA platform. Platform was configured for random 64-bit serialNumbers which at the time was considered far in excess of the then current BR requirement of 20-bit entropy.
30-Apr-16: QV creates DM intermediates intended for issuance of EV and OV public trust TLS. These intermediates are instantiated in separate partitions to the DM Private Trust PKIs, but issuance from them is still subject to the same platform setting of 64-bit random serialNumbers.
08-Jul-16: CABF Ballot 164 closes and passes. The phrase Corey highlighted above is added in place of the previous 20-bit entropy statement.
28-Mar-17: Transition of DM private PKIs and platform to DM DC’s is complete under supervision of EY as WebTrust Auditors. Public Trust CAs continue to operate out of QV DCs.
08-Jun-17: DM Point In Time Audit is complete. NOTE: DM is now responsible for platform configuration in compliance with BRs. DM considers current serialNumber setting of platform (64-bit SNs) as compliant with BRs Section 7.1
27-Oct-17: DM successfully completes WebTrust Period In Time.
04-Nov-17: QV & DM complete transfer of DM public trust intermediates from QV to DM under EY & KMPG audit observation
18-Dec-17: DM & UAE Roots submitted to Mozilla
02-Oct-18: DM successfully completes 2nd WebTrust Audit
06-Nov-18: DM submits new WebTrust Audits to Mozilla
23-Feb-19: 1:28am: Corey posts to MDSP claiming DM has mis-issuance of certs due to BRs Section 7.1
23-Feb-19: 3:36am: Post is observed by DM, internal investigation requested immediately.
23-Feb-19: 6:14pm: Internal investigation confirms 64-bit setting for all certificates, certlint/cablint/zlint do not complain about certs submitted with this setting. Confirm that settings appear to be compliant with BRs. Further validation sought from platform provider, support ticket opened.
25-Feb-19: 12:50am: Second email from Corey with expanded list of pre-certs and issued certs is posted to MDSP
25-Feb-19: 5:08am: Response posted to Corey on MDSP regarding DM’s investigation and the conclusion that DM is in compliance with BRs Section 7.1. NOTE: over next two days several folks weigh in on the 64-bit entropy vs 64-bit CSPRNG output wording of BRs and what that means.
26-Feb-19: 7:41pm: Wayne Thayer posts to MSDP that 64-bit serialNumber with 63-bit entropy are mis-issued under the BRs.
27-Feb-19:10:55am: DM responds to Wayne reiterating our position of compliance since BRs only say 64-bit output from a CSPRNG and do not say anything about entropy. However, DM commits to make a move to 128-bit serialNumbers from a CSPRNG in next change control to seek to alleviate concerns. More discussion continues on the list.
27-Feb-19: 11:55am: Ryan Sleevi posts historical data/discussions links to MDSP regarding the change resulting from Ballet 164 that detailed where subsequent discussion was had regarding some CAs that included serials of exactly 64 bits, and how those were considered that they could not be compliant, and they made the necessary changes.
27-Feb-19: 5:00pm: DM, following a review of data/links provided by Ryan, stops issuance of public trust certificates. Plans are begun for an immediate migration to 128-bit serial numbers with far exceeding the required entropy. DM announces decision on MDSP at 5:09pm
28-Feb-19: 500pm: DM action is past 24 hours: Testing of manual upgrade of platform without waiting for patch from vendor, roadmap of patch application so that other profiles i.e. private trust (not TLS public trust), can later be downgraded back to 64-bit – issuance is stopped for those profiles until patch can be applied. Change scripts created, tested and QA on pre-production, validated. Production change control scheduled and executed successfully. TLS issuance restored in production with confirmed 128-bit serialNumber from CSPRNG. Full list of still active certificates with 64-bit serialNumber is generated. Notices to respective customers are sent, and where possible phone calls made. Revocation and re-issuance of affected certificates begins for customers who are responsive. NOTE: Thursday is last day of work week in UAE, Friday & Saturday are week-end, most businesses open again on Sunday.
28-Feb-19: 9:30pm: Progress to date: 157 total certificates identified as still active that will be revoked during this exercise. Currently 45 have been re-issued. 18 revoked.
Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
Issuance was stopped at 5:00pm 27 Feb 2019. Platform is now upgraded to 128-bit serialNumbers, and issuance now with larger bit sized serialNumbers is active.
A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
all TLS certificates up until 5:00pm yesterday (27 Feb 2019) were issued with 64-bit serialNumber from 29 April 2016 until 27- Feb 2019. 157 TLS certificates still active as of yesterday. 45 have now been reissued, 18 revoked.
The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
This post will be updated with this info shortly, but please reference Corey Bonnel’s earlier post of the same
Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
CA platform was originally provisioned with 64-bit serialNumber during period this was explicitly allowed under BRs. When DM migrated private trust platform and began preparing for public trust issuance through WebTrust Audit preparations, this was after Ballot 164 had updated the BRs. DM was not privy to the discussions around Ballet 164 and the subsequent community discussions. DM analyzed the BRs post Ballot 164 and concluded based on the definition in Section 7.1, that the platform was complaint with the BRs. This continued to be our position until Ryan Sleevi posted historical data regarding the interpretations provided to other CAs after the adoption of Ballet 164. At this point DM realized that despite what the BRs said, the general community interpretation of Section 7.1 was different, and it had been consistently applied to all other CAs in the community. DM is not seeking an exception, and only wishes to be held to the same standard as all other CAs in the community, therefore it became necessary to make an immediate change. It was not bug originally, it was complaint at the time it was introduced. It avoided detection until now because the update to the BRs appears to have lost some specificity in it’s re-wording which existing CAs knew about at the time, but any new CAs coming after the change may not make the same conclusion without the benefit of the background information.
List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
Change is already active – CA now using 128-bit serialNumbers for any new certificates. There are 157 active TLS certificates at the time that were impacted and will be revoked. Expected completion for revocations is next 24 to 48 hours.