1518553 - Sectigo: Use of forbidden subjectPublicKeyInfo algorithm

Reporter

Description

•

6 years ago

The following problems have been found in certificates issued by your CA, and reported in the mozilla.dev.security.policy forum. Direct links to those discussions are provided for your convenience.

Please provide an incident report, as described at https://wiki.mozilla.org/CA/Responding_To_An_Incident#Incident_Report

The problems reported for your CA in the mozilla.dev.security.policy forum are as follows:
P-521 in certificates: https://groups.google.com/d/msg/mozilla.dev.security.policy/4gs5pKqTeK8/9XyMgrW8BgAJ

Ryan Sleevi

Reporter

Comment 1

•

6 years ago

Robin: Please provide a response.

Assignee: wthayer → Robin.Alden

Flags: needinfo?(Robin.Alden)

Robin Alden

Assignee

Comment 2

•

6 years ago

We have made an immediate code-change so that no further end entity TLS certificates may be issued that include a P-521 subscriber public key.

Since these certificates comply with the Baseline Requirements we are not clear whether (and in what timescale) Mozilla requires revocation of these certificates.

I will provide a fuller response by tomorrow.

Wayne Thayer

Updated

•

6 years ago

Whiteboard: [ca-compliance]

Ryan Sleevi

Reporter

Updated

•

6 years ago

Status: NEW → ASSIGNED

Whiteboard: [ca-compliance] → [ca-compliance] Next Update - 09-January 2019

Robin Alden

Assignee

Comment 3

•

6 years ago

How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

We say Corey's post to m.d.s.p. at about 10am UTC, Jan 8, 2019.

A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

00:00 February 28, 2017 Mozilla's Root Store Policy Archive shows that version 2.4 of the Root Store Policy had it's compliance date.
10:00 January 8, 2019 Corey's post to m.d.s.p.
18:00 January 8, 2019 Code released to live so that no further certificates with P-521 keys will be issued.

Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

Sectigo has stopped issuing certificates that include P-521 subscriber keys from CAs that chain to roots in Mozilla's root store.

A summary of the problematic certificates.
For each problem: number of certs, and the date the first and last certs with that problem were issued.

The first P-521 certificate issued after the policy v2.4 compliance date was issued on March 8th, 2017.
The last such certificate issued was issued on January 2nd, 2019.
237 such certificates were issued in total.

The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

We are working to provide this list and will aim to do so tomorrow, January 10th.

Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

We saw the discussion on m.d.s.p. before the Mozilla Policy change to 2.4 and we saw Gerv propose the updated language on-list.

I suspect we did not follow the link in Gerv's email to the github discussion.

We still managed to miss the crucial piece, i.e. that P-521 had been dropped as an acceptable algorithm.

List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We put a code-change live so that no further certificates with P-521 subscriber public keys will be issued.

We previously had an informal process to review Mozilla CA Policy changes.

We have our internal audit team now hooked into monitoring and tracking Mozilla Policy changes.

We are making a new review of our compliance with each item of Mozilla policy, reviewing both current issuance and historic certificate issuance to ensure compliance.
We will report the result of that review in this bug.

Flags: needinfo?(Robin.Alden)

Wayne Thayer

Updated

•

6 years ago

Whiteboard: [ca-compliance] Next Update - 09-January 2019 → [ca-compliance] Next Update - 10-January 2019

Robin Alden

Assignee

Comment 4

•

6 years ago

The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

6 of the 237 we found are code-signing certificates and so not in scope for Mozilla policy.
The other 231 are:

Ryan Sleevi

Reporter

Updated

•

6 years ago

Whiteboard: [ca-compliance] Next Update - 10-January 2019 → [ca-compliance] Next Update - 17-January 2019

Ryan Sleevi

Reporter

Comment 5

•

6 years ago

Thanks for the update, Robin.

We are making a new review of our compliance with each item of Mozilla policy, reviewing both current issuance and historic certificate issuance to ensure compliance.
We will report the result of that review in this bug.

Please report back. I've set the next update to be a week from the last update; please clarify if it will take more time, and if so, when to expect that report.

Flags: needinfo?(Robin.Alden)

Ryan Sleevi

Reporter

Comment 6

•

6 years ago

Robin: Any updates on the result of that review?

Robin Alden

Assignee

Comment 7

•

6 years ago

After a review of our compliance with Mozilla policy[1] we find that although we are substantially in compliance, in addition to the issue identified by the OP where we permitted p521 subscriber keys we have also identified that we had not restricted RSA subscriber keys to have moduli divisible by 8.
We will follow up with analysis of the prevalence of subscriber keys in issued certificates whose RSA moduli are not divisible by 8.

[1] https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/

Ryan Sleevi

Reporter

Comment 8

•

5 years ago

Robin: Can you provide the follow-up mentioned in Comment #7?

Ryan Sleevi

Reporter

Updated

•

5 years ago

Whiteboard: [ca-compliance] Next Update - 17-January 2019 → [ca-compliance] Next Update - 24-June 2019

Robin Alden

Assignee

Comment 9

•

5 years ago

We deployed code on 14-March-2019 to prevent the further issuance of certificates with RSA subscriber keys whose modulus size was not divisible by 8.
There is an interim list of the issued serverAuth certificates with such RSA keys provided at https://docs.google.com/spreadsheets/d/13jiciV0BlTNgGtDFDGS-mKZ-XQ_hMSBZRaitwhX-pUM/edit?usp=sharing.
We will refresh this dataset and post another update in the next week.

Ryan Sleevi

Reporter

Comment 10

•

5 years ago

Robin: It's been over a week since Comment #9. Do you have updates?

Ryan Sleevi

Reporter

Updated

•

5 years ago

Blocks: 1563579

Robin Alden

Assignee

Comment 11

•

5 years ago

Here is a refreshed list of the issued serverAuth certificates with such keys that do not meet Mozilla's policy with regard to the divisibility by eight of the RSA key size.
https://docs.google.com/spreadsheets/d/1yv1tPS-V56DROQkqsqTEbKW4tOeiLCt1D-fsZmoISBc/edit?usp=sharing

Wayne Thayer

Comment 12

•

5 years ago

Robin: for the record, can you confirm that Sectigo does not plan to revoke any of these certificates? I do not believe that revocation is required under current Mozilla policy.

Whiteboard: [ca-compliance] Next Update - 24-June 2019 → [ca-compliance]

Robin Alden

Assignee

Comment 13

•

5 years ago

Wayne: Sectigo does not plan to revoke any of these certificates.

Flags: needinfo?(Robin.Alden)

Robin Alden

Assignee

Comment 14

•

5 years ago

Corey has just posted to m.d.s.p that we have issued a number of further certificates with P-521 subject keys. We will investigate how we missed these and report back in this bug.

Flags: needinfo?(Robin.Alden)

Robin Alden

Assignee

Comment 15

•

5 years ago

On January 8th, 2019, I said in Comment #2:
"We put a code-change live so that no further certificates with P-521 subscriber public keys will be issued."
We had put a code-change live that would disallow P-521 subscriber public keys in end entity SSL certificates but the code change only fixed one of two code-paths that could lead to the issuance of an SSL certificate.

The code to which a change was made to disallow P-521 was in fact only present in the code branch for creating EV SSL certificates. The limitation on possible public key algorithms and key sizes was implemented completely separately for the non-EV SSL case.

Making the change to only one of the two code paths in which it was required meant that while P-521 keys were correctly disallowed in EV SSL certificates, the non-EV SSL case incorrectly retained support for P-521 keys. Why the difference in mechanisms? – Because the EV Guidelines existed before the Baseline Requirements and so the external policy requirements for supported public key algorithms and key sizes arose at different times.

That suggests that an early contributary factor to this issue happened in 2012 when the code changes to support the (then new) BRs were written. The checks on public key algorithm and key sizes should have been unified across SSL certificates at that time, but they were not.

This also raises the question of how the QA element of our software development process failed to identify this error. The change to implement the restriction on P-521 keys was made as an urgent response to the discovery that we were still issuing end entity certificates with P-521 subscriber public keys against Mozilla’s CA policy in January 2019. This change went through an accelerated development process and the pre-release QA was clearly inadequate. Post release QA was not recorded.

This is the final list of certificates issued by Sectigo after Mozilla’s policy changed to prohibit the use of P-521 subscriber keys:
https://docs.google.com/spreadsheets/d/1n1tjNd3DSGukhsEBLVRBmb1OUov0lS1CHIZkGIPRa6U/edit?usp=sharing

Remediation:
On July 22nd, 2019 we put a further code change live to close off the issuance of end entity TLS certificates containing P-521 subscriber public keys through the second (non-EV) code-path.
Going forward we will increase the level of detail that we record over pre-release QA for code changes and we will consider the detail of pre-release QA during the risk analysis meeting that authorizes each change for release.
We have auditing the completion of post-release QA for recent code changes and will ensure that the completion of post-release QA is recorded for each change released and we will leave development tickets open until post-release QA is recorded as completed.

Wayne Thayer

Comment 16

•

5 years ago

It appears that all questions have been answered and remediation is complete.

Status: ASSIGNED → RESOLVED

Closed: 5 years ago

Resolution: --- → FIXED

Robin Alden

Assignee

Updated

•

5 years ago

Flags: needinfo?(Robin.Alden)

Nobody; OK to take it and work on it

Updated

•

2 years ago

Product: NSS → CA Program

David Lawrence [:dkl]

Updated

•

2 years ago

Whiteboard: [ca-compliance] → [ca-compliance] [ov-misissuance] [dv-misissuance]