1664328 - GlobalSign: SHA-256 hash algorithm used with ECC P-384 key

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Rob Stradling]

https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/#512-ecdsa says:
"When a root or intermediate certificate's ECDSA key is used to produce a signature, only the following algorithms may be used, and with the following encoding requirements:
...
If the signing key is P-384, the signature MUST use ECDSA with SHA-384."

The "GlobalSign ECC CloudSSL CA - SHA384 - G3" intermediate CA has a P-384 key, but most of the certificates it has issued (https://crt.sh/?Identity=%25&iCAID=50609) have signatures that use ECDSA with SHA-256.

Comment 1

[Arvid Vermote]

We acknowledge this report and have started investigating the reported issue.

Comment 2

[Paul Brown]

Here follows incident report:

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

GlobalSign had been carrying out and investigation due to GTS incident on invalid curve hash https://bugzilla.mozilla.org/show_bug.cgi?id=1612389, this investigation had not uncovered any issue. However during an investigation for a separate issue for RSASSA-PSS Public key https://bugzilla.mozilla.org/show_bug.cgi?id=1630870, it was found that one CA had been incorrectly added to a previously configured profile without being updated with correct signing algorithm.

However it was missed out on performing the required actions for actively verifying if any certificates were issued with such a signature algorithm in the past and taking the necessary actions if any.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

Date	Action
February 10 2020	After GTS announce incident on invalid curve hash combination, we open ticket to investigate whether we are affected
February 13 2020	Accounts checked and verified that these have correct settings with regards to CA configured
February 20 2020	Further question to our infrastructure team on zlint implementation status in order that ticket can be closed
April 16 2020	Incident on issuance of certificate with RSASSA-PSS Public key https://bugzilla.mozilla.org/show_bug.cgi?id=1630870
April 23 2020	Infra team confirmed issuing certificate with wrong algorithm was not supported via Atlas (GlobalSign new CA system)
May 13 2020	As part of review being carried out on RSASSA-PSS - https://crt.sh/?id=2802864202 was issued via GCC (GlobalSigns legacy system).
May 25 2020	Compliance team ask for details on all configurations of CAs
May 26 2020	Upon receiving details of all CAs, Compliance team note one CA set to SHA256withECDSA which did not match its name (cloudssleccsha2g3CA actually had a P-384 key) - requested change to SHA384withECDSA
May 26 2020	Profile updated and request for a full EJBCA profile export
June 5 2020	Full Configuration Export requested as part of RSASSA-PSS incident remediation cannot be exported due to issue with one profile - ticket opened with Primekey
June 26 2020	Issue with export of config file solve and Compliance team able to parse config and confirm no other issuer profiles have wrong signature algorithm set or have misleading names with regard to configured CA
July 14 2020	Compliance team starts receiving automated Full Configuration Export for parsing/verification
July 15 2020	Parse results show all currently configured certificates to be correctly configured

3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.

GlobalSign has ceased issuance as of May 26 2020

If revocation is required, decision and rationale for delaying revocation. See "Timeline for revocation breached"

All non-expired remaining certificates are now revoked

4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.

21 certificates issued between 1 Aug 2017 and 13 May 2020

5. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.

https://crt.sh/?id=2802864202 (Expired)
https://crt.sh/?id=2637600829 (Revoked)
https://crt.sh/?id=2597110230 (Revoked)
https://crt.sh/?id=2586572564 (Revoked)
https://crt.sh/?id=2536505856 (Revoked)
https://crt.sh/?id=2004440901 (Expired)
https://crt.sh/?id=2004423059 (Expired)
https://crt.sh/?id=1995473997 (Expired)
https://crt.sh/?id=1493643533 (Expired)
https://crt.sh/?id=1493407346 (Expired)
https://crt.sh/?id=1476584968 (Expired)
https://crt.sh/?id=1476577935 (Expired)
https://crt.sh/?id=1476531292 (Expired)
https://crt.sh/?id=1473556980 (Expired)
https://crt.sh/?id=1314903185 (Expired)
https://crt.sh/?id=1312799930 (Expired)
https://crt.sh/?id=1300057995 (Expired)
https://crt.sh/?id=359273564 (Expired)
https://crt.sh/?id=355050149 (Expired)
https://crt.sh/?id=201007475 (Expired)
https://crt.sh/?id=182120957 (Expired)

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

In August 2017 an existing certificate profile was updated with a replacement CA. At that time the review process did not involve compliance approval or a programatic comparison between profiles and CAs, in this case the replacement CA algorithm did not match that of the original CA, and the profile name was not updated to match that of the replacement CA, as we do for all our profiles. Reviews of the issued test certificates failed to notice the issuer key/signature was in violation of the mozilla root policy due to the language used in version 2.5.

The issue was then noticed during investigation into another issue, the problem was fixed but there was no follow through of the checking of the population of previously issued certificates.

7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.

Monthly profile configuration exports are now being sent to compliance team for review. This review includes a mapping of issuer profiles with configured CAs, and a check against both.

As mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=1654896 we have re-informed the product team and development teams on the requirement that any new developments regarding compliance requirements have to be also tested on past certificates issued. We have modified our incident management checklists to include analyzing historical issuance data.

Comment 3

[Ryan Sleevi]

(In reply to Paul Brown from comment #2)

As mentioned in https://bugzilla.mozilla.org/show_bug.cgi?id=1654896 we have re-informed the product team and development teams on the requirement that any new developments regarding compliance requirements have to be also tested on past certificates issued. We have modified our incident management checklists to include analyzing historical issuance data.

Just trying to make sure I work through the timelines here.

2020-02-10 - GTS issue with wrong curve/hash
2020-04-16 - RSA-PSS issue
2020-05-13 - Last problematic cert issued (via GCC)
2020-07-23 - RSA modulus issue
2020-07-23 - GlobalSign instructs teams to ensure historic review as part of investigations.
2020-09-10 - Incident detected

Can you describe what set of issues GlobalSign has gone and examined historic issuance for? Has it, for example, re-run lint checks for the past 3 years of issuance, and ensured all certificates (expired, revoked, or valid) that have lint issues are clearly associated with an incident that has been reported and/or resolved? If I understand correctly, GlobalSign has/is already doing that for other (non-lint) issues, but this seems low-hanging fruit?

Can you describe your incident management process? For example, do you have four-eyes principles at play for ensuring these steps are followed, and that the actions taken are correct (i.e. not just that the box was checked, but it was correct in execution)

Comment 4

[Paul Brown]

Your timeline is basically correct however the issue in question, due to the fact it was found as a bi-product or intersection between an investigation for potential issues and investigation of an unrelated incident failed to be flagged as an incident of its own, whereas the actual incident that was being checked at that time was flagged for past issuance check. In July we added procedures to recheck historical data for any issues found during an investigation or disclosed in a bug reported by another CA, this being the one previous was missed due to it not flagging as an incident. We are still working towards a solution where we will have the resources in place for multiple ad-hoc reviews of historically issued certificates at any time, for instance for any new version of zlint - as detailed in https://bugzilla.mozilla.org/show_bug.cgi?id=1654896#c5:

We are also working on expanding both technical and human resource compliance bandwidth to ensure for each compliance-driven change the compliance team can, in parallel and using independent resources, perform an analysis on historical data wherever required. Current resources limitations within the compliance area require reliance on the engineering teams for some of these analyses. We expect to arrive at a level were each issuance analysis can be performed twice for all changes (by engineering and independently by compliance) by the first quarter of 2021.

Comment 5

[Ben Wilson]

Hi Paul,
Do you have any further updates that you can provide to complete/finalize this incident report with an eye toward bringing closure to it?
Thanks,
Ben

Comment 6

[Paul Brown]

Hi Ben

We are also working on expanding both technical and human resource compliance bandwidth to ensure for each compliance-driven change the compliance team can, in parallel and using independent resources, perform an analysis on historical data wherever required. Current resources limitations within the compliance area require reliance on the engineering teams for some of these analyses. We expect to arrive at a level were each issuance analysis can be performed twice for all changes (by engineering and independently by compliance) by the first quarter of 2021.

We remain on schedule to reach this goal by Q1 2021, apart from this we believe there are no actions outstanding

Comment 7

[Paul Brown]

Hi Ben

As an update: We are now starting the hiring process for a security & compliance data analyst in order to further refine this, however from the technical side we have updated our systems and the compliance team now have the capability to perform analysis on historical data without the need to engage members of the engineering team

Comment 8

[Ben Wilson]

Do you have any updates? Can this incident be closed?

Comment 9

[Arvid Vermote]

Hi Ben - we believe this bug can be closed. We will continue building our analysis capabilities within the compliance team, including independent log repositories of all issuance data, definition and implementation of custom compliance-focused monitoring and alerting rules and integration with our SOC for follow-up, handling and escalation.

Comment 10

[Ben Wilson]

I'll schedule this to be closed on or about Wed. 10-Mar-2021.