1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
We became aware following the creation of the bug ticket, which was filed at 21:55 UTC on 30/04/2021. We also received a certificate problem report via our email@example.com email address at 21:50 UTC on 30/04/2021. Both these reports created the corresponding tickets in our internal SOC, which escalated to the Compliance team, who started the investigation for those tickets at 01/05/2021 - 06:53 UTC . The initial focus of the report was the State field. The fact that the issue actually lay within the locality field became apparent upon closer inspection following Comment 2 at 09:44 UTC.
2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
|DD/MM/YYYY - Time in UTC
||Profile containing L=Noord Holland and S=Amsterdam was activated.
||Due diligence by a person who is not responsible for the collection of information becomes mandatory for all profile based vetting. reviously it was only required for EV profile based vetting.
||Allowlist State values Netherlands completed. S=Amsterdam was added, following an earlier customer request (non-SSL) containing S=Amsterdam in QIIS (Locality for that request was Amsterdam Zuidoost)
|30/04/2021 21:50 and 21:55
||Bugzilla ticket and certificate problem reports created. Corresponding GlobalSign internal SOC tickets created. SOC operators escalate to the compliance team.
|01/05/2021 - 06:53
||Start of investigation by the compliance team. Focus on the State value.
|01/05/2021 - 09:44
||Comment 2 at 09:44
|01/05/2021 - 09:46
||Deactivated profile, pending update to values. Combined vetting management and compliance review for all requested certificates.
|01/05/2021 - 10:04
||Reactivated profile containing updated information.
|01/05/2021 - 10:14
||Started the investigation for other non-reported certificates with exact same problem.
|01/05/2021 - 10:31
||All certificates identified (see #4), replacement process starts.
|01/05/2021 - 14:03
||Started the review of currently active profiles and full historic issuance review.
|05/05/2021 - 20:00
||All affected certificates listed in this incident report revoked.
|07/05/2021 - 20:00
||Targeted completion date of the review of currently active profiles
|31/05/2021 - 08:00
||Targeted completion date of full historic issuance review.
3. Whether your CA has stopped, or has not yet stopped, certificate issuance or the process giving rise to the problem or incident. A statement that you have stopped will be considered a pledge to the community; a statement that you have not stopped requires an explanation.
We halted issuance on the profile within minutes of Comment 2, at 09:46 UTC on 01/05/2021, pending updates to the profile. Our on-call vetting staff were contacted, and the updated profile was activated at 10:04 on the same day.
4. In a case involving certificates, a summary of the problematic certificates. For each problem: the number of certificates, and the date the first and last certificates with that problem were issued. In other incidents that do not involve enumerating the affected certificates (e.g. OCSP failures, audit findings, delayed responses, etc.), please provide other similar statistics, aggregates, and a summary for each type of problem identified. This will help us measure the severity of each problem.
5. In a case involving certificates, the complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem. In other cases not involving a review of affected certificates, please provide other similar, relevant specifics, if any.
6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
The Locality and State information were entered in the wrong fields, and this was not picked-up by a vetting operator. As we mentioned in Comment 1, the certificate requests were not flagged because the State value had been allowlisted. This allowlisting happened following the inclusion of the value in a different customer request. For that specific request, the Incorporating Agency listed the Locality as Amsterdam Zuidoost ("Amsterdam-Southeast"), which is a borough of Amsterdam. Even though these boroughs are part of the City of Amsterdam, they have a clearly defined territory and a directly elected district committee. The City of Amsterdam is one step up in terms of administrative levels.
Since the original activation of the profile, GlobalSign has introduced additional mandatory due diligence checks for all profile based vetting, to reduce the the risk of human error on any validation aspect (not just content of the certificate).
We are in the process of reviewing all currently-active profiles to make sure the Locality information is accurate. In parallel, we are performing a full review of historically issued certificates for inaccuracies in the Locality information.
7. List of steps your CA is taking to resolve the situation and ensure that such situation or incident will not be repeated in the future, accompanied with a binding timeline of when your CA expects to accomplish each of these remediation steps.
We are looking for automated ways in which to address this issue. We are analysing the implementation requirements for multiple solutions and will share the implementation date as soon as the analysis is complete. We are looking at the possibility of introducing an allowlist similar to the allowlist we currently have for the State field, or alternatively, seeing how we can leverage data from a third party source like Geonames to help safeguard the accuracy of the values in the Locality field. For this, we are also keeping in mind the comments made about non-obvious BR violations on https://bugzilla.mozilla.org/show_bug.cgi?id=1707073.
While this analysis is being performed, and pending a root cause fix, any new certificate requests are being reviewed by vetting management and compliance, to ensure that they only contain accurate values.
|End date DD/MM/YYYY
||Combined vetting management and compliance review for L values for all requested certificates.
||All affected certificates listed in this incident report revoked.
||Target end-date for the review of currently-active profiles.
||Target end-date of full historic issuance review.
||Analysis of implementation requirements for multiple solutions.