This report covers 3 certificates as described below.
- How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, via a discussion in mozilla.dev.security.policy, or via a Bugzilla bug), and the time and date.
#1) We monitor the issuance of the new certificate products closely, so we observed these issues as part of our auditing process. Once we noticed this issue shortly after issuance (see #2 below).
#2 and #3) These were both for the same customer. We noticed the #2 and #3 shortly after issuance.
- A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
Issued: Mar 5 15:52:47 GMT
Observed: Mar 5 15:59 GMT
Communicated with Customer: March 5
Revoked: Mar 9, 09:20:52 GMT: We reached out to the customer on Thursday, 3/5 Friday 3/6, but it took until Monday to confirm revocation with them
Issued: Mar 13 08:59:58 2020 GMT
Observed: Mar 13 09:12
Communicated with Customer Mar 13. We communicated to the customer; however, because it was the weekend they did not get the message until Monday.
Revoked: Mar 20 03:18:51
Initially it was our understanding that the bug encountered in #2 was caused by the customer's selection of the UCC SAN option when requesting the certificate, so we had them submit a new order without that option selected. This was reviewed and approved for issuance by management (the validation agents did not issue with out approval, following the guidance provided to them as part of the remediation of #2 s stated below). Given this had same issue as #2 we stopped all issuance until a full root cause analysis could be identified.
Issued: Mar 14 16:01:07 2020 GMT
Observed: Mar 14 16:40
Communicated with Customer to let them know we plan to revoke: March 15
Revoked: Mar 20 10:43:05
General Incident related tickets and events:
Mar 9: Provided updated training to the Validation agents that process QWAC orders regarding the content of the OrganizationIdentifier
Mar 13: Based on #2, Provided direction to stop issuance of all QWAC certificates until further notice except if explicitly authorized my a member of the compliance team.
Mar 13, 11:55: opened ticket for developers defining the errors in QWAC #2 and #3 for their analysis and investigation
Mar 14: Stopped all issuance until root cause is identified
Mar 17 03:12 - Development team identified possible cause, but needed more time to confirm.
Mar 17 10:10 - Development team confirm bug in the RA application after certificate request data has been edited.
Mar 19 : Patch installed to address #2 and #3
Mar 20: Issuance permitted to resume
- Whether your CA has stopped, or has not yet stopped, issuing TLS/SSL certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
We temporarily stopped issuance of all QWAC certificates until the patch was applied and now we have continued issuance.
- A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
As listed above
- The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
As listed above.
- Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
#1: The Applicant entered entire PSP Identifier into the PSP Number field, "PSDFR-ACPR-10278" instead of 10278. While our validation agent verified the information with the proper authorities, they missed the fact that "PSDFR-ACPR" was repeated in the OrganizationIdentifier.
#2 and #3: The issue was due to a bug in our RA platform which is was encountered when the order data was modified. The result was a malformed certificate with the issued discussed above. Our QWAC certificate issuance is new and we've only issued a couple of certificates. The issue was detected within an hour of issuance.
- List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
#1: We provided a refresher training to the validation agents so they are aware of exactly how to validate and verify these fields in QWAC certificates. On 30 March we will be updating the QWAC ordering page with improved guidance for the Applicant as well as adding field validation to be sure that the customer entered PSP Number does not start with VAT, NTR or PSP.
#2 and #3: We have updated our QA testing process and methodology to include testing of modified certificate requests. In the past we've tested valid and invalid requests, but haven't included all the manual steps to modify each identity field that is editable to verify that no unforeseen issues were introduced. This is now in place.