Closed Bug 1586795 Opened 6 years ago Closed 5 years ago

NetLock: Issuance of intermediates after 2019-01-01 that do not comply with Mozilla Policy

Categories

(CA Program :: CA Certificate Compliance, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: ryan.sleevi, Assigned: varga.viktor)

Details

(Whiteboard: [ca-compliance] [ca-misissuance])

During a spot-check of Mozilla Policy Compliance, I discovered that NetLock issued intermediates that do not conform with Mozilla Policy 2.6.1

Mozilla Policy 2.6.1 requires (formatted for readability), in Section 5.3:

Intermediate certificates created after January 1, 2019, with the exception of cross-certificates that share a private key with a corresponding root certificate:

  • MUST contain an EKU extension; and,
  • MUST NOT include the anyExtendedKeyUsage KeyPurposeId; and,
  • MUST NOT include both the id-kp-serverAuth and id-kp-emailProtection KeyPurposeIds in the same certificate.

The following intermediates, issued after 2019-01-01, lack an EKU extension:

Please provide an incident report, as detailed at https://wiki.mozilla.org/CA/Responding_To_An_Incident

Flags: needinfo?(varga.viktor)
Status: NEW → ASSIGNED
  1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

As part of the ongoing audit we are reviewing the complete certificate issuance and Policy compliance.
On 2019-10-04, we finished this verification and identified the non-compliant and mentioned section.
I was preparing the report, but Mr. Ryan reported it on 2019-10-07 before we did.

  1. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

i. On the 2019-10-04 we identified and analyzed the bug, and the following tickets were introduced to our JIRA:

  1. We decided to replace those certificates and migrate the end entity certificates to the new CAs. We didn't revoke immediately the subCAs, because the end-entity should migrated before as we do not want to disrupt the usage of those.
  2. The modification of the x509lint to disable the possibility of this error by the code.
    The following modification will suits this:
    SetCertInfo(CERT_INFO_NO_EKU);
    if (type == SubscriberCertificate)
    { Set--Error(WARN_NO_EKU); }
    to
    SetCertInfo(CERT_INFO_NO_EKU);
    if (type == SubscriberCertificate)||(type == IntermediateCA)
    { Set-Error(ERR_NO_EKU); }

ii. On 2019-10-07 we got the report from Ryan.

  1. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

We are not issuing certificates with this problem anymore.

  1. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

The report already includes all the three affected certificates.
Number of certs: 3
First and last date: 2019-04-02; 2019-05-27

  1. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

The report already includes all the three affected certificates.

  1. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

The following causes leaded to this errors:

i. There is some contradiction between the practice and the BRG and EVGL.
According to BRG 1.6.6 section 7.1.6.3 and EVGL 1.7.0 section 9.3.4 for a Subordinate CA that is an affiliate of the Issuing CA MAY include the AnyPolicy identifier in the CertificatePolicies.
Following these policies, we created the following 2 SubCA certificates back in September of 2016 :
https://crt.sh/?caid=33991
https://crt.sh/?caid=34165
After we finished the EV inclusion at the Microsoft we did a lot of inspection, to find out, why the end entity certificates aren't green in IE or EDGE.
In the end we found out, that the general implementation and the policy of those sections are not the same, the subordinate CA that is an affiliate of the Issuing CA SHALL have an AnyPolicy at minimum.

ii. After we finally found out the cause of the "greenlessness" we immediately replaced it. Before the issuance of these SubCA certificates we checked everything that is technically possible for a precheck before the actual issuance, but there was no tool to detect this type of problem.
The third certificate was issued with the same settings.

  1. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

The following development steps will exclude the possibility of future errors, and to resolve the actual problem.

i. Replacement of these intermediate certificates and migration of the end entity certificates to this new CA,
the target deadline for the new certs is 10-15 except the last one:
https://crt.sh/?q=f90aca63cb8bb44f8d91b864474bb42aae177e73fdfe6f4acbd12a013a415c15
This is a signer CA and for these there are no recommendations for the subCA EKUs and it's need to be tested for the usable EKU combinations before the replacement, which needs more time.

ii. The modification of the x509lint to disable the possibility of this by code. the target deadline for this 10-15.
The following modification will suits this:
SetCertInfo(CERT_INFO_NO_EKU);
if (type == SubscriberCertificate)
{ Set--Error(WARN_NO_EKU); }
to
SetCertInfo(CERT_INFO_NO_EKU);
if (type == SubscriberCertificate)||(type == IntermediateCA)
{ Set-Error(ERR_NO_EKU); }

Flags: needinfo?(varga.viktor)

Viktor: Mozilla specifically notified CAs of this change and Netlock responded that they understood and would comply [1]. Please explain why Netlock failed to meet the commitment that was made, and how this will be prevented in the future.

[1] https://ccadb-public.secure.force.com/mozillacommunications/CACommResponsesOnlyReport?CommunicationId=a051J00003rMGLL&QuestionId=Q00072,Q00073

Flags: needinfo?(varga.viktor)

If I understand correctly:

  • The first two certificates were misissued because NetLock originally issued certificates in violation of Microsoft policy, revoked them, but re-used the (old) profiles to generate new versions, and did not review that the new profiles conformed to the new requirements.
  • There's no explanation as to why the third certificate happened.

The fix mentioned in Comment #1 would still not comply with Mozilla Policy, so that's equally concerning.

(In reply to Ryan Sleevi from comment #3)

If I understand correctly:

  • The first two certificates were misissued because NetLock originally issued certificates in violation of Microsoft policy, revoked them, but re-used the (old) profiles to generate new versions, and did not review that the new profiles conformed to the new requirements.

No. The first two certificates are not violating the MS policy. The original certificates were compliant with all published policies at issuance time, but because they are not going to green, we need to replace it. After we identified what are the causes of the problem we replaced those certificates. Unfortunately we missed this requirement at the review before the issuance.

It was a replacement of the earlier near to expiration CA certificate:
https://crt.sh/?id=12721773
It was issued with the same settings as the two before, because we missed this requirement.

The fix mentioned in Comment #1 would still not comply with Mozilla Policy, so that's equally concerning.

Can you tell me, why the code not fits for this? We are blocking issuance, if x509lint gives an error.

(In reply to Wayne Thayer [:wayne] from comment #2)

Viktor: Mozilla specifically notified CAs of this change and Netlock responded that they understood and would comply [1]. Please explain why Netlock failed to meet the commitment that was made, and how this will be prevented in the future.

[1] https://ccadb-public.secure.force.com/mozillacommunications/CACommResponsesOnlyReport?CommunicationId=a051J00003rMGLL&QuestionId=Q00072,Q00073

Human error. To prevent human errors like this, we reviewed our processes, and added another contact person to CCADB (which was not possible at the time of commitment).
(look at https://bugzilla.mozilla.org/show_bug.cgi?id=1572992 )

Flags: needinfo?(varga.viktor)

Dear Ryan,
Can you give me clarification about your opinion?
Yours, Viktor

The fix mentioned in Comment #1 would still not comply with Mozilla Policy, so that's equally concerning.

Can you tell me, why the code not fits for this? We are blocking issuance, if x509lint gives an error.

Flags: needinfo?(ryan.sleevi)

Thanks for the NI Viktor!

The issue is the proposed change doesn't encompass all of the Mozilla Requirements within Policy 2.6.1. It is not simply sufficient to ensure the EKU extension is present, but that the EKU extension is consistent with policy. Comment #0 captures those, while the proposed change in Comment #1 does not fully account for those.

Flags: needinfo?(ryan.sleevi)

Thank you Ryan!
At first I thought you found some bug in the code.
You have right, for blocking the full combination the following code also needed:

// checks.c
// insert after line 1391
if (type == IntermediateCA) {
SetError(ERROR_ANY_EKU_IN_INTERMEDIATE_MOZILLA);

}
//insert after line 1425
if (GetBit(cert_info, CERT_INFO_SERV_AUTH) && GetBit(cert_info, CERT_INFO_EMAIL) )
{
if (type == IntermediateCA) { SetError(ERROR_INVALID_EKU_COMBO_IN_INTERMEDIATE_MOZILLA); }
}

Maybe both the type checking is unnecessary, because

  1. on 2020-04-01 AnyEKU will be not usable anymore in end entity certs.
  2. if the serverAtuth, emailProtection combo invalid on the level of intermediate, it needs to be invalid on end entity too.

Can you do a quick check on this code too?
Thanks, Viktor

Flags: needinfo?(ryan.sleevi)

The first check - anyEKU in intermediate - is good to have regardless. The second check - two EKUs in intermediates - is good, but as you note, it'd also be a problem for leaf certs. So you could simply ignore the type of cert, and check it as an invalid combo.

Note that the second check will result in false-positives for certificates that are cross-signed (which are permitted to have EKUs, under certain constraints, per Comment #0).

You can see discussion about this in the following check - https://github.com/zmap/zlint/pull/323 - which proposes to add a variety of Mozilla policy checks to zlint.

Flags: needinfo?(ryan.sleevi)

The modified checking code will be in PROD in 2019-11-18.
We are not planning cross-certificates, so its good for us.
We will mention this, when commiting back the changes to x509lint.

I'm a little unclear. Are these CA certificates going to be revoked/replaced?

Flags: needinfo?(varga.viktor)

Dear Ben,
In short yes, its in progress. I woild like to udpate the ticket soon as we finsihed with it.
Yours, Viktor

Viktor: It's been 7 months since NetLock was informed its not compliant with Mozilla policy, and there's been seemingly zero action to come into compliance with Mozilla policy. Comment #12 doesn't meet the bar of https://wiki.mozilla.org/CA/Responding_To_An_Incident#Keeping_Us_Informed

Ben: I'd like to suggest Mozilla consider adding these to OneCRL, based on the lack of information/responsiveness. NetLock has had quite a bit of time to come into compliance and minimize any customer impact, and despite Comment #12, does not seem committed to actually doing so, based on the timeline of this bug.

Flags: needinfo?(bwilson)
Flags: needinfo?(bwilson)
QA Contact: wthayer → bwilson

Dear Ben and Ryan,

First of all let me assure you, Netlock is fully committed to reach and keep the full compliancy in case of Mozilla Root Policy so please accept my apologies again for the delay!
Below, I’m summarizing the actions we took until the last entry:

Regarding EV CA please find below the action points:

• On 10th of February we issued the new certs, however we’ve realized that we had to fix the issuer software as well due to HSM firmware incompatibility so within a short period we started we initiated that development as well
• On 11th of March: COVID-19 regulation came into force in Hungary as well and caused some delay on the development
• On 8th of April, finally we released and deployed the fix, allowed the issuance of OCSP, SC certificate from that point
• On 18th of May Hungary got back to the regular operation, so we started the migration of the customers regarding the impacted CAs.
• 31th of May, We’ve scheduled the revocation of old certificates and finishing the migration.

In case of MKB Subca please find below the status:
• Majority of certificates are in cards, so we had to postpone the mass-replacement since the quarantine and other COVID restrictions. Right now We are allowed to meet with customers again so we’ve initiated the swaps. The deadline here is end of June. (30th of June)

In order to keep the transparency on the proper level and grant the continuous communication, Peter Farkas (Project Manager) and Eszter Dolgos (Product Owner) have been assigned to the project. Please, feel free to approach them in case of any further question.

Let me highlight again, We are keeping in our mind the guidance of Mozilla (https://wiki.mozilla.org/CA/Responding_To_An_Incident#Keeping_Us_Informed) and reacting in the shortest time if any bug pops up!

Kind regards,
Viktor Varga

Flags: needinfo?(varga.viktor)

Dear Ben and Ryan,

Let me kindly give you a short status update regarding our actions made in the last 7 days.
All our affected customers have been informed about the necessary EV SSL certificate changes and revocations. The finalization of the exact plan for the migration of the certificates is currently in progress.

Kind Regards,
Eszter

I'll avoid setting NextUpdate this round, since the next update is scheduled for the end of this month with the revocation of the EV CA.

I'm setting Needs-Info for Ben, since the plan not to revoke MKB CA until July (maybe) is a little concerning. This is especially so, given that the cert was issued 2019-05-27, the non-compliance was flagged 2019-10-07, and it may not be until 2020-06-30 that this certificate ends up being revoked.

(In reply to Varga Viktor from comment #14)

• On 10th of February we issued the new certs, however we’ve realized that we had to fix the issuer software as well due to HSM firmware incompatibility so within a short period we started we initiated that development as well

Can you provide more technical details here? What does this mean, what was the incompatibility, and how was it fixed?

• On 8th of April, finally we released and deployed the fix, allowed the issuance of OCSP, SC certificate from that point

I'm not sure what the abbreviation SC means - Subscriber certificates?

In case of MKB Subca please find below the status:
• Majority of certificates are in cards, so we had to postpone the mass-replacement since the quarantine and other COVID restrictions. Right now We are allowed to meet with customers again so we’ve initiated the swaps. The deadline here is end of June. (30th of June)

This seems like it will be a continual challenge if your publicly trusted hierarchy is being used for private trust purposes. What sort of changes are being made to make sure that you can safely issue a Subordinate CA for a particular purpose, if that purpose will make it difficult to comply with the existing requirements (e.g. a CA for smart cards or, say, for hardware manufacturing, both which may pose challenges to replace/revoke the Sub-CA)? Some CAs have moved to fully transition their private or 'other' trusted purposes into separate hierarchies (e.g. an S/MIME root and a TLS root and a client cert root), precisely to avoid the challenges now being faced.

Flags: needinfo?(dolgos.eszter)
Flags: needinfo?(bwilson)

Kathleen,
Could you please add the MKB SubCA 5 to OneCRL? Here is a link to it - https://crt.sh/?q=f90aca63cb8bb44f8d91b864474bb42aae177e73fdfe6f4acbd12a013a415c15.
Thanks,
Ben

Flags: needinfo?(bwilson) → needinfo?(kwilson)

(In reply to Ryan Sleevi from comment #16)

Dear Ben,

we regret to see that you are planning to add MKB SubCA 5 to OneCRL. Please take into consideration that we are currently moving as fast as we possibly can to remedy the situation, but since this affects close to 200 chipcards, where we will need physical presence, handling this in the current situation due to the lockdown is not easy. We are committed to moving past this issue as soon as possible, and will try to expedite the process, but getting this done before the 30th of June is a stretch. Please allow us to resolve this issue with the client in order for this bank (MKB) to maintain business continuity.
Previously you inquired about the HSM firmware incompatibility. The ProtectServer HSM received a new firmware update where the MofN method changed and this was not clear in the documentation. This issue was not visible at first and thus passed testing.
You also asked about the meaning of SC previously. This was a typo and should have been SCT.
Furthermore, you wished to know more about why we use public trusted hierarchies for private trust purposes. I think this is a misunderstanding as we believe this to be a public keychain use-case. These certificates are being used to sign publicly circulated documents. As mentioned before our plan is to retire this CA as soon as possible and never had nor are we planning to create solutions which contradict the regulations.
At this point let me implore you to give us a little more time so we can safely lay this issue to rest.

Best regards,
Eszter

Flags: needinfo?(dolgos.eszter)

Adding to OneCRL will not impact their use in non-Mozilla products or, at present, non-TLS uses. By your own description, that should have zero impact. However, accepting risk for millions of Firefox users does seem unreasonable, and this performs an important mitigation for the risk caused by NetLock’s failures.

Is it incorrect to say these are not being used for TLS?

Flags: needinfo?(dolgos.eszter)

(In reply to Ryan Sleevi from comment #19)

Is it incorrect to say these are not being used for TLS?

It is correct, they are not being used for TLS.

Flags: needinfo?(dolgos.eszter)

Dear Ben and Ryan,

we have issued new certificates for all parties affected by this problem previously this week but unfortunately despite the fact that we have contacted the customers personally and expressed the urgency of the situation they have reacted slower than expected. Since the 31st of May and the first of June are public holidays in order to make the transitions problem free for the end customers the EV SSL end user certificates and EV SSL intermediate certificates will be revoked on 2nd July EOD. I would like to reassure you that we are doing everything in our power to remedy this situation as fast and painlessly as possible.

Sincerely yours,
Viktor Varga

Dear Ben and Ryan,

Hereby I would like to update you about the actions we have taken until EOD 06.02: revocation of the EV CA and all end user certificates has occurred and our customers have managed to finish migrations.

Kind Regards,
Eszter

I have indicated in the CCADB that the "MKB SubCA 5" subCA is ready to be added to OneCRL, so that it will be added as part of our next batch of OneCRL updates. As Ryan noted in comment #19, OneCRL only impacts TLS certificates.

I also confirm that the other two subCAs, "NETLOCK Trust Qualified EV CA 2" and "NETLOCK Trust EV CA 2", have been revoked and are also indicated in the CCADB as ready to add to OneCRL.

Flags: needinfo?(kwilson)

Dear Ben and Ryan,

Let me update you about the latest actions that have been taken: MKB SubCA 5 has been successfully renewed and uploaded to CCADB. Our customer is currenty verifying that all the end user certificates are functioning correctly, we are awaiting confirmation. Revocation of old certificate will occur until 30th June, 2020.

Kind Regards,
Eszter

Dear Ben and Ryan,

hereby I would like to confirm that revocation of the old MKB SubCA 5 certificate has occurred today.

Kind Regards,
Eszter

Dear Ryan,

please kindly give us feedback on above comment. Is there any further information you require from us to close this issue?

Thank you!

Kind Regards,
Eszter

Flags: needinfo?(ryan.sleevi)

Thanks for setting the N-I bit! In terms of next steps and expectations:

  1. File a bug for the failure to revoke according to the BR-defined timelines. You should provide an incident report according to https://wiki.mozilla.org/CA/Responding_To_An_Incident . The goal of this (separate) incident report is to track the compliance failure of a failure to revoke, and look at the steps you're taking to ensure that any sub-CA from a publicly trusted hierarchy can and will be revoked within 7 days, as required by the BRs.
  2. For this incident report, I don't see a clear plan of action for why the Mozilla requirement was missed or what's being done to ensure future compliance. I understand human error contributed, but I think it's worth revisiting this issue and looking carefully through the responses to think "What could have been done to prevent this".
  3. Before closing this out, it would be good to "redo" the report from Comment #1, with all of the new details and insight you've learned specific to the original failure.

Again, we'll keep the delayed revocation as a separate incident. This will focus on the original failure and what sort of controls are being added to prevent future similar failures.

Flags: needinfo?(ryan.sleevi) → needinfo?(dolgos.eszter)

Dear Ryan,
As requested in the point 3 of your comment, here is our summary report.
We are also preparing a separate ticket about the failure to meet the Mozilla Root policy revocation deadline.

„How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done”.

i. We became aware of the problem on October 4, 2019, when we finished this verification and identified the non-compliant and mentioned section.
We were preparing the report, but Mr. Ryan reported it on October 7, 2019 before we could have sent it to you .
ii. On October 4,2019 we started to identify and analyze the bug, and tickets were introduced to our JIRA in order to block the possibility of this error.

We have decided to replace those certificates and migrate the end entity certificates to the new CAs. We didn't revoke immediately the subCAs, because the end-entity certificates should be migrated first because we do not want to disrupt the usage of those certificates. The missussuance was reviewed, and we found out that the BR does not make mandatory the immediate revocation. Therefore, we didn't realize that Mozilla Policy also provides revocation time rules.

iii. On October 07,2019, we got the report from Ryan.
iv. On October 09, 2019 to prevent human errors like this, we have reviewed our processes, and added another contact person to CCADB (which was not possible at the time of first commitment).
v. On November 18, 2019. The modified checking code that was meant to prevent repeating the error went live on November 18, 2019.
vi. On 10th of February we issued the new certs, however we’ve realized that we had to fix the issuer software as well due to HSM firmware incompatibility so within a short period we started that development as well
vii. On 11th of March: COVID-19 regulation came into force in Hungary as well and caused some delay in the development
viii. On 8th of April, finally we released and deployed the fix, we made the necessary maintenance and repair that allowed us to do the final steps (the issuance of OCSP responders and Transparency signer Certificate)
ix. On 18th of May Hungary got back to the regular operation, so we started the migration of the customers regarding the impacted CAs.
x. 31th of May, we revoked the old EV SSL certificates and finish the migration.
xi. 30h of June we revoked the old MKB SubCA certificate.

"Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation."

We are not issuing certificates with this problem anymore and also a technical block applied on this bug on November 18, 2019.

"A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued."

Number of certs: 3
First and last date: 2019-04-02; 2019-05-27

https://crt.sh/?id=1379548755
https://crt.sh/?id=1490728474
https://crt.sh/?id=1639470777

"The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem."

See at summary of certificates.

"Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now."
The following causes led to these errors:
When we were creating these new subCAs, the timed requirement of Mozilla Root Policy was missed because of human error, and 3 subCAs was issued without the needed extensions.
We also missed the Mozilla Root policy revocation time requirements.

"List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things."

You can see the list of steps in the timeline above.

Kind Regards,
Viktor

Flags: needinfo?(ryan.sleevi)

Thanks for providing this, but I'm still concerned that Comment #29 is still missing a lot of the data shared elsewhere on this bug, when the whole goal of Comment #28 was to get the CA to provide this information (in part, showing that they have fully integrated it). My concern is that this incident report seems to treat it as "Something bad, we fixed it", without looking to understand or address the "why/how something bad happened in the first place"

You can see this through Comment #2, 10 months ago. I understand you responded in Comment #5, but you can see how that's not integrated in Comment #29. Similarly, Comment #11 / Comment #13 points out that Netlock was still violating policy, but nothing in Comment #29 looks to address that systemically, or show how holistically policy is being reviewed going forward. I don't see a reference to the separate bug yet, in this issue, so that's also good to add.

As I mentioned, we want to treat this incident holistically, understanding both the compliance failure and the factors that lead to it. This includes details about the bug (which Comment #1 touched on), but also understanding everything that went wrong, such as misunderstanding the policy (e.g. around revocation) and committing to a policy that you didn't actually implement (e.g. the CA communication).

Flags: needinfo?(varga.viktor)
Flags: needinfo?(ryan.sleevi)
Flags: needinfo?(dolgos.eszter)
Flags: needinfo?(varga.viktor) → needinfo?(dolgos.eszter)

Thanks for the clarification, i will redo the summary, but I would like to answer until that.
The factor which leds to the violations:
Missing review of the long lasting CA processes, when a new policy item became effective.

Two key dates were missed:

a) January 1st 2019 - EKUs mandated
These SubCAs were replacement of older CAs which had the same settings, but under this change rules changed.
To be sure, we don't get the same error, the x509lint was integrated as soon as possible after the failure.

b) January 1st 2020 - revocation of the CAs mandated by the Mozilla Policy 2.7
With the Mozilla policy 2.7 effective on this date, the revocation rules were changed, which means that after that change the subCAs need to be revoked following the BR 4.9.1 timing.
Because this ticket was started before this date, the revocation rules of 2.6.1 policy was in focus. This caused to miss this date too.

Now there is a review in place when a new effective date coming, to verify any long lasting CA processes against the new effective rules.

Flags: needinfo?(ryan.sleevi)
Flags: needinfo?(ryan.sleevi)

Dear Ryan,
The bug is available at https://bugzilla.mozilla.org/show_bug.cgi?id=1656882
Yours, Viktor Varga

Flags: needinfo?(ryan.sleevi)

Do you have a redone report, as per Comment #31?

Flags: needinfo?(ryan.sleevi)

„How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done”.

i. We became aware of the problem on October 4, 2019, when we finished this verification and identified the non-compliant and mentioned section.
We were preparing the report, but Mr. Ryan reported it on October 7, 2019 before we could have sent it to you .
ii. On October 4,2019 we started to identify and analyze the bug, and tickets were introduced to our JIRA in order to block the possibility of this error.

We have decided to replace those certificates and migrate the end entity certificates to the new CAs. We didn't revoke immediately the subCAs, because the end-entity certificates should be migrated first because we do not want to disrupt the usage of those certificates. The missussuance was reviewed, and we found out that the BR does not make mandatory the immediate revocation. Therefore, we didn't realize that Mozilla Policy also provides revocation time rules.

iii. On October 07,2019, we got the report from Ryan.
iv. On October 09, 2019 to prevent human errors like this, we have reviewed our processes, and added another contact person to CCADB (which was not possible at the time of first commitment).
v. On November 18, 2019. The modified checking code that was meant to prevent repeating the error went live on November 18, 2019.
vi. On 10th of February we issued the new certs, however we’ve realized that we had to fix the issuer software as well due to HSM firmware incompatibility so within a short period we started that development as well
vii. On 11th of March: COVID-19 regulation came into force in Hungary as well and caused some delay in the development
viii. On 8th of April, finally we released and deployed the fix, we made the necessary maintenance and repair that allowed us to do the final steps (the issuance of OCSP responders and Transparency signer Certificate)
ix. On 18th of May Hungary got back to the regular operation, so we started the migration of the customers regarding the impacted CAs.
x. 31th of May, we revoked the old EV SSL certificates and finish the migration.
xi. 30h of June we revoked the old MKB SubCA certificate.

"Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation."

We are not issuing certificates with this problem anymore and also a technical block applied on this bug on November 18, 2019.

"A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued."

Number of certs: 3
First and last date: 2019-04-02; 2019-05-27

https://crt.sh/?id=1379548755
https://crt.sh/?id=1490728474
https://crt.sh/?id=1639470777

"The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem."

See at summary of certificates.

"Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now."
The following causes led to these errors:
When we were creating these new subCAs, the timed requirement of Mozilla Root Policy was missed because of human error, and 3 subCAs was issued without the needed extensions.
We also missed the Mozilla Root policy revocation time requirements.

The factors were:

  • the previously used linting method was not perfect - corrected with the integration of x509lint (with added checks)
  • for long lasting CA activities (like these iCA issance were) were not checks implemented when a MRP was changed - now after the second bug the process was changed
  • human error - it was too easy to forget about that one rule, which was not went in force on August 13, 2018 with the MRP 2.6.1 - we thought that we solved it with the change of the contact, but from the second bug we learnt that the process needs to be reviewed.

The following linked timeline shows were was the problem:
https://bugzilla.mozilla.org/attachment.cgi?id=9172841
The separated and long processes missed those red feedbacks. Now the compliance is responsible and monitoring these activities as external to IT.

"List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things."

You can see the list of steps in the timeline above.
Mississuance was blocked with the implemented security controls November 18, 2019.

Are there any remaining remediations that you are working on or that you have completed since your last update?

Flags: needinfo?(u663533) → needinfo?(varga.viktor)
Whiteboard: [ca-compliance] → [ca-compliance] Next Update 2020-12-01

It's completed. My last update was a summarizing of the ticket as requested
All those CAs were replaced with new CAs.

Flags: needinfo?(varga.viktor)

I will close this bug on or about Wednesday 18-November-2020 unless there are other issues to discuss.

Flags: needinfo?(bwilson)
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Flags: needinfo?(bwilson)
Resolution: --- → FIXED
Product: NSS → CA Program
Whiteboard: [ca-compliance] Next Update 2020-12-01 → [ca-compliance] [ca-misissuance]
You need to log in before you can comment on or make changes to this bug.