1462423 - NetLock: CN not in SAN

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Andrew Ayer]

NetLock has issued 7 pre-certificates in which the sole DNS SAN (vargaviktor.hu) is not the CN:

https://crt.sh/?sha256=693D3A4F1FC5418FFD8EE1CDE463923D49B21D3125F3664EF3151F87C4CEBDD0
https://crt.sh/?sha256=426CEED2844C74E4F78B7BA2E60E801DC53EB121B3F4C5B26D865ECC36067CBC
https://crt.sh/?sha256=8D564037F6CD64D171695C4C6998697451E5ED775900300EB2BA68301983D4CA
https://crt.sh/?sha256=1C0E3F0D41E523349AB31E65553CDA9B5BC656213E941B2EA76336BA181E4BFA
https://crt.sh/?sha256=7C6EC6A35D0F1F5CD0AD366A45288B679B215D4C27757B8CECFED49AE5450FA8
https://crt.sh/?sha256=E12E0C9DEA383796B323250FAB95EFBEAC9A502DB86193DA93A0F18E38B3C923
https://crt.sh/?sha256=DD01348DAA0F0009C107F8DED83BC401028EE38832B21EED81FD72A555295406

Note that the last pre-certificate seems to correspond to this certificate:

https://crt.sh/?sha256=DD89CA8E19A6E3E66700BB72AD2B004C94FFD6BDDA1E1C00495C921BA9CBD406

It has the same serial number and SPKI, but an entirely different SAN extension, which does include the CN. This suggests that the final certificate might not be misissued.  Nevertheless, RFC6962 says that pre-certificates are a binding commitment to issue a certificate, and thus pre-certificate misissuance is equivalent to certificate misissuance.

I've been in contact with [e-mail] about this issue.

Comment 1

[Wayne Thayer]

Viktor: As Andrew stated, this appears to be a set of misissued [pre]certificates. Please provide an incident report, as described here:
https://wiki.mozilla.org/CA/Responding_To_A_Misissuance#Incident_Report
The incident report should be posted to the mozilla.dev.security.policy forum and added to this bug.

Comment 2

[Ryan Sleevi]

Wayne: A couple updates on this:

1) There does not seem to be any update or response for 8 months
2) These certificates appear to have been revoked - for example, to check https://crt.sh/?sha256=693D3A4F1FC5418FFD8EE1CDE463923D49B21D3125F3664EF3151F87C4CEBDD0 , because it's using a pre-cert signing CA, we check for the serial number instead from the CTCA's issuer, and find https://crt.sh/?id=470763825 - revoked on 2018-06-11

This suggests the CA has taken some steps to address the issue, but has not acknowledged or filed an incident report. Further, this is even messier, as the precertificate has the bug, but the final certificate does not; thus, the precertificate represents a committment to issue a different certificate for the same serial number, and of course, would not match the issued certificate.

Comment 3

[Wayne Thayer]

Emailed primary and secondary POCs requesting a response.

Comment 4

[Varga Viktor]

Dear Wayne,

Thank you for your letter.

Unfortunately the ticket reminder (in May) from the bugzilla.mozilla.org didn’t arrived to me, this is why we don’t replied on this Mozilla ticket.
In short, we got Andrew Ayers letter in May, and we acted as needed. We responded to him under the ticket number Ticket#2018051710003351.

We will also update the ticket with more info soon.

Yours, 
Viktor Varga

Comment 5

[Varga Viktor]

Dear Mozilla,

Following you can find our report about the problem reported about the Bug 1462423.

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

On 2018-05-17 Andrew Ayer reported the problem through the standard email channel: [e-mail] used for revoking certificates. Unfortunately we missed the Mozilla bug ticked reminder at this time.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

After we got Andrews mail, we immediately stopped the issuance until the investigation. We identified 7 certificates with this problem (which have problematic CT certificates) and communicated with the end user about the problem. It was a configuration error in the production system on these certificates, which was the root cause.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

Upon receiving Andrew's email, we stopped the issuing certificates until the conclusion of the investigation. After the identification of affected certificates and cause, we made the necessary corrections, double checked, and then restarted the issuing.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
   

7 incorrect CT certificates were issued between 2018-05-10 and 2018-05-17 connected to issuing 7 standard certificates.

4. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

The problematic CT certificates have the following ids:
https://crt.sh/?id=453465677
https://crt.sh/?id=453471736
https://crt.sh/?id=454888650
https://crt.sh/?id=455334246
https://crt.sh/?id=463346340
https://crt.sh/?id=463665252
https://crt.sh/?id=465748957

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
   

It was a misconfiguration in the production system. A placeholder config entry was left in a specific cert type. We have new test cases and protocol in place to avoid this problem in the future.
7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We already modified the configuration and also added extra validation to block these types of errors.

sincerely yours,
Viktor Varga
Netlock

Comment 6

[Julien Cristau [:jcristau]]

(In reply to Varga Viktor from comment #5)

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

After we got Andrews mail, we immediately stopped the issuance until the investigation. We identified 7 certificates with this problem (which have problematic CT certificates) and communicated with the end user about the problem. It was a configuration error in the production system on these certificates, which was the root cause.

This doesn't actually seem to be a timeline, so doesn't really answer the question from the template.

Comment 7

[Varga Viktor]

Here are the updates to the point 2.

1. On 2018-05-17 Andrew Ayer reported the problem.
2. At 14:35 local time we stopped the issuance of the SSL certificates until the investigation of the problem.
3. At 17:30 local time we modified the problematic config.
4. At 23:30 local time we replied the our founding to Andrew Ayer.
5. Next day, our customers of affected certificates were contacted, to talk about the issue and the replacement of these certs.
   (because this was at the beginning of the CT mandation, not all of the customers wants to replace it immediately, because they worked for a few days before the CT forcing Chrome will be popular)
6. New certificates were issued for these customers, with right and after the replacement of the 7 affected certs each of the m were revoked. (these were revoked between 05-23 and 06-18, after the customers replaced the affected ones.)

Comment 8

[Ryan Sleevi]

Thanks. With a clearer understanding of both timeline and the root cause, can you now share more details about

We have new test cases and protocol in place to avoid this problem in the future.

You can see examples of detailed responses at https://bugzilla.mozilla.org/show_bug.cgi?id=1518555#c6 or https://groups.google.com/forum/#!msg/mozilla.dev.security.policy/vl5eq0PoJxY/W1D4oZ__BwAJ

Comment 9

[Ryan Sleevi]

Any updates?

Comment 10

[Varga Viktor]

On 05-18 the following immediate actions were taken:

1. The placeholder was set as a break point.
   If it is found in a request, it blocks the process before issuing the end user certificate.

2. The testers were mandated to use certlint instead x509lint
   When testing with x509lint this is not reported as an error, so it was replaced.
   (generally we prefer the C versions, but as it turns out this was not complete)

Our long term plan:
3. The integration of one of the lints into our software as set by the Mozilla recommended practices. The planned deadline for this is the end of 2019H1.
As part of this integration process we found the problem with x509lint.
Unfortunately this check was commented out from the code, because of some IDN false positives.

Comment 11

[Ryan Sleevi]

Wayne: Checking in with you on the timeline here. Based on Comment #10, it sounds like it will take NetLock ~1 year to implement pre-issuance linting (based on the 05-18 to end of 2019H1).

Comment 12

[Wayne Thayer]

Ryan: while it is certainly discouraging that a CA would choose to live with the risk of continued misissuance for so long, Mozilla has not set any deadline for implementing pre-issuance linting so I don't think it's appropriate to question the proposed timeline.

Comment 13

[Ryan Sleevi]

Any updates?

Comment 14

[Ryan Sleevi]

Wayne: No updates, and I'm not sure if your Comment #12 meant to close out this issue or to wait for the implementation of pre-issuance linting?

Comment 15

[Wayne Thayer]

Emailed primary and secondary POCs again requesting a response.

Created bug 1572992 requesting an incident report for repeated failures to respond and provide timely updates.

Comment 16

[Varga Viktor]

Dear Ryan,
I was not sure, reading your comment, that you want report about the implementation of linting.
After your later question to Mr. Wayne and the change of needinfo tag also foxed me, I thougt that you don't need reply anymore from me.
Now I understand that only the striked needinfo is cleared.
Of-course the linting with some other checks were in time implemented.
Yours, Viktor

Comment 17

[Wayne Thayer]

(In reply to Varga Viktor from comment #16)

Of-course the linting with some other checks were in time implemented.

Viktor: will you please provide some more details on this comment?

• when was linting implemented?
• is the linting performed prior to certificate issuance, post issuance, or both?
• are you using a publicly available linter like zLint? If so, how do you ensure that the linter code is kept up to date?

Comment 18

[Varga Viktor]

Dear Wayne,
Here are the detailed answers on your questions. I added 5 new affected certificates to this case.
Yours, Viktor

1. when was linting implemented?

Our release which included the x509lint was deployed on June 27th in our productive issuance systems. Later we need to update our code on august 26th, because 5 new mis-issued certificates.

The cause of the were the implementation way we followed to integrate the x509lint.

1. Because the false positives for non-SSL certificates (and an error blocks the issuance of a certificate), we need to implement these checks only on SSL certificates. (The x509lint does not handle the RFC 4043 permanent id SAN part and gives the "ERROR: Invalid type in SAN entry" message and also gives false positives on OCSP certificates.)
2. This filtering was updated later on august 26th after we found out, that it doesn't stopped some certificates with problems, which should be covered by. (It is filtering out the PSD2 specific certificates also. This filtering problem resulted, that after a wrong configuration 4 pre-certificates and 1 public were issued with wrong UserNotice length.
   The affected CT certs were:
   https://crt.sh/?id=1797981238
   https://crt.sh/?id=1793479883
   https://crt.sh/?id=1792373369
   https://crt.sh/?id=1773952434
   The issued public cert is attached to the ticket. The replacement certificate for this certificate was already issued,
   but the customer asked a week to replace it on its server before revoke.

These were identified by a verification trough the crt.sh on august 22nd, when a follow up check was done. The configuration
The problem was solved with the update on august 26th.
I do not started new ticket for these faults, because its connected to handling this bug.
If you would like a different ticket for these please mention it, and I will open another ticket.

Our next ETSI audit will cover also this incident case too.

2. is the linting performed prior to certificate issuance, post issuance, or both?

The linting is only before the issuance.

3. are you using a publicly available linter like zLint? If so, how do you ensure that the linter code is kept up to date?

We integrated the x509lint with a minimum set of modifications.

1. In Issue #9 ( https://github.com/kroeckx/x509lint/issues/9) a part of code was commented out in 2016, which means, x509lint does not compares the SAN with the commonName. We need this functionality to handle the original problem in this ticket well.
2. Because the false positives for non-SSL certificates (and an error blocks the issuance of a certificate), we need to implement these checks only on SSL certificates. (The x509lint does not handle the RFC 4043 permanent id SAN part and gives the "ERROR: Invalid type in SAN entry" message and also gives false positives on OCSP certificates.)
3. To update it as soon as need, I am watching https://github.com/kroeckx/x509lint on github, to get alert about updates.
4. Our developers can handle the task of update code from the master branch and compile it with our modifications.

Comment 19

[Varga Viktor]

Attachment: SSL certificate with wrong userNotice length

Comment 20

[Wayne Thayer]

It appears that all questions have been answered and remediation is complete.