1391063 - QuoVadis: Non-BR-Compliant Certificate Issuance

Status: RESOLVED FIXED

(CA Program :: CA Certificate Compliance, task)

Description

[Kathleen Wilson]

The following problems have been found in certificates issued by your CA, and reported in the mozilla.dev.security.policy forum. Direct links to those discussions are provided for your convenience.

To continue inclusion of your CA’s root certificates in Mozilla’s Root Store, you must respond in this bug to provide the following information:
1) How your CA first became aware of the problems listed below (e.g. via a Problem Report, via the discussion in mozilla.dev.security.policy, or via this Bugzilla Bug), and the date.
2) Prompt confirmation that your CA has stopped issuing TLS/SSL certificates with the problems listed below.
3) Complete list of certificates that your CA finds with each of the listed issues during the remediation process. The recommended way to handle this is to ensure each certificate is logged to CT and then attach a CSV file/spreadsheet of the fingerprints or crt.sh IDs, with one list per distinct problem.
4) Summary of the problematic certificates. For each problem listed below: number of certs, date first and last certs with that problem were issued.
5) Explanation about how and why the mistakes were made, and not caught and fixed earlier.
6) List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
7) Regular updates to confirm when those steps have been completed.

Note Section 4.9.1.1 of the CA/Browser Forum’s Baseline Requirements, which states:
“The CA SHALL revoke a Certificate within 24 hours if one or more of the following occurs: …
9. The CA is made aware that the Certificate was not issued in accordance with these Requirements or the CA’s Certificate Policy or Certification Practice Statement; 
10. The CA determines that any of the information appearing in the Certificate is inaccurate or misleading; …
14. Revocation is required by the CA’s Certificate Policy and/or Certification Practice Statement; or 
15. The technical content or format of the Certificate presents an unacceptable risk to Application Software Suppliers or Relying Parties (e.g. the CA/Browser Forum might determine that a deprecated cryptographic/signature algorithm or key size presents an unacceptable risk and that such Certificates should be revoked and replaced by CAs within a given period of time).

However, it is not our intent to introduce additional problems by forcing the immediate revocation of certificates that are not BR compliant when they do not pose an urgent security concern. Therefore, we request that your CA perform careful analysis of the situation. If there is justification to not revoke the problematic certificates, then explain those reasons and provide a timeline for when the bulks of the certificates will expire or be revoked/replaced. 

We expect that your forthcoming audit statements will indicate the findings of these problems. If your CA will not be revoking the certificates within 24 hours in accordance with the BRs, then that will also need to be listed as a finding in your CA’s BR audit statement.

We expect that your CA will work with your auditor (and supervisory body, as appropriate) and the Root Store(s) that your CA participates in to ensure your analysis of the risk and plan of remediation is acceptable. If your CA will not be revoking the problematic certificates as required by the BRs, then we recommend that you also contact the other root programs that your CA participates in to acknowledge this non-compliance and discuss what expectations their Root Programs have with respect to these certificates.


The problems reported for your CA in the mozilla.dev.security.policy forum are as follows:

** Failure to respond within 24 hours after Problem Report submitted
https://groups.google.com/d/msg/mozilla.dev.security.policy/PrsDfS8AMEk/w2AMK81jAQAJ
The problems were reported via your CA’s Problem Reporting Mechanism as listed here:
https://ccadb-public.secure.force.com/mozilla/CAInformationReport
Therefore, if this is the first time you have received notice of the problem(s) listed below, please review and fix your CA’s Problem Reporting Mechanism to ensure that it will work the next time someone reports a problem like this.

** Certificates with metadata-only subject fields (at least one subject field that only contains ASCII punctuation characters)
https://groups.google.com/d/msg/mozilla.dev.security.policy/Sae5lpT02Ng/-lsC11JnBwAJ
Prevent further issuance of certs with N/A and other metadata but revocation not necessary in this case. 

** Short / sequential-looking serial numbers
https://groups.google.com/d/msg/mozilla.dev.security.policy/vl5eq0PoJxY/uD-Li1w1BgAJ
https://groups.google.com/d/msg/mozilla.dev.security.policy/YequotPYLdc/J0K3lUyzBwAJ

** Invalid dnsNames (e.g. invalid characters, internal names, and wildcards in the wrong position)
https://groups.google.com/d/msg/mozilla.dev.security.policy/CfyeeybBz9c/lmmUT4x2CAAJ
https://groups.google.com/d/msg/mozilla.dev.security.policy/D0poUHqiYMw/Pf5p0kB7CAAJ

Comment 1

[Stephen Davidson]

Acknowledged.  I will prepare separate responses for the 4 problem reports.

Comment 2

[Stephen Davidson]

Attachment: Spreadsheet of OU with metadata

Comment 3

[Stephen Davidson]

** Certificates with metadata-only subject fields (at least one subject field that only contains ASCII punctuation characters)
https://groups.google.com/d/msg/mozilla.dev.security.policy/Sae5lpT02Ng/-lsC11JnBwAJ
Prevent further issuance of certs with N/A and other metadata but revocation not necessary in this case. 

QuoVadis became aware of the issue on August 9 in the above mentioned m.d.s.p. posting.

The issue was caused by a misreading of the BR.  Section 7.1.4.3 (i) deals with Certificate Field: subject:organizationalUnitName.  Section 7.1.4.3 (j), which contains the metadata prohibition, deals with Other Subject Fields and begins “All other optional attributes…”  

QuoVadis has identified 16 valid SSL certificates with problematic OU fields.  These include the use of a single dot, hyphen/s, blank space, NA and N/A.  A spreadsheet of the certificates is attached.  QuoVadis does not intend to revoke these certificates.

We are currently testing a filter in our certificate management system to prevent the use of creative notations to show the disuse or inapplicability of the field, such as the use of only ASCII punctuation characters, blank space, NA or N/A.  We will be vigilant for additional variations.

That filter will be in production by September 1.

QuoVadis has informed its external auditors of the issue.

Comment 4

[Stephen Davidson]

Attachment: Siemens_SerialEntropy.xlsx

** Short / sequential-looking serial numbers
https://groups.google.com/d/msg/mozilla.dev.security.policy/vl5eq0PoJxY/uD-Li1w1BgAJ
https://groups.google.com/d/msg/mozilla.dev.security.policy/YequotPYLdc/J0K3lUyzBwAJ

(copying text sent to m.d.s.p listserv below) 

Thanks Ryan, and I note your further posting regarding prompt response.  Noting your desire for detail, I have hesitated to respond with partial answers as both Siemens and QuoVadis are working hard to fix the issues with the Siemens CA and to replace the certificates as quickly as possible.

However, I don’t wish to delay getting more information to you, and ask for your patience if complete information comes in iterations.

Siemens has previously indicated that the affected certificates are installed on high profile websites and infrastructure for Siemen’s group companies around the world, and that a rushed revocation would create more damage than could be expected from the serial number noncompliance.  

We have been working with Siemens to dramatically advance the date by which the affected certificates can be replaced and revoked.  Siemens predicts that the vast majority of the certificates can be replaced by September 30, 2017 with the few difficult cases following.

Addressing your questions:
1)      The failure was one of process rather than deployed code.  QuoVadis made an indepth review of the Siemens CA, policies and practices when we took over the rootsigning, just before the BR changes which raised the serial entropy requirements.  At that time it was compliant.   QuoVadis formally updates Siemens on changes to applicable standards, and the Siemens PKI team independently monitors groups like the CABF public and m.d.s.p. lists. Siemens were aware of the pending change to 64-bit serials and were prepared to implement them.  

I note that at the same time planning was underway to move from the in-house CA to an OSS CA – precisely for the reason of easing compliance with the increasing pace of change in technical aspects of the BR and other standards, such as CT, CAA and serial entropy.  It appears that by oversight, the update to bring the inhouse CA to 64-bits was not deployed, and our expectation was that the check would be made in the external audit.  The long transition to the OSS CA is close to completion.

2)      QuoVadis has a dedicated head of compliance and risk management who, in addition to overseeing QuoVadis’ own measures, supervises its external sub-CAs including detailed discussions on evolving standards, checks on implementations, as well as ongoing monitoring of certificate issuance.  There is frequent communication with Siemens and our other root-signed customers.

Siemens has a significant and mature internal audit and external audit regime.  QuoVadis placed too much reliance on the external ETSI TS 102042 V2.4.1 NCP+ DVCP/OVCP audit report for what should have been textbook issues like the serial number entropy.  Going forward, QuoVadis will increase the formality of notifications regarding BR approved ballots, requiring documented evidence of compliance by the effective date, and notification to auditors for scope.

Like many CAs, QuoVadis uses crt.sh/certlint to check certificate issuance including for external sub-CAs.  This perhaps led to a false sense of security, as certlint does not highlight issues with serial number entropy.  Moreover, the fleeting window of visibility in some crt.sh reports may not reveal older issues or certificates that have not appeared in CT. QuoVadis is introducing routine use of certlint in its own certificate management system, and will build an expanded view for our external subCAs (the new Siemens CA will log all SSL in CT, and we intend our other external sub-CAs to do so before the Google deadline).  

3)       I do not have sufficient information yet to answer your questions regarding the auditor’s practices, and cannot comment on Digicert’s (nor Verizon’s) previous practices.

4)       The list of affected certificates is attached in spreadsheet form;  they will be uploaded to CT as well.  You will note that the number has declined – Siemens' previous report did not take into account that some of the certificates had already previously been revoked for other reasons.   The spreadsheet also includes certificates issued during the Digicert/Verizon root signing period.

Comment 5

[Ryan Sleevi]

(In reply to Stephen Davidson from comment #3)
> QuoVadis has identified 16 valid SSL certificates with problematic OU
> fields.  These include the use of a single dot, hyphen/s, blank space, NA
> and N/A.  A spreadsheet of the certificates is attached.  QuoVadis does not
> intend to revoke these certificates.

Per the original message (Comment #0)

> If there is justification to not revoke the problematic certificates, then explain those reasons and provide a timeline for when the bulks of the certificates will expire or be revoked/replaced. 

Can you please provide that justification, as well as the relevant data?

Comment 6

[Ryan Sleevi]

(In reply to Stephen Davidson from comment #4)
> I note that at the same time planning was underway to move from the in-house
> CA to an OSS CA – precisely for the reason of easing compliance with the
> increasing pace of change in technical aspects of the BR and other
> standards, such as CT, CAA and serial entropy.  It appears that by
> oversight, the update to bring the inhouse CA to 64-bits was not deployed,
> and our expectation was that the check would be made in the external audit. 
> The long transition to the OSS CA is close to completion.

In the spirit of providing meaningful post-mortems, so that the community can be informed of how a CA handles incidents, and the overall Web PKI can improve through the collective experiences, I believe it would be useful to cover this as part of a public post-mortem.

For example, why was the in-house CA not deployed? What factors contributed to that delay? What controls could have detected or prevented this delay, or mitigated the risk from such delays? Why was QuoVadis not aware of the delay - or the impact it would have on compliance?

I think it's understandable to say 'mistakes happen', but our opportunity here is to look at the systemic factors that enabled those mistakes, and understand what, if any, changes can be made. The goal is not to assign blame to individuals, but rather, to look holistically at how it happened, and what can be done to prevent both this _specific_ case and the general set of problems in the future.

It sounds like QuoVadis was aware there were issues in the legacy system, but was waiting on a new audit report to ensure the transition was made. Is that a correct understanding of your response?

> 2)      QuoVadis has a dedicated head of compliance and risk management who,
> in addition to overseeing QuoVadis’ own measures, supervises its external
> sub-CAs including detailed discussions on evolving standards, checks on
> implementations, as well as ongoing monitoring of certificate issuance. 
> There is frequent communication with Siemens and our other root-signed
> customers.
> 
> Siemens has a significant and mature internal audit and external audit
> regime.  QuoVadis placed too much reliance on the external ETSI TS 102042
> V2.4.1 NCP+ DVCP/OVCP audit report for what should have been textbook issues
> like the serial number entropy.  Going forward, QuoVadis will increase the
> formality of notifications regarding BR approved ballots, requiring
> documented evidence of compliance by the effective date, and notification to
> auditors for scope.

I think this is a good example of identifying an underlying cause - too great of reliance on external audits. It sounds like your mitigations are to formally establish a process for changes.

Can you clarify what you mean by "notification to auditors for scope"?

> Like many CAs, QuoVadis uses crt.sh/certlint to check certificate issuance
> including for external sub-CAs.  This perhaps led to a false sense of
> security, as certlint does not highlight issues with serial number entropy. 
> Moreover, the fleeting window of visibility in some crt.sh reports may not
> reveal older issues or certificates that have not appeared in CT. QuoVadis
> is introducing routine use of certlint in its own certificate management
> system, and will build an expanded view for our external subCAs (the new
> Siemens CA will log all SSL in CT, and we intend our other external sub-CAs
> to do so before the Google deadline).  

This similarly sounds like a positive mitigation for risk.

In further exploring what steps QuoVadis is taking with respect to externally operated sub-CAs, it sounds like one possible mitigation is to ensure that QuoVadis is also listed as an authorized user of completed audit reports on behalf of sub-CAs (in addition to the sub-CA organization), and equally provided access to the audit documentation.

This would allow QuoVadis' compliance team the same visibility as the sub-CAs auditor, both in evidence and report, and would further help ensure appropriate oversight.

Was this considered? If not, is it something you can implement? If it was considered, why was it not implemented?



Overall, I think the responses are close to being to a place to close this issue out, but I'm hoping we can gain additional insight and mitigations to ensure appropriate supervision of independently-operated sub-CAs.

Comment 7

[Stephen Davidson]

Attachment: QuoVadis SSL with accents in SAN

** Invalid dnsNames (e.g. invalid characters, internal names, and wildcards in the wrong position)
https://groups.google.com/d/msg/mozilla.dev.security.policy/CfyeeybBz9c/lmmUT4x2CAAJ
https://groups.google.com/d/msg/mozilla.dev.security.policy/D0poUHqiYMw/Pf5p0kB7CAAJ
 
QuoVadis noted the July 26 and August 13 m.d.s.p. posts.  The both relate to the same issue, namely the use of accents (invalid characters) in SAN fields.
 
QuoVadis identified 8 SSL certificates within their validity period with invalid characters in SAN fields.  All have been revoked (6 previously) and are identified in the attached spreadsheet. 
 
The certificates were manually approved by an administrator, contrary to documented QuoVadis practice.   When it was discovered, training was reinforced and additional training provided in the use of punycode. The problem has not reoccurred since then.
 
QuoVadis is currently testing an improved filter in our certificate management system to systematically prevent the use of unallowed characters in CN and SAN fields for SSL.  That filter will be in production before October 1.
 
In addition, QuoVadis is introducing routine use of certlint in its certificate management system to catch these issues earlier. I cannot provide a date this will be effective, but in the meantime, QuoVadis makes use of crt.sh and other tools to identify issues.
 
QuoVadis has informed its external auditors of the issue.

Comment 8

[Jonathan Rudenberg]

(In reply to Stephen Davidson from comment #7)
> Created attachment 8900824 [details]
> QuoVadis SSL with accents in SAN

Can you please ensure that all certificates affected by all of the problems mentioned in this issue are logged to CT?

Comment 9

[Stephen Davidson]

(In reply to Jonathan Rudenberg from comment #8)
> (In reply to Stephen Davidson from comment #7)
> > Created attachment 8900824 [details]
> > QuoVadis SSL with accents in SAN
> 
> Can you please ensure that all certificates affected by all of the problems
> mentioned in this issue are logged to CT?

Yes, these will be logged next week.

Comment 10

[Stephen Davidson]

(In reply to Ryan Sleevi from comment #6)

> For example, why was the in-house CA not deployed? What 
> factors contributed to that delay? What controls could have 
> detected or prevented this delay, or mitigated the risk from 
> such delays? Why was QuoVadis not aware of the delay - or the 
> impact it would have on compliance?

As previously described, QuoVadis and Siemens have an ongoing formal dialogue and periodic conference calls regarding changes to the CA requirements.  There was a process failure to confirm that the 64-bit serials had been implemented in the inhouse CA. Siemens management believed it had been done, and QuoVadis had expected it to be covered in the external audit.

QuoVadis had active sampling – but our focus (and that of Siemens) was perhaps overly on appropriate validation and certlint-type errors. “Infrastructure” type changes to the BR now receive more weight in our own external checklists/sampling.

Although the process has worked well in the past, with the increasing pace and complexity of BR changes, QuoVadis and Siemens have agreed to step up the “per ballot” confirmation of compliance directly between us.  

> It sounds like QuoVadis was aware there were issues in the 
> legacy system, but was waiting on a new audit report to ensure 
> the transition was made. Is that a correct understanding of 
> your response?

Not exactly.  QuoVadis encouraged the adoption of the established OSS CA to reduce the possible unknowns of the inhouse CA, and to meet the changing requirements re CT, CAA, etc.  Siemens has been working since March 2017 on implementation of the new OSS CA, working with QuoVadis and their external auditors, DQS.  The new CA will have a new ETSI re-audit in place, and will log to CT.  Additionally, the next planned audit for February 2018 will step up from ETSI TS 102042 to ETSI EN 319 411-1.

Siemens has already started the exchange of all affected SSL certificates and its internal users are scheduled to complete the replacement by latest September 30. Siemens will report progress achieved at that time.  A small number of certificates may require additional time due to pinning issues; software updates are already in progress. Siemens will provide more information on these fringe cases as above.

> Can you clarify what you mean by "notification to auditors for
> scope"?

As you have noted in the past, the industry needs to ensure that current standard versions are used in all audits.  QuoVadis will work with Siemens to ensure that a clearer inventory of change areas that should get special attention is covered in their external audit scope.

> In further exploring what steps QuoVadis is taking with 
> respect to externally operated sub-CAs, it sounds like one 
> possible mitigation is to ensure that QuoVadis is also listed 
> as an authorized user of completed audit reports on behalf of 
> sub-CAs (in addition to the sub-CA organization), and equally 
> provided access to the audit documentation.

> This would allow QuoVadis' compliance team the same visibility 
> as the sub-CAs auditor, both in evidence and report, and would 
> further help ensure appropriate oversight.

> Was this considered? If not, is it something you can 
> implement? If it was considered, why was it not implemented?

Generally external audit firms are reluctant to share their internal working papers.  That being said we have a good relationship with this auditor and Siemens, and will make that request.  However, as described before, we intend to step up emphasis of changed areas in the scope of that audit, and to increase the formality of our own confirmations.

On a separate note, given the additional compliance work that QuoVadis intends to devote on our few existing SSL external subCAs, QuoVadis has made the decision to halt additional root signings involving serverauth.

Comment 11

[Ryan Sleevi]

Below is my summary of the issues and remediation plan:

1) Metadata-only subject OU fields
- See Comment #0, Comment #3
- Remediation
  - 2017-09-01 - Deploy a new filter in place for the OU field to address ASCII punctuation and other variations of "not applicable"

2) Short/Sequential Serials
- See Comment #0, Comment #4, Comment #10
- Existing Mitigations
  - 2017-03-XX - QuoVadis and Siemens had already identified deficiencies in Siemens in-house CA system and were transitioning to something more industry compliant, to reduce the risk of such issues
  - QuoVadis performs active sampling of issued certificates to detect compliance issues
- Remediation
  - 2017-08-25 - QuoVadis has decided to no longer certify new third-party subordinate CAs for serverAuth
  - 2017-08-25 - QuoVadis and Siemens have implemented a 'per ballot' review process between compliance teams to ensure compliance with the Baseline Requirements is met for all individual ballots
  - 2017-08-25 - QuoVadis will work more closely with Siemens' auditors to review scope and appropriateness
  - 2017-09-30 - Replace existing certificates and revoke existing. Note: It is expected that there may be some certificates valid beyond this date; QuoVadis and Siemens will provide updates at this date.
  - XXXX-XX-XX - Siemens will deploy new CA software

3) Invalid dNSNames
- See Comment #0, Comment #7
- Remediation
  - 2017-08-13 - Additional training was performed for administrators regarding the proper use of Punycode
  - 2017-08-24 - Monitoring via tools such as crt.sh for post-issuance problems
  - 2017-10-01 - Filtering for CN and subjectAltName fields to ensure proper A-labels
  - XXXX-XX-XX - Deployment of pre-issuance linting checks


Is that a correct/complete summary? Assuming it is, I've set this issue as "remediation-accepted". There are specific milestones that we want to ensure are met, or that updates are provided should they not be met, as outlined above. This includes 2017-09-01, 2017-09-30, 2017-10-01, and the deployment of Siemens' new CA software.

Based on the lack of a firm date, it sounds like Siemens' CA may not be prepared for the upcoming CAA compliance date - is that correct? Can you confirm that QuoVadis has a process to ensure compliance for the set of certificates to be replaced prior to 2017-09-30 to comply with the CAA policy (such as ensuring that Siemens is the Domain Operator, for example, as specified in the BRs)

Leaving the "Needs-Info" in place will cause Bugzilla to issue periodic reminders to provide updates to this bug, so clearing it is not advised :)

Comment 12

[Ryan Sleevi]

Was the deployment of #1 met?
Is the deployment of #2 on track?
Is the deployment of #3 on track?

Comment 13

[Stephen Davidson]

Hello Ryan:
Here is an update regarding Siemens (#2).
The affected certificates were issued by “Siemens Issuing CA Internet Server 2016”.  As described before, a new CA platform called “Siemens Issuing CA Internet Server 2017” will be used for issuance going forward, including the replacement of the affected certificates.  That CA has been disclosed in the CCADB.
QuoVadis, Siemens and their external auditor worked to verify compliance of the new CA, and an audit update was performed 23-30 August 2017.  Audit report is at https://www.siemens.com/corp/pool/pki/siemens_etsi.pdf
The new CA was given the go live to start issuing certificates on September 8 2017, both to commence the replacement of the affected certificates and to resume routine issuance for the group.  Certificates are being logged to Pilot and Rocketeer (full CT compliance will be in place before the deadline).
Siemens has actively engaged with their certificate holders around the world to replace/revoke the affected certificates and believe that nearly all the certificates will be replaced by end of September (leaving perhaps 6 certificates that require additional time due to pinning issues).  As we approach the end of the month, Siemens will provide an update of that progress.
For CAA purposes, Siemens has currently opted out as they are the DNS Operator for the domains to which they issue, which has been confirmed.  Siemens may however opt to use CAA in future.
The fixes for (#1) and (#3) are underway - delayed somewhat due to CAA - I will provide an update when they are in full production (expected to be complete soon).
Regards, Stephen

Comment 14

[Ryan Sleevi]

Can you please provide a concrete amended timeline for #1 and #3? As covered in https://wiki.mozilla.org/CA/Responding_To_A_Misissuance (which expanded upon and gave concrete examples for https://wiki.mozilla.org/CA:MaintenanceAndEnforcement#Potential_Problems.2C_Prevention.2C_Response ), part of this process is designed to help understanding timing and how the CA treats things with urgency.

I note that the audit was an ETSI TS 102 042 audit (criteria last updated 2013-02), rather than the EN 319 411-1 (criteria last updated 2016-02). Does QuoVadis feel this appropriately demonstrates the level of compliance and supervision expected? This is not a rejection of that audit - as Mozilla still permits it - but a highlighting that the level of controls and compliance demonstrated leave significant gaps in a parent CAs' ability to make an informed decision about a child CAs compliance to the Baseline Requirements.

Comment 15

[Stephen Davidson]

(In reply to Ryan Sleevi from comment #14)

The Siemens audit previously used the ETSI TS 102 042 criteria, with a full audit each year.  As such, the PITA/re-audit for the new CA this summer used that criteria, which is still allowed under Mozilla policies.  QuoVadis is mindful of the differences with the newer EN 319 411-1 criteria, and is working with Siemens and their auditors to transition to EN 319 411-1 as soon as possible.

Update for #1 and #3 coming.

Comment 16

[Stephen Davidson]

(In reply to Ryan Sleevi from comment #14)

Confirming that the update for #1 (improved filter for OU field to address ASCII punctuation and variations of "not applicable") and #3 (improved filter for CN/SAN fields for invalid dNSNames, accents) are in production.  
Improved cert linting is an ongoing project.

Comment 17

[Ryan Sleevi]

Do you have an update with respect to #2 per comment #13?

Comment 18

[Stephen Davidson]

(In reply to Ryan Sleevi from comment #17)

Thank you for your patience.  There were 1128 certificates affected by this issue; virtually all have been replaced already.  595 have already been revoked, with a further 209 scheduled to be revoked today, 128 before October 25, and 170 before October 31.  26 certificates are problematic (more than the 6 previously reported), requiring software updates to be pushed, but revocation is expected before November 15.

Comment 19

[Markus Wichmann]

Hello all, 

just to confirm, the announced 209 certificates were revoked today, will be visible by EOB on our CRL. 

\Markus

Comment 20

[Stephen Davidson]

(In reply to Ryan Sleevi from comment #17)

As of today, only 123 of the affected certs remain unrevoked or unexpired.  Additional revocations will occur by Monday Oct 30 which will leave only the few problematic certs, which will be revoked before November 15.

Comment 21

[Stephen Davidson]

(In reply to Ryan Sleevi from comment #17)

As of today, only 7 of the affected certs remain unrevoked or unexpired.  These are the last problematic certs - and are also expected to be revoked in the coming weeks.

Comment 22

[Stephen Davidson]

Confirming that all the affected certificates were revoked and that the CA "Siemens Issuing CA Internet Server 2016" was itself revoked today 12/15/2017.  CCADB has been updated accordingly.

Comment 23

[Gervase Markham [:gerv]]

Stephen: can you update us on Siemens' current plans with respect to certificate issuance? Do they intend to continue maintaining an on-prem intermediate, or are they moving to a managed solution? If the former, can you summarise the changes being put in place to ensure a lack of repetition of the various problems I seem to recall they have had?

Gerv

Comment 24

[Stephen Davidson]

Hi Gerv:  I will be visiting and working with the Siemens PKI team in Germany during the week of Jan 22, and will respond following those discussions.  Best regards, Stephen

Comment 25

[Stephen Davidson]

Hello Gerv: Thanks for your patience.
We have been closely engaged with Siemens regarding their trusted external CA.  As noted above, they previously used an internally-developed CA and have now transitioned to EJBCA which should resolve many of the technical BR compliance issues exemplified by the serial entropy issue.  We have been working closely with them on the configuration of the CA.
In addition, we have undergone a detailed discussion with Siemens' experienced PKI team using the Mozilla BR Self Assessment, which has led to further improvements in the CA, the certificate management system, and their policies.  This effort included face-to-face meetings in Germany between Siemens and QuoVadis (including myself) during the week of January 22.  These discussions included expectations regarding the independent audit against ETSI EN 319 411 and resulting opinion letter.
Siemens is logging all SSL issuance to CT, and has implemented improved CAA checking.
At this stage, QuoVadis believes that Siemens is able to continue operating a compliant external CA, and commits to continue working closely with Siemens towards that end.
As noted elsewhere, QuoVadis has an existing monitoring tool for external CAs such as Siemens that focused on Subject fields.  QuoVadis will introduce an improved tool for “live” post issuance scanning of CT logs for all CAs associated with its roots using a zLint based tool in Q2.  In the meantime, QuoVadis undertakes frequent checks using public tools including crt.sh and censys.io.
Although Siemens worldwide is a large and diverse organization, the Siemens PKI team is also undertaking steps to identify groups within its SSL userbase that could benefit from automation of certificate enrollment.
Regards, Stephen

Comment 26

[Wayne Thayer]

Stephen: please explain the recent examples of low serial number entropy identified by ZLint: https://crt.sh/?caid=52410&opt=cablint,zlint,x509lint&minNotBefore=2017-01-01

Also, have all of the actions identified in comment 11 been completed?

Comment 27

[Firefox Bug Husbandry Bot]

Changing QA contact per https://bugzilla.mozilla.org/show_bug.cgi?id=1438254

Comment 28

[Stephen Davidson]

We have been aware of this since Jan 17 via post-issuance/CT linting and engaged immediately with the CA.  We did not report it as it is a zLint Warning rather than an Error.  QuoVadis, the CA, and EJBCA/Primekey had active discussions regarding the Warning (and thank Rob Stradling/crt.sh for his input).

We believe this is a false alarm rather than a systematic issue with the CA.  You may have noticed that other CAs are sporadically throwing the same error:
https://crt.sh/?caid=1348&opt=cablint,zlint,x509lint&minNotBefore=2017-01-01

These certificates – which represent a small proportion of the respective CAs’ issuance -- seem to have a common feature:  the first octet of the binary representation of the serial number consists only of zeros.  According to a related discussion for zLint, the way OpenSSL handles serial numbers may lead to false positives in zLint:
https://github.com/zmap/zlint/issues/187#issuecomment-351720085 

As previously noted, Siemens changed platforms to EJBCA last year to avoid issues with serial number entropy. EJBCA issues 64bit (8byte) serial numbers by default with the BR in mind. To be on the safe side and based on QV’s best practices, Siemens decided to change their SSL CA serial number octet size from 8 to 20, going well beyond the BR.  This change required knockon updates to their (non CA) provisioning systems followed by related testing.  As such, it will be introduced in their production environment by the latest date of Mar 8, 2018.

Regarding the actions in comment 11, all are complete with the exception of pre-issuance linting.

Comment 29

[Wayne Thayer]

(In reply to Stephen Davidson from comment #28)
> We have been aware of this since Jan 17 via post-issuance/CT linting and
> engaged immediately with the CA.  We did not report it as it is a zLint
> Warning rather than an Error.  QuoVadis, the CA, and EJBCA/Primekey had
> active discussions regarding the Warning (and thank Rob Stradling/crt.sh for
> his input).
> 
> We believe this is a false alarm rather than a systematic issue with the CA.
> You may have noticed that other CAs are sporadically throwing the same error:
> https://crt.sh/?caid=1348&opt=cablint,zlint,x509lint&minNotBefore=2017-01-01
> 
> These certificates – which represent a small proportion of the respective
> CAs’ issuance -- seem to have a common feature:  the first octet of the
> binary representation of the serial number consists only of zeros. 
> According to a related discussion for zLint, the way OpenSSL handles serial
> numbers may lead to false positives in zLint:
> https://github.com/zmap/zlint/issues/187#issuecomment-351720085 
> 
Great explanation, thank you Stephen.

> As previously noted, Siemens changed platforms to EJBCA last year to avoid
> issues with serial number entropy. EJBCA issues 64bit (8byte) serial numbers
> by default with the BR in mind. To be on the safe side and based on QV’s
> best practices, Siemens decided to change their SSL CA serial number octet
> size from 8 to 20, going well beyond the BR.  This change required knockon
> updates to their (non CA) provisioning systems followed by related testing. 
> As such, it will be introduced in their production environment by the latest
> date of Mar 8, 2018.
> 
> Regarding the actions in comment 11, all are complete with the exception of
> pre-issuance linting.

Is there a date for pre-issuance linting to be implemented by Siemens?

Comment 30

[Stephen Davidson]

(In reply to Wayne Thayer [:wayne] from comment #29)

> Is there a date for pre-issuance linting to be implemented by Siemens?

Primekey recently added support for EST (RFC 7030) external script validators (facilitating linting)in EJBCA Enterprise 6.11.  We are investigating the feature, but do not have an implementation date. As noted elsewhere, we will implement an enhanced post issuance but live linting tool in Q2.

Comment 31

[Rufus Buschart]

On March 1st Siemens CA changed from 64 bit serial numbers to 160 bit serial numbers. First certificates appear on crt.sh since yesterday eg https://crt.sh/?id=345568939

We will keep you informed about the activation of a linter in EJBCA.

Comment 32

[Stephen Davidson]

Siemens successfully implemented pre-issuance linting of TLS/SSL certificates using zLint over Summer 2018.  Siemens continues to evaluate additional linting tools.