Closed Bug 1475115 Opened Last year Closed 7 months ago

Telia: Qualified Audit Statements

Categories

(NSS :: CA Certificate Compliance, task)

task
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: wayne, Assigned: pekka.lahtiharju)

Details

(Whiteboard: [ca-compliance])

Telia has supplied the following 2018 audit statements:

https://support.trust.telia.com/download/CA/TeliaWebTrustForCAReport2018.pdf
https://support.trust.telia.com/download/CA/TeliaSSLBaselineReport2018.pdf
https://support.trust.telia.com/download/CA/TeliaEVWebTrustReport2018.pdf

The WebTrust for CA report contains one qualification and the BR report contains 9 qualifications. Telia also provided a response:
https://repository.trust.teliasonera.com/TeliaAuditPublicResponse2018.pdf

Items 4 and 5 document misissuances. Please provide an incident report for these, as described here:
https://wiki.mozilla.org/CA/Responding_To_A_Misissuance#Incident_Report
The incident report should be posted to the mozilla.dev.security.policy forum and added to this bug.

Telia's response states that they will take specific steps to remediate issues 1,6,7, and 9. Please also update this bug when each of those steps has been completed.
Incident reports are now in mozilla.dev.security.policy forum

https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/34gZxUX3EBc
https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/5K7LZfIveNQ

Please help guiding the discussion that I wanted to start in the latter one.

Progress in issues 1,6,7,9 
1 - Incomplete gateway CPS
Ongoing: In July Telia has stopped using this CPS and related certificates. Those are replaced by Telia DV certificates documented in the other Telia CPS. We'll create one last update to gateway CPS in August where we fix the listed problem and state that no new certificates are created.
6 - new contract with Delegated Third Party
Preparations done, will signed after vacations
7 - improvement in weekly security review
Preparations done, will start after vacations
9 - Missing documentation related to vulnerability correction process 
Fixed, the new required documentation has been done.
Inera submitted qualified audit statements in bug 1481141 which reference Telia as "an  independent  service organization that provides Certification Authority (CA) hosting services for Inera" and include Telia's CPS in their scope.
Correct, Telia has provided technical CA services for Inera and thus the Inera audit report includes some same deviations that were listed on Telia CA Audit. Telia originated deviations will be fixed at the same time when Telia fix its own deviations.
Status update to Telia audit issues 1,6,7,9 
1 - Incomplete gateway CPS
Closed in August: In July Telia has stopped using this CPS and new certificates. Those are replaced by Telia DV certificates documented in the other Telia CPS. Telia has done the final update to this CPS stating that no more used and CAA chapter 4.2.4 was added. Final CPS is here: https://repository.trust.teliasonera.com/Telia_Gateway_Certificate_CPS_v1_6.pdf (valid 30th Aug 2018).
6 - new contract with Delegated Third Party
Ongoing: Preparations done, will signed probably in September
7 - improvement in weekly security review
Ongoing - mostly done; one weekly review related to Firewall system changes is still to be created
9 - Missing documentation related to vulnerability correction process 
Closed in July: the new required documentation has been done.
Thank you for the update.

Given the number and severity of these audit issues, I am going to recommend that Mozilla require Telia to take two additional corrective actions:
1. Obtain point-in-time WebTrust for CAs and BR audits once all remediation is completed to confirm that the problems are fixed
2. Begin performing period-of-time audits on a 6-month cycle to ensure that additional problems are identified, reported, and fixed more quickly.

I will soon post a message to mozilla.dev.security.policy with these recommendations.
1. We have already internally committed to re-audit all areas of issues to ensure that all current issues are solved.  

2. We’d like to propose doing extra audits by using so called AUP model (Audit Upon Procedures) until addressed issues are solved. This would allow us to focus more on targeted areas.  We could define the scope together.

We can arrange a meeting together with you and auditor to discuss this more.
(In reply to pekka.lahtiharju from comment #6)
> 1. We have already internally committed to re-audit all areas of issues to
> ensure that all current issues are solved.  
> 
That is good to hear. Do you plan to have your auditor perform point-in-time WebTrust for CAs and Baseline Requirements audits and to provide the reports to Mozilla?

> 2. We’d like to propose doing extra audits by using so called AUP model
> (Audit Upon Procedures) until addressed issues are solved. This would allow
> us to focus more on targeted areas.  We could define the scope together.
> 
I have two concerns with AUP audits:
1. My understanding of AUP audits is that the reports are confidential - they may only be shared with parties that sign a contract to receive the report. That is not acceptable to Mozilla because everything we do must be shared with our community. Will you please ask your auditor if the proposed AUP reports can be made publicly available?
2. Once the findings from the latest audits have been remediated and a point-in-time audit report has been completed confirming that fact, then I would not want additional AUP reports to focus only on the areas that have been remediated. What would you propose as the scope of the periodic AUP audits?
> We can arrange a meeting together with you and auditor to discuss this more.
I would like to have answers to the questions above before considering a meeting with your auditors.
Flags: needinfo?(pekka.lahtiharju)
We confirmed that you are probably right that AUP results can't be public. We’ll provide a new proposal to this bug early next week. In that proposal we'll address concerns you have raised. Next we need to have more preliminary negotiations with our auditors.
Flags: needinfo?(pekka.lahtiharju)
(In reply to Wayne Thayer [:wayne] from comment #7)
> (In reply to pekka.lahtiharju from comment #6)
> > 1. We have already internally committed to re-audit all areas of issues to
> > ensure that all current issues are solved.  
> > 
> That is good to hear. Do you plan to have your auditor perform point-in-time
> WebTrust for CAs and Baseline Requirements audits and to provide the reports
> to Mozilla?

This is exactly what we have planned. Now we have agreed with auditor that audit will start in October and results are available in December 2018.

> 
> > 2. We’d like to propose doing extra audits by using so called AUP model
> > (Audit Upon Procedures) until addressed issues are solved. This would allow
> > us to focus more on targeted areas.  We could define the scope together.
> > 
> I have two concerns with AUP audits:
> 1. My understanding of AUP audits is that the reports are confidential -
> they may only be shared with parties that sign a contract to receive the
> report. That is not acceptable to Mozilla because everything we do must be
> shared with our community. Will you please ask your auditor if the proposed
> AUP reports can be made publicly available?
> 2. Once the findings from the latest audits have been remediated and a
> point-in-time audit report has been completed confirming that fact, then I
> would not want additional AUP reports to focus only on the areas that have
> been remediated. What would you propose as the scope of the periodic AUP
> audits?
> > We can arrange a meeting together with you and auditor to discuss this more.
> I would like to have answers to the questions above before considering a
> meeting with your auditors.

Soon after the results of the extra Point-In-Time-audit the next Period-Of-Time audit is about to begin. It will be our next and wider audit 02-05/2019. Its results are available in June 2019. If both these audits give acceptable results we'd like to move back to normal 12-month cycle.
(In reply to pekka.lahtiharju from comment #9)

> > > 2. We’d like to propose doing extra audits by using so called AUP model
> > > (Audit Upon Procedures) until addressed issues are solved. This would allow
> > > us to focus more on targeted areas.  We could define the scope together.
> > > 
> > I have two concerns with AUP audits:
> > 1. My understanding of AUP audits is that the reports are confidential -
> > they may only be shared with parties that sign a contract to receive the
> > report. That is not acceptable to Mozilla because everything we do must be
> > shared with our community. Will you please ask your auditor if the proposed
> > AUP reports can be made publicly available?
> > 2. Once the findings from the latest audits have been remediated and a
> > point-in-time audit report has been completed confirming that fact, then I
> > would not want additional AUP reports to focus only on the areas that have
> > been remediated. What would you propose as the scope of the periodic AUP
> > audits?
> > > We can arrange a meeting together with you and auditor to discuss this more.
> > I would like to have answers to the questions above before considering a
> > meeting with your auditors.
> 
> Soon after the results of the extra Point-In-Time-audit the next
> Period-Of-Time audit is about to begin. It will be our next and wider audit
> 02-05/2019. Its results are available in June 2019. If both these audits
> give acceptable results we'd like to move back to normal 12-month cycle.

Pekka: I am not certain that I understand this response. Will you please provide more detail?

Here is my position:
Mozilla policy requires an unbroken sequence of period-of-time audits. Regardless of the point-in-time audit that you have planned for October, the next period-of-time audit will need to begin on April 1, 2018. I would like this audit to be performed over the 6-month period April 1, 2018 to September 30, 2018. The reason for also performing the point-in-time audit is that I expect this period-of-time audit to be qualified (I think that some problems identified on the last audit were not corrected before the start of the current audit period), but the point-in-time audit should show that all qualifications have been fixed in October. After October, the next period-of-time audit would cover 6-months from October 1, 2018 to March 30, 2019. Once Mozilla receives the unqualified audit for the period ending March 31, 2019, we will decide if we are satisfied that Telia can return to annual audits.
Flags: needinfo?(pekka.lahtiharju)
We are committed to do all necessary fixes to our processes. But we are concerned about the extra efforts and costs related to each extra audit. Each periodical audit includes lots of irrelevant steps like physical site visits etc. That's why we originally planned to do only point-in-time audit (or AUP) this Autumn. Point-in-time audit will tell if all found issues are properly solved and if new certificates included in its Autumn scope are unqualified. I think that we could anyway see if the issue in the next periodical audit is originated because of (not yet fixed) issues in the last qualified audit or if it is a new issue. But if both Autumn audits are required by you then we are ready to do those. Auditor resourcing to do both audits may be an issue also; we have to verify that they are capable to do both.
Flags: needinfo?(pekka.lahtiharju)
Mozilla will accept the following as remediation for the audit issued documented in this bug:

1. Point-in-time WebTrust for CAs and WebTrust BR with Network Security audits performed no later than 31-October, 2018. These audit reports must include all of the root and intermediate certificates that were in the scope of the prior period's qualified audits, they must be delivered to Mozilla within 90 days of the audit date, and they must conform to the other requirements of section 3 of Mozilla policy. In addition, the Management Assertions must include statements indicating that each issue identified as a qualification in the prior period's audit statements has been fixed. If any qualifications or "other matters" are noted indicating that either report is not "clean", Mozilla reserves the right to require Telia to perform additional audits.
2. Telia's next period-of-time audit must be conducted no later than March 31, 2019 covering a period beginning on April 1, 2018. In other words, if no additional problems are discovered, Telia may remain on its existing period-of-time audit cycle.

Pekka: Please confirm that you understand and agree to these conditions.
Flags: needinfo?(pekka.lahtiharju)
We can understand and agree to these conditions. Current status of each issue is below. Please, give your comments if our fixing plan (especially regarding 3&5) is acceptable for Mozilla. Auditor opinion will be given to public after the point-in-time audit.

1 - Gateway CPS missing CAA chapter
Fixed in August. Also usage of this CPS has stopped in August.
2 - Incomplete key generation script documentation
Fixed in July. CA keys created with incomplete documentation were re-created in August.
3 - Incomplete root certificate fields
We can't stop using old roots within this timeframe. Qualification 3 stated that our current Root certificates (created in 2002 and 2007) have details that were later defined to be invalid. Our excuse is that in 2002 and 2007 when these Root CA certificates were created there weren’t any clear requirements for noted issues. The first CA Browser Forum BR document appeared in 2011 (effective 2012). Telia has a plan to create a new root CA in Oct 2018 that will be compliant with all current requirements, but we can't start using it until it is widely accepted by all browsers. We need Mozilla to keep our old roots valid until the new root is widely accepted. We also hope that Mozilla will accept our new root CA application swiftly.
4 - OV OID used in DV certificates
Fixed in June 2018 (OID fixed). Most invalid certificates are already revoked. The rest are replaced&revoked in organized way in September 2018 to prevent server access failures. Delay was because we evaluated the issue not to cause harm to anybody and we wanted to finish issue 2 (create our DV CA again with a proper script/documentation) and to prevent customer problems related to revocation. Public discussion of this incident was put here: https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/34gZxUX3EBc. We have improved our processes accordingly to prevent similar errors in the future.
5 - Weak E value verification
Fixed in July 2018 (stop using E). We decided not to revoke thousands of old E values. Our conclution of discussions in public Mozilla forum (https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/5K7LZfIveNQ) was that "best practice" for E would be to use some strong verification method but it is not illegal to use weaker verification methods like we had done. Our weak method was described in our old CPS (current CPS states that we don't use E any more). BR only requires, regarding E, that CPS must be followed and some verification must be used. Revocation of thousands of certificates would harm our PKI community a lot; nobody would get any benefit from large E/OU revocation.
6 - Incomplete 3dr Party contract
Still ongoing. Will be solved in September by moving this personnel to our direct control.
7 - Incomplete weekly security review
Fixed in August. One last support system review was added to regular routines in September.
8 - Incomplete monthly log review
Fixed in August. 
9 - Incomplete documentation related to Vulnerability correction process
Fixed in July.
Flags: needinfo?(pekka.lahtiharju)
(In reply to pekka.lahtiharju from comment #13)
> We can understand and agree to these conditions. Current status of each
> issue is below. Please, give your comments if our fixing plan (especially
> regarding 3&5) is acceptable for Mozilla. Auditor opinion will be given to
> public after the point-in-time audit.
> 
I will post this information to the mozilla.dev.security.policy forum so that the Mozilla community has the opportunity to comment on your plan.
Pekka: when can we expect to receive the point-in-time audit statements?
Flags: needinfo?(pekka.lahtiharju)
PIT audit is basically done now. PIT date was 31st Oct 2018. Auditors promised that the report will be given before the end of January 2019. I'll inform here when we have got the report.
Flags: needinfo?(pekka.lahtiharju)
Whiteboard: [ca-compliance] → [ca-compliance] Next Update - 1-February 2019
Status: NEW → ASSIGNED

Now the required extra Telia PIT Audit statements are ready and we have added those to our public site here:

https://support.trust.telia.com/download/CA/Telia-WebTrust-for-CA-Report-2018-10-31.pdf
https://support.trust.telia.com/download/CA/Telia-SSL-Baseline-Requirements-Report-2018-10-31.pdf

In statements there is still one open issue which was beforehand expected: Incorrect Root CA certificate format. Our excuse for it is still this:

In 2002 and 2007 when these Root CA certificates were created there weren’t any clear requirements for noted issues. The first CA Browser Forum BR document appeared in 2011 (effective 2012). Telia has already created a new root CA in late 2018 that is compliant with all requirements at this time.

Our next Period-of-time audit is soon starting in normal schedule. Results from it are expected at the end of June 2019.

Now we expect to get a permission from Mozilla to start a process to add our new Root CA to Mozilla to replace the old ones. We plan to apply SMIME, SSL and EV rights for it. We need to keep our old roots active until the new Root is widely accepted. We expect this to take years. Are you now satisfied and is it OK from your point of view for us to craate now an application for a new Root?

I have reviewed the point-in-time audit statements and confirmed that they do not list any existing or new qualifications other than the error in their existing root noted in comment #13.

This concludes the remediation of this issue.

Pekka: regarding submission of your new root to Mozilla, you may proceed with the submission if Telia believes that it is now fully compliant with Mozilla's requirements. Given the number of recent issues that have been reported, the Mozilla community will be looking closely at a new application from Telia, and it would be advisable for Telia to take as much time as necessary to be confident that no issues will be discovered with the new application.

Status: ASSIGNED → RESOLVED
Closed: 7 months ago
Resolution: --- → FIXED
Whiteboard: [ca-compliance] Next Update - 1-February 2019 → [ca-compliance]
You need to log in before you can comment on or make changes to this bug.