Closed Bug 1873739 Opened 11 months ago Closed 10 months ago

Google Trust Services: uses "DNSSec-mostly" and DTPs for DNS resolution

Categories

(CA Program :: CA Certificate Compliance, task)

Product:
CA Program
Component:
CA Certificate Compliance
Type:
task

Tracking

(Not tracked)

Status:
RESOLVED INVALID

People

(Reporter: notifications, Assigned: gts-external)

Details

(Whiteboard: [ca-compliance] [uncategorized] [external])

Attachments

(2 files)

gts-dns-trace.pcap
6.25 KB, application/vnd.tcpdump.pcap
Details
cert.cer
1.68 KB, application/x-x509-ca-cert
Details
Attached file gts-dns-trace.pcap —

Steps to reproduce:

Hello,

Google Public DNS Recursive servers (also known as "Quad-8") may disable DNSSec temporarily as documented [1]:

If Google Public DNS cannot validate a response (due to misconfiguration, missing or incorrect RRSIG records, etc.), it will return an error response (SERVFAIL) instead. However, if the impact is significant (e.g. a very popular domain is failing validation), we may temporarily disable validation on the zone until the problem is fixed.

As such it must not be used for any task with the goal of domain validation, as previously clarified and understood in the community [2]:

Buypass also cannot be certain whether DNSSEC validation was performed during the lookups, due to this part of Google's DNS service documentation [...]

Google Trust Services issues certificates by using the following recursive DNS validators for getting CAA records of a domain:

  • Google public DNS resolvers

Google Trust Services issues certificates by using the following recursive validators when validating a DNS-01 ACME challenge:

  • Google public DNS resolvers
  • using a Delegated Third Party (DTP): Cloudflare

I'm sure something smart is done like a comparison of the result. But the fact of the matter is that both sources are insufficient: Cloudflare is a DTP and Quad-8 is "DNSSec-mostly" (see above).

I did not verify DNS resolution of HTTP-01 and TLS-ALPN-01 challenges, I'd imagine that the behavior is similar to either CAA lookup (Quad-8 only) or DNS-01 challenge (Quad-8 and Cloudflare).

Evidence:

dns01.domainagain2024.fun was validated as per:

https://crt.sh/?sha256=01F81842F8E53AB2E5B4943CB7422B20971B9D56839F379F072357EC36E05A94

It looks like GTS only submits pre-certificates to the CT's (which is acceptable afaik), so I'm attaching the actual leaf certificate here.

Here's a snipped of the DNS traffic that the nameserver has seen during the validation. The complete and unredacted capture of the DNS traffic is attached as tcpdump capture to this incident.

74.125.43.71.46765 > 95.216.160.90.53: [udp sum ok] 64832% [1au] TXT? _aCme-challEnGe.DNs01.domaiNagAin2024.FuN. ar: . OPT UDPsize=1400 DO (70)
---> identified as Google Public Resolvers as per [3]

172.253.233.5.41619 > 95.216.160.90.53: [udp sum ok] 55110% [1au] TXT? _aCmE-cHaLlenGe.DnS01.DOMaINagaIN2024.fun. ar: . OPT UDPsize=1400 DO [ECS 201.0.229.0/24/0] (81)
---> identified as Google Public Resolvers as per [3]

172.253.5.3.39771 > 95.216.160.90.53: [udp sum ok] 5312% [1au] TXT? _acme-CHalLeNGE.dnS01.DOmaINaGAIn2024.FUN. ar: . OPT UDPsize=1400 DO [ECS 203.113.129.0/24/0] (81)
---> identified as Google Public Resolvers as per [3]

162.158.37.30.39045 > 95.216.160.90.53: [udp sum ok] 13928 [1au] TXT? _acme-challenge.dns01.domainagain2024.fun. ar: . OPT UDPsize=1452 DO (70)
--> Cloudflare is a Delegated Third Party (DTP)

172.68.224.96.39338 > 95.216.160.90.53: [udp sum ok] 27963 [1au] TXT? _acme-challenge.dns01.domainagain2024.fun. ar: . OPT UDPsize=1452 DO (70)
--> Cloudflare is a Delegated Third Party (DTP)

172.68.9.220.56766 > 95.216.160.90.53: [udp sum ok] 28793 [1au] TXT? _acme-challenge.dns01.domainagain2024.fun. ar: . OPT UDPsize=1452 DO (70)
--> Cloudflare is a Delegated Third Party (DTP)

CAA CHECKS:

74.125.45.131.54979 > 95.216.160.90.53: [udp sum ok] 59119% [1au] Type257? dNS01.domAInAGAIN2024.FUN. ar: . OPT UDPsize=1400 DO (54)
---> identified as Google Public Resolvers as per [3]

173.194.95.137.50587 > 95.216.160.90.53: [udp sum ok] 64329% [1au] Type257? doMainAGaIn2024.fUN. ar: . OPT UDPsize=1400 DO (48)
---> identified as Google Public Resolvers as per [3]

I'm attaching the following files:

gts-dns-trace.pcap --> complete DNS traffic at the nameserver
cert.cer --> the issued certificate

[1] https://web.archive.org/web/20240109183556/https://developers.google.com/speed/public-dns/faq#gdns_validation_failure
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=1839305#c13
[3] https://web.archive.org/web/20240109183556/https://developers.google.com/speed/public-dns/faq#locations

Attached file cert.cer —

Here's corroboration from DCV Inspector: https://dcv-inspector.com/test/2a87fdfb8c6aed848fe644d269ab44ef

Does parent company of CA considered DTP for that CA or isn't? CA is Google Trust Services LLC and public resolver is from Google LLC isn't it?
not sure how Google Trust Services Europe Ltd fits in there too:

Hi Lukas,

The validations you and Andrew have posted are from Google Trust Services’ multiple perspective domain validation (MPDV). CloudFlare’s DNS service is used solely as an alternative perspective. Our current MPDV requirements are:

  1. That our primary perspective validates the result.
  2. At least three out of five of our geographically distributed alternate perspectives validate the result.
  3. At least two different DNS resolver implementations validate in order to mitigate bugs or vulnerabilities in any single implementation.

Google Trust Services does not consider Google Public DNS authoritative for DNSSEC checks. We operate our own DNSSEC validator outside the influence of other engineers / teams at Google.

Generally though, DNSSEC checking is not a requirement for CAs under the Baseline Requirements.

  • DNSSEC only appears once in the Baseline Requirements as a stipulation for ignoring CAA failures. GTS never ignores CAA failures under any circumstances.
  • RFC 8659 (CAA) strongly RECOMMENDS using DNSSEC (https://datatracker.ietf.org/doc/html/rfc8659#section-5.1) but it is not a MUST.

DNSSEC is a requirement of the CAA ACME extension, RFC 8657, though CAs do not have to follow it unless they place a requirement on it in their CPS.

We have been following the recent discussions in https://bugzilla.mozilla.org/show_bug.cgi?id=1838421, https://bugzilla.mozilla.org/show_bug.cgi?id=1839305, https://bugzilla.mozilla.org/show_bug.cgi?id=1872371, and the CABForum Working Groups. We are also familiar with the historical discussions in https://bugzilla.mozilla.org/show_bug.cgi?id=1651026 and https://bugzilla.mozilla.org/show_bug.cgi?id=1670337. Our implementation is different from these other cases as the primary validation is via our CA specific validator and other sources are used for multi perspective comparisons.

Happy to answer further questions you may have.

Google Trust Services

Hello,

my goal is to clarify the requirements and make sure all CA's are hold to the same (exact) standards.

Google Trust Services does not consider Google Public DNS authoritative for DNSSEC checks. We operate our own DNSSEC validator outside the influence of other engineers / teams at Google.

So you first make the determination whether a domain is DNSSec enabled or not, and based on that you either use Google Public DNS server for domain validation (when the domain is not DNSSec enabled) or your independent DNSSEC validator. Is that correct? Can you confirm whether this determination is done based on Google Public DNS services or independently?

Can you explain whether you consider Google Public DNS a Delegated Third Party and why, based on the BR definition :

Delegated Third Party: A natural person or Legal Entity that is not the CA but is authorized by the CA, and whose activities are not within the scope of the appropriate CA audits, to assist in the Certificate Management Process by performing or fulfilling one or more of the CA requirements found herein.
Certificate Management Process: Processes, practices, and procedures associated with the use of keys, software, and hardware, by which the CA verifies Certificate Data, issues Certificates, maintains a Repository, and revokes Certificates.

--

Generally though, DNSSEC checking is not a requirement for CAs under the Baseline Requirements.

  • DNSSEC only appears once in the Baseline Requirements as a stipulation for ignoring CAA failures. GTS never ignores CAA failures under any circumstances.
  • RFC 8659 (CAA) strongly RECOMMENDS using DNSSEC (https://datatracker.ietf.org/doc/html/rfc8659#section-5.1) but it is not a MUST.

DNSSEC is a requirement of the CAA ACME extension, RFC 8657, though CAs do not have to follow it unless they place a requirement on it in their CPS.

I'm confused - there appears to be an understanding that complying with BR (and RFCs and it's own CPS of course) is not enough. Additional requirements documented in other incidents in this bugtracker, CCADB ML as well as MDSP ML are said to be non-optional.

Are you saying you disagree with the statement from Antonis that implies that DNSSec checking is a hard requirement (see below)?

https://bugzilla.mozilla.org/show_bug.cgi?id=1839305#c13

Buypass [...] cannot be certain whether DNSSEC validation was performed during the lookups

There are 3 possible outcomes in my opinion:

  • there is consensus that full DNSSec validation for enabled domains is a hard requirement for CAs
  • there is consensus that full DNSSec validation for enabled domains is NOT a hard requirement (however strong recommendations may be)
  • there is no consensus

I'm trying to ascertain which of those possibilities is the case (despite GTS actually do performing DNSSec validation as per your comment).

Thank you,
Lukas

All of the observed DNS queries came from IP addresses associated with either Google Public DNS or Cloudflare. Is Google Trust Services using Google Public DNS for its "primary" perspective? If so, is Google Public DNS operated and audited in accordance with the Baseline Requirements? If not, could you explain why your DNS queries are coming from an IP address also used by Google Public DNS according to https://developers.google.com/speed/public-dns/faq#locations?

Flags: needinfo?(gts-external)

Also, the following statement:

GTS never ignores CAA failures under any circumstances.

is inconsistent with your CPS, which states:

When checking CAA records, a lookup failure is treated as permission to issue if:

  • the failure is outside the CA's infrastructure;
  • the lookup has been retried at least once; and
  • the domain's zone does not have a DNSSEC validation chain to the ICANN root.

Which statement is true?

Type: defect → task
Version: 4.0 → unspecified

We realize validation is at the core of being a CA and we appreciate all of the comments and questions. We are working on a thorough response and will reply no later than EOD Thursday.

GTS, we understand you intend to provide a thorough response to all questions no later than the end of today, and we want to emphasize the following (that was already asked) should be directly addressed:

  1. Whether you rely on Google public DNS as your primary perspective. (understanding that you have implemented MPDV)
  2. If Google public DNS is your primary perspective, is it in scope of your annual audit?
  3. If Google public DNS is not your primary perspective, why has certificate issuance outlined in this report shown DNS queries only from Google public DNS IPs (and Cloudflare)?
  4. Under what conditions do you ignore CAA lookup failures?

Thank you for your patience as we worked to get these questions and answers sorted.

We will start by describing our system with more detail before diving into our interpretation of Google DNS and how it relates to Delegated Third Parties.

Current GTS Validation Design
Currently our full validation begins with our RA creating six separate requests to our Domain Validation Service (DVS) each with a unique requested perspective. The primary perspective reaches out to local Google DNS backends via an internal RPC. These backends do share some infrastructure with Google Public DNS including egress IP addresses. For the alternate perspectives, our code will select five available nodes, each from a different geographical region and assign a popular DoH server. 8.8.8.8, 8.8.4.4 and 1.1.1.1 is the preferred set as seen in the posted logs. 8.8.8.8 and 8.8.4.4 of course both being Google’s servers. Although we ask all six perspectives to perform DNSSEC on their own and we respect failures, we additionally run our DNSSEC validator on the results of every query. Missing records necessary to complete the validation are requested from the associated perspective.

GTS validates the challenges from multiple perspectives using multiple implementations to mitigate bugs, vulnerabilities, or cache poisoning for any particular perspective. We require that at least Google DNS and one other resolver implementation agree to the validation. We guarantee that three different DNSSEC implementations (Google DNS, independent DNSSEC validator, and a third-party) agree to the results for every successful authorization.

Finally, to reduce the risk of attacks like local BGP hijacks [1], we require that our primary perspective as well as three of the five globally distributed perspectives agree. Prior work has demonstrated that MPDV solutions increase the cost of BGP attacks [2, 3] and that it is believed to be a worthwhile investment for the ecosystem [4, 5].

Google DNS and Delegated Third Parties
Google Trust Services does not interpret our use of Google DNS to qualify it as a Delegated Third Party (DTP). Our reading of the Baseline Requirements (BR) definition of a DTP is that it must both be a separate legal entity as well as outside the scope of the audit [6].

CAs are allowed to use third-parties, but the CA must own the process and the third-party must be in scope for the audit. We believe this interpretation is substantiated by the original discussions [7, 8] of ballot 204 [9], which introduced the language. GTS’ usage of Google DNS is in scope of our WebTrust audit. Our usage of Google DNS satisfies the requirements of the BRs and NSRs. A high level overview of some of the controls enforced across this infrastructure can be found in Chapter 14 of the “Building Secure and Reliable Systems'' book [10].

DNSSEC as a Requirement Clarification
We do not believe the linked thread (https://bugzilla.mozilla.org/show_bug.cgi?id=1839305#c13 & https://bugzilla.mozilla.org/show_bug.cgi?id=1839305#c18) establishes that DNSSEC is a requirement. It was pointed out that DNSSEC checks could not be relied upon from Google’s Public DNS and that not checking DNSSEC is discouraged. This follows the RFCs and BRs as we have interpreted them. If others have a different reading of specific text, we would be happy to raise the question up in appropriate channels.

Ignoring CAA Failures Clarification
Regarding the difference comment#7 raised regarding our statement about never ignoring CAA failures vs. the more permissive text in the CPS, the statement that we do not ignore CAA failures is correct. Version 5.5 of our CPS stated the more permissive approach allowed by the BRs: "CAs are permitted to treat a record lookup failure as permission to issue if:". GTS, however, does not exercise those options in our codebase. CPS 5.6 reflects the more stringent approach we take [11].

Ongoing Work
We are happy that the CA/B Forum’s Server Certificate working group is discussing DTP definition and related terms [12]. Clearer definitions and understandings will help avoid similar issues in the future and GTS will, of course, be happy to participate and adapt to any new terminology.

Google Trust Services is focused on continually improving our validation design to be more reliable and resilient to attacks. We appreciate the community’s continued input and suggestions to reach this goal.

[1] https://www.princeton.edu/~pmittal/publications/bgp-tls-usenix18.pdf
[2] https://www.usenix.org/system/files/sec21fall-birge-lee.pdf
[3] https://www.usenix.org/system/files/usenixsecurity23-cimaszewski.pdf
[4] https://lists.cabforum.org/pipermail/validation/2023-March/001875.html
[5] https://github.com/ryancdickson/staging/pull/8
[6] https://github.com/cabforum/servercert/blob/main/docs/BR.md#161-definitions
[7] https://lists.cabforum.org/pipermail/public/2017-March/041884.html
[8] https://lists.cabforum.org/pipermail/public/2017-May/042440.html
[9] https://cabforum.org/2017/07/11/ballot-204-forbid-dtps-domainip-ownership/
[10] https://google.github.io/building-secure-and-reliable-systems/raw/ch14.html
[11] https://pki.goog/repo/cps/5.6/GTS-CPS.html#4-2-4-certificate-authority-authorization-caa-records
[12] https://github.com/cabforum/servercert/pull/475

(In reply to Chris Clements from comment #9)

GTS, we understand you intend to provide a thorough response to all questions no later than the end of today, and we want to emphasize the following (that was already asked) should be directly addressed:

  1. Whether you rely on Google public DNS as your primary perspective. (understanding that you have implemented MPDV)
  2. If Google public DNS is your primary perspective, is it in scope of your annual audit?
  3. If Google public DNS is not your primary perspective, why has certificate issuance outlined in this report shown DNS queries only from Google public DNS IPs (and Cloudflare)?
  4. Under what conditions do you ignore CAA lookup failures?

We hope our response in comment#10 addressed these points satisfactorily, Chris. Nonetheless, we are answering them directly here as well:

  1. Yes. The primary perspective reaches out to local Google DNS backends, which is done via an internal RPC. These backends share some infrastructure with Google Public DNS.
  2. Yes. GTS’ usage of Google DNS is in scope of our WebTrust audit.
  3. Addressed by question #1.
  4. We do not treat failures as permission to issue under any circumstances despite the BRs allowing that. Version 5.6 of our CPS reflects the more stringent approach we take.
Flags: needinfo?(gts-external)

What actually happens if Google public DNS's dnssec-ignore closer activates for examle.com?

  1. yours resolver are different server behind same NAT and don't care about what public server's config
  2. public DNS team will give you list of exception and GTS will add to blacklist
  3. no explict notification but your server would catch it (like nsec for dnskey on tld side)
  4. it would not distiguishable from domain without dnssec

Thank you for your response Google Trust Services!

While your approach to domain validation is thorough, and helps protect against a plethora of attacks, I believe we need to ensure that you are doing so while meeting all the minimum requirements. As I'm sure you'd agree, overperforming in one area should not grant exemptions from requirements in other areas.

With that in mind, I'd like to inquire a little bit more information on the following:

(In reply to Google Trust Services from comment #10)

Google DNS and Delegated Third Parties
Google Trust Services does not interpret our use of Google DNS to qualify it as a Delegated Third Party (DTP). Our reading of the Baseline Requirements (BR) definition of a DTP is that it must both be a separate legal entity as well as outside the scope of the audit [6].

CAs are allowed to use third-parties, but the CA must own the process and the third-party must be in scope for the audit. We believe this interpretation is substantiated by the original discussions [7, 8] of ballot 204 [9], which introduced the language. GTS’ usage of Google DNS is in scope of our WebTrust audit. Our usage of Google DNS satisfies the requirements of the BRs and NSRs. A high level overview of some of the controls enforced across this infrastructure can be found in Chapter 14 of the “Building Secure and Reliable Systems'' book [10].

You state above that Google DNS is not a Delegated Third Party. As you say, the BRs seem to establish two criteria:

  1. Separate Legal Entity

I believe this is met, as "Google Trust Services LLC" and "Google Trust Services Europe Limited" are separate legal entities from "Google LLC". Since Google DNS seems to be offered by "Google LLC", it is a separate legal entity.

  1. [The] "activities are not within the scope of the appropriate CA audits"

In the quote above you claim that Google DNS is fully covered by the Google Trust Services audits.

However, in Comment #11, you state something different:

Yes. GTS’ usage of Google DNS is in scope of our WebTrust audit.

I would therefore like to ask the following question:

Is your usage of Google DNS in the scope of the audit, or is the entire Google DNS in scope of the audit?

My concern is the following:

Does Google Trust Services Auditors simply look at the line in code asking 8.8.8.8 something, or do they look at the code running at 8.8.8.8? Put simply, if Google DNS had no code reviews, no capacity planning, no risk assessment, no NSR compliance, etc. would this GTS audit yield a "success" or a "failure" if the "nslookup" line in your code had all of that?

Thanks

Flags: needinfo?(gts-external)

(In reply to tjtncks from comment #12)

What actually happens if Google public DNS's dnssec-ignore closer activates for examle.com?

  1. yours resolver are different server behind same NAT and don't care about what public server's config
  2. public DNS team will give you list of exception and GTS will add to blacklist
  3. no explict notification but your server would catch it (like nsec for dnskey on tld side)
  4. it would not distiguishable from domain without dnssec

The MPDV validation described in comment#10 would be unaffected by any of Google’s Public DNS infrastructure potentially ignoring DNSSEC. The architecture is closest to your statement in #1. All of the infrastructure is composed of a series of microservices and the service performing the independent DNSSEC validation we referred to is controlled and operated independently of other Google DNS infrastructure.

Happy to answer any more questions.

GTS will respond to comment#13 no later than EOD Tuesday, January 23rd.

(In reply to Google Trust Services from comment #10)

Current GTS Validation Design
Currently our full validation begins with our RA creating six separate requests to our Domain Validation Service (DVS) each with a unique requested perspective. The primary perspective reaches out to local Google DNS backends via an internal RPC.
[...]
Missing records necessary to complete the validation are requested from the associated perspective.

So, if I understand correctly, the primary requests to subscriber DNS servers are done by Google LLC ('s DNS servers), with additional requests for "missing records" and dnssec validation being done by your own servers and services?

How does this work for zones that don't have DNSSEC or signing enabled? Do you just blindly trust Google LLC (and Cloudflare) to not return fabricated or otherwise incorrect results? I believe that's exactly what the "no delegation of DNS" clause exists for to prevent.

Assignee: nobody → gts-external
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [ca-compliance]

(In reply to comment #13)

Thank you for your response. We’ll try to answer your statements and questions in order, below.

(In reply to Google Trust Services from comment #10)

Google DNS and Delegated Third Parties
Google Trust Services does not interpret our use of Google DNS to qualify it as a Delegated Third Party (DTP). Our reading of the Baseline Requirements (BR) definition of a DTP is that it must both be a separate legal entity as well as outside the scope of the audit [6].

CAs are allowed to use third-parties, but the CA must own the process and the third-party must be in scope for the audit. We believe this interpretation is substantiated by the original discussions [7, 8] of ballot 204 [9], which introduced the language. GTS’ usage of Google DNS is in scope of our WebTrust audit. Our usage of Google DNS satisfies the requirements of the BRs and NSRs. A high level overview of some of the controls enforced across this infrastructure can be found in Chapter 14 of the “Building Secure and Reliable Systems'' book [10].

You state above that Google DNS is not a Delegated Third Party. As you say, the BRs seem to establish two criteria:

  1. Separate Legal Entity

I believe this is met, as "Google Trust Services LLC" and "Google Trust Services Europe Limited" are separate legal entities from "Google LLC". Since Google DNS seems to be offered by "Google LLC", it is a separate legal entity.

  1. [The] "activities are not within the scope of the appropriate CA audits"

The DTP definition lays out a three-fold test: (1) The activity is performed by an entity that is not the CA, (2) the activity is not within the scope of the CA’a audit and (3) the activity is a duty of the CA stipulated by the BR.

In the quote above you claim that Google DNS is fully covered by the Google Trust Services audits.

However, in Comment #11, you state something different:

Yes. GTS’ usage of Google DNS is in scope of our WebTrust audit.

In comment #10, we responded that “GTS’ usage of Google DNS is in scope of our WebTrust audit.”

In comment #11, we responded that “Yes. GTS’ usage of Google DNS is in scope of our WebTrust audit.”

I would therefore like to ask the following question:

Is your usage of Google DNS in the scope of the audit, or is the entire Google DNS in scope of the audit?

GTS does not use the entirety of Google DNS. The parts we use are in scope for the audit.

My concern is the following:

Does Google Trust Services Auditors simply look at the line in code asking 8.8.8.8 something, or do they look at the code running at 8.8.8.8? Put simply, if Google DNS had no code reviews, no capacity planning, no risk assessment, no NSR compliance, etc. would this GTS audit yield a "success" or a "failure" if the "nslookup" line in your code had all of that?

We do not want to speak on behalf of our auditors, so we requested a response from them for this question. Their reply was that they "conducted their examination in accordance with attestation standards established by the American Institute of Certified Public Accountants, specifically US – AT-C section 205, Examination Engagements (AICPA, Professional Standards) (“AT-C 205”). This included an assessment of the risks related to Google DNS and our usage of it."

Our auditors performed the procedures they believed appropriate to provide a reasonable basis for their opinion.

We hope this helps answer any questions and clears up any confusion.

(In reply to comment #15)

(In reply to Google Trust Services from comment #10)

Current GTS Validation Design
Currently our full validation begins with our RA creating six separate requests to our Domain Validation Service (DVS) each with a unique requested perspective. The primary perspective reaches out to local Google DNS backends via an internal RPC.
[...]
Missing records necessary to complete the validation are requested from the associated perspective.

So, if I understand correctly, the primary requests to subscriber DNS servers are done by Google LLC ('s DNS servers), with additional requests for "missing records" and dnssec validation being done by your own servers and services?

How does this work for zones that don't have DNSSEC or signing enabled? Do you just blindly trust Google LLC (and Cloudflare) to not return fabricated or otherwise incorrect results? I believe that's exactly what the "no delegation of DNS" clause exists for to prevent.

Thanks for the question. Google Trust Services does not blindly trust Google LLC to resolve DNS. GTS has full view of the source and cryptographic proof of the service versions running at all times. Multi-party Authorization (MPA) is enforced for changes by the infrastructure itself [1]. We have greater assurance of the code running than we would if we downloaded a resolver binary / source and ran it ourselves [2]. Ken Thompson’s “Reflections on Trusting Trust” comes to mind [3].

Controls like MPDV and the independent DNSSEC validator provide additional defense-in-depth. For instance with MPDV, the internal controls over Google DNS would have to fail as well as for a third-party to collude or be successfully attacked.

The “no delegation of DNS clause” you refer to does not apply if the “activities are in scope of the CA audits” as described in comment#10. GTS’ usage of Google DNS is in scope of our WebTrust audit.

[1] https://google.github.io/building-secure-and-reliable-systems/raw/ch14.html
[2] https://cloud.google.com/docs/security/infrastructure/design
[3] https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf

Flags: needinfo?(gts-external)

Hello Google Trust Services,

you make a very good point on comment #17 which inspires confidence on the setup in relation to resolvers and DNSSEC. From what I understand, both Google Trust Services and Google / Cloudflare will do DNSSEC verification most of the time, while GTS will do it all the time. You also validate NSEC and NSEC3 records in this from . down to the FQDN, right?

I would only advise caution, as a 24 hour sample I took from 4 CT logs and some heuristic analysis suggests that almost all Google Trust Services precertificates included are issued to customers of Cloudflare or Google (itself and Cloud). You therefore seem to rely on your two largest Subscribers (I assume) for DNS, with almost all certificates in CT being issued by them.

My question will focus on comment #16 . I don't know if this is the right incident, or if you want to file a new one to examine this separately, but even after your response in that comment I still have no idea what is covered under your WebTrust audit.

The new statement is:

GTS does not use the entirety of Google DNS. The parts we use are in scope for the audit.

I am still not sure if the WebTrust audit covers only Google Trust Services code, processes, organizational control, logging, etc. Do you cover code, processes, procedures, controls, and infrastructure that is not owned or maintained by Google Trust Services? For example code by Google, Inc. or infrastructure by Google, Inc. etc. If yes, are all involved Google, Inc. engineers, management, etc. aware of the BRs, the audits, and everything required to be included? Does EY review and interview for example Google DNS staff, code, procedures, infrastructure, etc. during their audits?

I am trying to understand explicitly which parts are in scope, and which are not. I believe the current response does not address Comment #9 fully.

You brought us a quote from your auditors, but I'm afraid it has no signal in it. It can be distilled down to "Trust us, we did a good job. We say it's fine." -- I'm not saying that it's not, but we need to verify that here. Audits are an integral part of the WebPKI, but our collaborative incident response here is equally important.

I have downloaded your latest audit statements from EY, dated 2023-11-13. In the Management Assertion and in the Report I see the locations audited to be:

New York, USA & South Carolina, USA & Oklahoma, USA & Ghlin, Belgium & Zurich, Switzerland

(I'm sorry, I can't copy the text from the PDF, as it requires a password. I guess it's DRM'd)

However, in the Google DNS documentation, in the samples we have in Comment #2 and the main post, we see locations that don't seem to be (from the metro codes) within these locations.

How is this explained in the context of this incident? If (parts of) Google DNS is included in the WebTrust audit, does GTS Management and Auditors not include its locations in the audit statement? Does GTS use / operate separate Google DNS infrastructure within WebTrust audited spaces? If yes, what happens with the metro areas outside of the list above that appeared in traces?

Thanks!

Flags: needinfo?(gts-external)

(In reply to comment #18)

Thanks for your interest in helping secure the Web PKI. Your questions appear to be partially directed towards auditors. As with any audit, Google Trust Services does not have access to the auditor’s confidential notes or working papers, and it would be inappropriate for us to speak on their behalf. If you feel current audits are not transparent enough, one way to move the ecosystem forward would be to encourage the adoption of long-form audit reports. This discussion would best be had in the appropriate working groups.

We believe there might be a misunderstanding about the audit requirements, however, that are worth pointing out. Under the relevant professional auditing standards, there is no requirement that the auditors fully verify the codebase of dependencies and find bugs. It is neither an expectation nor viable for auditors to review every line of config and code, instead they independently assess the controls implemented by the CA so that they can opine on the CA’s assertion that they have operated their CA based on the WebTrust Services Principles and Criteria [1]. For instance, BIND has more than 500k lines of code and has a very active community [2]. It would not be feasible or effective for an auditor to assess each change or each line of code. The argument could be expanded to include the CA’s chosen operating system which is generally in an even more privileged position. We believe CAs should have as many strong controls as possible, which is why GTS invests in defense-in-depth approaches.

[1] https://www.cpacanada.ca/-/media/site/operational/ms-member-services/docs/webtrust/01618_ms_ssl-baseline-with-network-security_final_aoda-compliant.pdf
[2] https://gitlab.isc.org/isc-projects/bind9/-/commits/main

Flags: needinfo?(gts-external)

Hello Google Trust Services,

perhaps you misunderstood my questions. I did not ask for auditors to review Google DNS' entire codebase and find bugs or security issues.

Here's what I am trying to understand:

You say Google DNS is included in your Webtrust Audit.

If it is properly included, then this is not a valid incident. If it is not properly included, this is a valid incident, and it will require, for example, a revocation of the entire certificate population whose domains were validated using Google DNS.

Due to your inability over the past 3 weeks to provide a clear answer to our questions, as per the requirements you are held up to, we still cannot make a determination on whether your use of Google DNS constitutes a Delegated Third Party case or not.

If you feel like this is a question for your auditors, please point them to this bug and ask them to respond. I couldn't find any contact details in the reports to do that myself.

We are simply trying to understand this one thing to know how to further process this report.

Thanks

Flags: needinfo?(gts-external)

(In reply to comment #20)

GTS stated in comment #10, comment #11, and comment #16 that our use of Google DNS is in scope for our Webtrust audit and therefore does not constitute a Delegated Third Party per the BR definition. We thank everyone for their work in securing the WebkPKI ecosystem and, now that we believe we’ve answered all outstanding questions, respectfully request that this bug be closed out as ‘invalid'.

Flags: needinfo?(gts-external)

Let me make sure I'm understanding this correctly.

DNSSEC Validation

In comment 10, we learned that GTS is in fact doing DNSSEC validation independently of what the resolvers claim. And I'm assuming we've not seen the queries for DNSSEC validation since they're being validated "offline", e.g. as specified in "we additionally run our DNSSEC validator on the results of every query. Missing records necessary to complete the validation are requested from the associated perspective." So from an external perspective, we won't be able to see DNSSEC validation being done.

DTP or Not

Since Google Trust Services isn't using a third party service that, for example, has requests such as this:

ValidateACME_DNS_01_CHALLENGE(domain, expected_value)

And due to the other requirements of being considered a DTP, this doesn't fall within the definition of being a DTP. I'm inclined to agree with this understanding, mainly since the other understanding would open up effectively everything to be considered a DTP. For example, the ISPs that provide the networking to Google Trust Services, and all other CAs, would come under scope of the audit.

From my understanding based on the comments provided, Google is not delegating domain validation to any third party service (including 8.8.8.8). In the one case that it seems like it does delegate that responsibility (dnssec), the validation is actually still happening offline.

My Conclusion

Based on the answers provided here, I do not think that GTS is mis-categorizing its use of 8.8.8.8 not being a DTP. GTS seems to be using these various DNS providers as a transportation of data only, and not a validation for the data. The validation is handled separately, and independently outside of the control of the third parties.

This is different from the buypass incident we saw earlier, since an offline validation of dnssec was not being performed in their case, and only a single perspective (8.8.8.8) was being used for getting the data. That puts a significantly larger amount of the validation responsibility on the third party service.

(Tiny bit more rambling here)

DNS is an unencrypted protocol. The only ways I can see risks from that being mitigated is:

  • DNSSEC validation
  • Introducing delays and "waiting periods" for DNS (not really feasible in the case of domain validation for certificate issuance)
  • Ensuring stale DNS responses are not being returned

I do not believe running authoritative DNS queries actually falls under any of the risk-mitigation factors mentioned here, mainly due to the fact that without DNSSEC validation false responses can be injected through ISPs down the path without the CA knowing. And even with DNSSEC, replay attacks (or just outright stale responses) are still possible. The replay attacks are not a problem for DNS-01 challenges, but are definitely a problem for CAA record validation. Multiple perspectives in DNS validation can help mitigate the stale response/replay attack problem.

Anyway, rambling aside, I do not think a CA gains much out of running authoritative DNS lookups if they're not also doing multiple perspectives and DNSSEC validation. The data there can't really be fully trusted.

The feedback I have for various root programs, and the CA/B forum is that, we can reduce these entire class of problems going into the future by:

  1. Making DNSSEC validation a MUST and not a SHOULD/MAY.
  2. Requiring multiple perspective for all methods of domain validation to reduce the impact a single ISP, provider, etc can have on domain validation.

GTS or any other CA can also volunteer to bring these up in a future CA/B meeting.

(In reply to amir from comment #23)

(Tiny bit more rambling here)

DNS is an unencrypted protocol. The only ways I can see risks from that being mitigated is:

  • DNSSEC validation
  • Introducing delays and "waiting periods" for DNS (not really feasible in the case of domain validation for certificate issuance)
  • Ensuring stale DNS responses are not being returned

I do not believe running authoritative DNS queries actually falls under any of the risk-mitigation factors mentioned here, mainly due to the fact that without DNSSEC validation false responses can be injected through ISPs down the path without the CA knowing.

To detect striped doesn't DNSSEC answer doesn't validator have to climb DNS tree for NSEC/DS record? Not really requires authoritative resolver but Validator need to climb the DNS tree anyway(keychain for signed zone or nsec record for unsinged zone)

currently dvc-inspector doesn't check this because it doesn't record request for main zone so someone need to craft some setting to check this though

Whiteboard: [ca-compliance] → [ca-compliance] [uncategorized] [external]

GTS continues to monitor this bug for further comments and questions. If there are none, however, we kindly request that this be closed and set to the invalid state.

Flags: needinfo?(bwilson)

If there is no further discussion, then I intend to close this case as Invalid on Friday, 9-Feb-2024. The rationale is that DNSSEC is not a requirement of the CA/Browser Forum (CABF)'s TLS Baseline Requirements (TLS BRs), and the main subject of this bug was whether Google DNS was an unaudited Delegated Third Party (DTP) of GTS. While there are some gray areas in applying prohibitions on the use of DTPs for domain validation, which need to be clarified by the CABF, GTS explained that the observed DNS behaviors were attributable to its implementation of Multiple Perspective Domain Validation and that its domain validation practices were sufficiently within the scope of GTS' WebTrust audit. So, under these circumstances, I don't believe that GTS' use of Google DNS results in an issue of non-compliance with the CABF's TLS BRs.

(In reply to Google Trust Services from comment #17)

(In reply to comment #15)

(In reply to Google Trust Services from comment #10)

Current GTS Validation Design
Currently our full validation begins with our RA creating six separate requests to our Domain Validation Service (DVS) each with a unique requested perspective. The primary perspective reaches out to local Google DNS backends via an internal RPC.
[...]
Missing records necessary to complete the validation are requested from the associated perspective.

So, if I understand correctly, the primary requests to subscriber DNS servers are done by Google LLC ('s DNS servers), with additional requests for "missing records" and dnssec validation being done by your own servers and services?

How does this work for zones that don't have DNSSEC or signing enabled? Do you just blindly trust Google LLC (and Cloudflare) to not return fabricated or otherwise incorrect results? I believe that's exactly what the "no delegation of DNS" clause exists for to prevent.

Thanks for the question. Google Trust Services does not blindly trust Google LLC to resolve DNS. GTS has full view of the source and cryptographic proof of the service versions running at all times. Multi-party Authorization (MPA) is enforced for changes by the infrastructure itself [1]. We have greater assurance of the code running than we would if we downloaded a resolver binary / source and ran it ourselves [2]. Ken Thompson’s “Reflections on Trusting Trust” comes to mind [3].

Does this mean that GTS can either:

  • at any point completely stop using Google LLC's DNS resolver, or
  • block the deployment of unwanted versions of the resolver and force the deployment of a fixed version of these resolvers,

if the Google LLC DNS resolvers are found to be problematic when used for BR validation purposes by GTS?

The provided answer does seem to imply this, but not obviously so, and my reading of this could be wrong.

Flags: needinfo?(gts-external)

(In reply to comment #27)

GTS has full control over how we resolve DNS and, if needed, could completely stop using Google LLC’s DNS resolver.

Flags: needinfo?(gts-external)
Status: ASSIGNED → RESOLVED
Closed: 10 months ago
Flags: needinfo?(bwilson)
Resolution: --- → INVALID
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: