1663213 - DoH used even with NXDOMAIN response to canary domain

Status: RESOLVED FIXED

(Core :: Networking: DNS, defect, P2)

Description

[Thayne]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0

Steps to reproduce:

I configured my DNS server to return NXDOMAIN for use-application-dns.net, as described athttps://support.mozilla.org/en-US/kb/canary-domain-use-application-dnsnet

I DID NOT manually enable Application DNS/DoH (athough really, there shouldn't be a difference between automatically enabling DNS/DoH and manually enabling it, or at least there should be an option to "use application DNS unless the canary domain indicates I shouldn't")

Actual results:

Firefox used application DNS with cloudflare instead of my custom (company) DNS server, resulting in incorrect DNS responses.

Expected results:

Firefox should use my configured DNS server rather than Cloudflare's DNS servers.

Comment 1

[Valentin Gosu [:valentin]]

Thank you for your report.

It's not clear as to why this would happen. Could you please attach the output you get when you navigate to about:support ? I'm curious about the value of the network.trr.mode pref. You can also get it by going to about:config and looking for that pref.

Also take a look at about:telemetry#events-tab There should be a bunch of doh-rollout events there. I'm curious as to what they say.

Thanks!

Comment 2

[Thayne]

my network.trr.mode is currently 5, but I have since manually disabled DoH, because it was preventing me from connecting to internal sites. I could try to reproduce, however I'm not sure how to get back into the state I was in.

I also don't see any doh-rollout events in my `about:telemetry#events-tab. Possibly since I've restarted my computer since I disabled DoH?

Comment 3

[Valentin Gosu [:valentin]]

If you've made any changes to any TRR settings or prefs then the doh-rollout heuristics will be disabled (they assume you know if you want to use DoH or not). In that case the canary domain will not be queried.

Comment 4

[Thayne]

I understand that. I had not made any changes when I encountered this bug. But have since manually disabled DoH, because it was causing me problems. But since I have made a change, I don't know how to go back to the heuristics mode, and therefore can no longer reproduce.

This is the order things happened in:

1. I got a new laptop and installed firefox
2. I got the notification about DoH and I selected "Got it", thinking this would keep me in heuristics mode. I also partly chose this, because I know this is most likely what other employees will select, and I have had complaints from other employees that use Firefox, that DoH was preventing them from accessing internal sites.
3. I noticed that the canary domain was not being respected.
4. I manually disabled DoH, because I need to get work done.

I am not the only one at my company that has run into this problem.

If you can provide instructions on how to reset the doh-rollout so I am in heuristics mode again, I would be happy to try to reproduce and provide more information.

Comment 5

[Valentin Gosu [:valentin]]

Thanks for the clarification.
The easiest way would be to create a new profile and try with that one. [1]
Once you do that, and you confirm that this is reproducible, restart Firefox and try to gather some logs using the instructions at [2]

[1] https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Multiple_profiles
[2] https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging

Thanks!

Comment 6

[Thayne]

Hmm, I can't reproduce with a new profile. I wonder if it has something to do with the fact that the profile I saw this with was synced with my firefox account? Are about:config settings or doh rollout settings synced between browsers at all?

Comment 7

[Valentin Gosu [:valentin]]

To my knowledge doh-rollout prefs are not synced. Nihanth, can you confirm?

Comment 8

[(inactive) Nihanth Subramanya [:nhnt11]]

(In reply to Valentin Gosu [:valentin] (he/him) from comment #7)

To my knowledge doh-rollout prefs are not synced. Nihanth, can you confirm?

That's correct! DoH prefs are not included in Sync. You can see what prefs are included by filtering about:config for services.sync.prefs.sync. (each sync'd pref has a corresponding entry that starts with that prefix, IIUC).

Comment 9

[Kershaw Chang [:kershaw]]

Can you still reproduce this? If yes, it'd be great if you can help to get some logs.
Thanks.

Comment 10

[Thayne]

Unfortunately, I haven't been able to reproduce this anymore. It's possible I did something to get me into that state, but I don't know what.

Another possibility is that I may have temporarily entered a state where I was using DNS servers that didn't respond with NXDOMAIN for the canary domain. Like if I was temporarily disconnected from my VPN. If the heuristics determined that it should use DoH, would that ever persist even if the DNS servers change?

Comment 11

[Valentin Gosu [:valentin]]

Thanks for the info. Please keep an eye out for this in case it happens again.
Cheers!

Comment 12

[Thayne]

This may still be an issue. Several employees at my company are reporting issues with DNS not working as expected on our VPN when using Firefox, and I suspect DoH is to blame. I will see if I can gather more information.

Comment 13

[Thayne]

There has been at least one other employee who ran into this problem when he initially was not on the VPN (thus not getting NXDOMAIN responses for the canary domain) and then continued to get incorrect dns responses after he connected the VPN (and was thus getting NXDOMAIN response for the canary domain).

Comment 14

[Kershaw Chang [:kershaw]]

Valentin, could you take a look?

Comment 15

[Valentin Gosu [:valentin]]

(In reply to Thayne from comment #12)

This may still be an issue. Several employees at my company are reporting issues with DNS not working as expected on our VPN when using Firefox, and I suspect DoH is to blame. I will see if I can gather more information.

Ah, I see. So what I suspect is going on is the following:

1. User resolves names using DoH
2. User connects to VPN
3. Network change clears regular Do53 cache, but not DoH cache
4. DNS resolution finds entry in the cache and returns bad result.

It's not clear if this is the exact use case encountered by the reporter, but possibly related. Without logs to confirm I'll assume this is the problem and try to find a fix.

Comment 16

[Valentin Gosu [:valentin]]

Attachment: Bug 1663213 - Clear DoH cache when the TRRService mode goes from ON to OFF r=#necko

This is necessary in order to avoid using cached entries on a network where
they might not be valid (such as when using a VPN).

This might not fix things when turning on the VPN does not cause a network
change event and thus no network.trr.mode/doh-rollout.mode change actually
happens.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/223338b5183e
Clear DoH cache when the TRRService mode goes from ON to OFF r=necko-reviewers,kershaw

Comment 18

[Narcis Beleuzu [:NarcisB]]

Backed out for xpcshell failure on test_trr.js

Backout link: https://hg.mozilla.org/integration/autoland/rev/ab82da735f6c6cecabf08d7eef7388140a454763
Log link: https://treeherder.mozilla.org/logviewer?job_id=322863673&repo=autoland&lineNumber=4169

Comment 19

[Valentin Gosu [:valentin]]

I fixed the failure on the socket process run by also syncing doh-rollout.* prefs into the socket process - which is something we should have been doing all along.

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3d360356c59f
Clear DoH cache when the TRRService mode goes from ON to OFF r=necko-reviewers,kershaw

Comment 21

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/3d360356c59f