1837717 - Intermittent failure to load pages when DNS over HTTPS set to Increased or Max Protection

Status: RESOLVED INCOMPLETE

(Core :: Networking: DNS, defect)

Description

[re+github+er]

Attachment: troubleshooting info.json

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0

Steps to reproduce:

To reproduce:

1. Create a new profile.
2. In DNS of HTTPS settings (about:preferences#privacy) set "Enable secure DNS using:" to "Increased Protection" and leave the provider set as "Cloudfare (default)".
3. Perform a search from the address bar (e.g. "@ddg best extensions for firefox" which opens "https://duckduckgo.com/?t=ffab&q=best+extensions+for+firefox&ia=web").
4. Successively open the links returned by this search in order in new tabs by Ctrl left clicking on each link in the search results page in turn.
5. Repeat opening the links until a page in a newly opened tab is blank indicating that it has failed to load.

Notes:

1. If the tab with the page failing to load is refreshed or the link is opened again with a Ctrl left click from the search results page then the page will then load.
2. The problem does not occur if "Enable secure DNS using:" is set to "Default Protection".
3. The problem also occurs if "Enable secure DNS using:" is set to "Max Protection".
4. The problem also occurs but less frequently if the provider is set to "NextDNS".
5. The problem also occurs and more frequently if the provider is set to Quad9 ("https://dns.quad9.net/dns-query").
6. In other testing with the "uBlock Origin" plugin enabled the tabs with pages failing to load will be called "about.blank-scheme" by the plugin.
7. In other testing with the "uBlock Origin" plugin enabled where a new independent tab is created and then a URL manually entered in the address bar, then the tabs with pages failing to load will be called "newtab.about-scheme" by the plugin.
8. The problem was first seen to occur while running Firefox 113.0.1 (2023511191846) in late 2023-05, possibly after DNS over HTTPS was set to Quad9 but unfortunately the connection with DNS over HTTPS was not realised, so this was not confirmed.
9. It wasn't until Firefox 114.0.1 was installed that the connection with DNS over HTTPS was identified and testing in a new profile was performed. It also happens that Firefox 114.0 moved the DNS over HTTPS settings ("DNS over HTTPS settings are now part of the Privacy & Security section of the Settings page and allow the user to choose from all the supported modes" from https://www.mozilla.org/en-US/firefox/114.0/releasenotes/) but the problem appears to have been happening before this change.

Actual results:

A new tab fails to load its page about once in every 5 to 50 times a new tab is opened.

Expected results:

All new tabs should load their pages.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::DOM: Security' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[Daniel Veditz [:dveditz]]

I think this is working as intended. When the DNS server doesn't respond or returns an error, in the default mode we fallback to local DNS. The page loads fine as far as the user can tell, but there's no way to know when you are or aren't getting DOH protection. If you set it to increased protection then we won't load content using insecure DNS, so when secure DOH requests fail you won't get your content when the remote DOH server is busy or rate-limits you.

Comment 3

[Daniel Veditz [:dveditz]]

In Firefox nightly I get an error page when this happens rather than a blank tab, saying we couldn't load the DNS securely and offering options to try again or to set an exception and go ahead and try to load the DNS insecurely. I had to enable a hidden pref to expose this UI which means it's still a work in progress, but I believe it's intended to address your concerns or at least explain what's going on better than nothing. We can't do anything about the reliability of your chosen DOH server.

I assuming you're loading these links from different sites, because once we get the DNS information for a site we do cache it for a while and subsequent loads shouldn't have a problem. It's possible the DOH servers use a very short TTL, but that would be a bad idea on their part if they're already having trouble keeping up with requests such that they fail 1 in 50 times, let alone 1 in 5.

Comment 4

[re+github+er]

The most significant issue in Firefox 114 is that only a blank page rather than a "Server not found" or similar error page is shown when the remote DNS fails (if Secure DNS is set to Max) or both the remote and local DNS fail (if Secure DNS is set to Increased). From your testing this is not the case in the current Firefox Nightly build so apparently this issue has already been identified and resolved.

The remaining issue in Firefox 114 is the low reliability of both the Secure DNS Increased and Max Protection options using both of the pre-set DNS providers (the default Cloudfare and the non-default NextDNS). From memory I believe that my original testing was performed with a VPN active (NordVPN) and so on reflection I can see how it's possible that the remote DNS lookups were being rate limited due to my IP address being shared with other VPN users using the same VPN server.

In light of this I just reperformed my testing now using Max Protection Secure DNS with the default Cloudflare DNS provider and I found:

• Without any VPN active: 0 pages failed to load after loading pages from 50 different sites
• With a VPN active (NordVPN): 7 pages failed to load after loading pages from 50 different sites
  So it seems that without a VPN active the intermittent page load failure problem is not present (at least using Max Protection Secure DNS).

As my testing suggests if using Increased and Max Protection Secure DNS with a VPN active can decrease DNS reliability significantly then it would be sensible to amend the descriptions of both of these Secure DNS options in the Privacy & Security Settings (about:preferences#privacy) as well as the relevant online documentation (https://support.mozilla.org/en-US/kb/dns-over-https) to make it clear that using Increased and Max Protection Secure DNS options with a VPN active may cause DNS to become unreliable. Especially since the Default Protection Secure DNS option states that Firefox may "Turn off when VPN, parental control, or enterprise policies are active" so the potential incompatibility of using Secure DNS with a VPN has already been recognised by Firefox developers.

Comment 5

[Valentin Gosu [:valentin] (he/him)]

Hi,

Thank you for this report.
There were a couple of VPN related changes that could affect this behaviour: bug 1647985 and bug 1825538.
In order to confirm the bug here I will ask you to do 2 things:

1. Capture some HTTP logs when reproducing this bug: https://firefox-source-docs.mozilla.org/networking/http/logging.html
   • this will allow us to see what the answer from the DoH server is and what is actually failing.
2. You can help us identify the exact change that caused this issue by using mozregression: https://mozilla.github.io/mozregression/quickstart.html
   • this means reproducing the bug around 10 times using different builds of Firefox until we find the one that contains the change causing the bug.

Thanks!

Comment 6

[re+github+er]

Hi Valentin, will hopefully complete task 1 in the next 48 hours, and task 2 within the week depending on how fast I get up to speed with mozregression.

Comment 7

[re+github+er]

(In reply to Valentin Gosu [:valentin] (he/him) from comment #5)

1. Capture some HTTP logs when reproducing this bug: https://firefox-source-docs.mozilla.org/networking/http/logging.html
   • this will allow us to see what the answer from the DoH server is and what is actually failing.

@[:valentin]: While capturing HTTP logs the log file was quite large (3.7 GB) using the default logging options "timestamp,sync,nsHttp:5,cache2:5,nsSocketTransport:5,nsHostResolver:5".

Can you suggest a narrower logging filter to reduce the file size so that it's hopefully less than 100 MB?

Comment 8

[Valentin Gosu [:valentin] (he/him)]

Just start the logging before loading the page, then stop it immediately after. You don't need to capture more than a couple page loads.
The logs are text files, so they compress rather well, so you can put them in a zip archive that won't be too big.

Comment 9

[re+github+er]

(In reply to Valentin Gosu [:valentin] (he/him) {{ PTO 3 July - 6 August }} from comment #8)

Just start the logging before loading the page, then stop it immediately after. You don't need to capture more than a couple page loads.
The logs are text files, so they compress rather well, so you can put them in a zip archive that won't be too big.

Okay will capture only until the first page fails to load and gzip the text file.

Unfortunately I also got auto-upgraded to 114.0.2 and it seems that under this version the page loads appeared to happen even without a VPN active. Am retesting to make sure it's not due to some temporary issues with DNS.

Comment 10

[Valentin Gosu [:valentin] (he/him)]

Please reopen if this is still happening and you can provide logs. Thanks!