1722661 - Firefox on VPN Split tunneling does not resolve local DNS

Status: RESOLVED INVALID

(Core :: Networking, defect)

Description

[Alex Haan]

Attachment: firefox-vpn-dns-resolve-log.txt

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0

Steps to reproduce:

I asked a question about this issue I have been on the support site first: https://support.mozilla.org/en-US/questions/1339832 and had decided to report it as a bug after no response and more than 50 views.

I'm using Firefox on Ubuntu 20.10, running Gnome. During this time I've also used Firefox 89 and probably also 88.

Working from home, a few months ago, to reduce the load on the office internet connection, Split Tunneling was enabled in the VPN server configuration. And in my VPN configuration I have "Use this connection only for resource on its network" enabled for IPv4 (and IPv6 is disabled).

Since then I'm having DNS resolve issues in Firefox for domains defined by the DNS server that's on the VPN network. This happens for our local development domains and some internal services.

Our development URL structure is like projectname.[optional personal identifier.]localhost.[type].nl. Without the [optional personal identifier], or when nothing is configured for it, the ip address is 127.0.0.1. [type] is one of two TLD's we control.

• As far as I know I do not have DoH enabled (the attached about:networking log tells that TRR is off or disabled).
• I do have dnsmasq enabled (with configuration for the above domain formats to route to 127.0.0.1 also) in an attempt to fix some issues before the split tunneling was configured correctly. And in an attempt to try and get Firefox to handle these DNS resolving issues.
• dig on the command line for the same domain works just fine.
• Adding hostnames to /etc/hosts works. But that's becoming a chore, working on dozens of projects over months. There are also some internal services that I have to define myself.

Some relevant about:config settings:

• network.dns.disablePrefetch is set to true.
• network.trr.mode is set to 5 (but 0 doesn't make a difference).
• network.trr.resolvers is emptied.
• I attempted to fix things by defining network.trr.builtin-excluded-domains to localhost,local,*.localhost.[type].nl,*.localhost.[type].nl,*.hq.[type].nl.

Let me know if you want any other settings or logs.

Actual results:

Firefox (usually*) fails to resolve domains defined by the DNS server through the VPN connection.

(*) Sometimes it seems to work for some time. But after a few hours it suddenly fails. This could be because I've toggled the "Use this connection only for resource on its network" VPN setting at some point. I'm not sure about all such occurrences.

I've attached a log from about:networking with replaced actual projectname and type.

Is there something in the log that explains why it doesn't seem to be using the DNS resolver of the operating system? Can I verify otherwise that DoHIs there something else I can try?

Comment 1

[Alex Haan]

Apologies for the markup there. Can't fix the markup because https://bugzilla.mozilla.org/show_bug.cgi?id=1666041 .

Also the "What should have happened" section seems to have been dropped. But that was short and obvious:

The development domains and internal services resolve just like dig or other tools do.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Networking' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Comment 3

[Valentin Gosu [:valentin] (he/him)]

(In reply to Alex Haan from comment #0)

• Adding hostnames to /etc/hosts works. But that's becoming a chore, working on dozens of projects over months. There are also some internal services that I have to define myself.

That's because as far as I can tell, the error isn't coming from TRR but from the system's getaddrinfo (if the logs you posted are correct)
This could be caused by a bug in NSPR or quite likely a problem in the system's stub resolver, which is why editing /etc/hosts fixes it.
We have seen bugs before that turning VPNs on/off might affect functioning of the DNS system.
Could you check if the problems persist when you restart Firefox?

Comment 4

[Alex Haan]

Thank you for your message Valentin.

Before restarting the browser I just tried a random prefix for our pattern (including http://). At first that failed as before.
Just to make sure, I tried a 'DNS Lookup' for that in about:networking. That took several seconds - making me think I hadn't pressed the "Resolve" button. So I did press it again.
I then also tried to resolve the URL with dig in a console. I actually forgot what I tried first.

Dig of course worked. But then I noticed that the 'DNS Lookup' also showed 127.0.0.1. And indeed, visiting the random url now works.

There's no use right now to restart Firefox, because (for now) any other random (matching our pattern) URL actually works.
Though I don't know whether dnsmaq took care of the mapping, or whether the internal DNS server handled it.

Is there any follow up testing I can do? Or should I wait till the lookup somehow fails and try a restart then?

Comment 5

[Alex Haan]

A day later and it happened again. So I tried a restart of Firefox as Valentin suggested.

That doesn't work. The "DNS Lookup" in about:networking keeps giving NS_ERROR_UNKNOWN_HOST.

Comment 6

[Valentin Gosu [:valentin] (he/him)]

So after a restart DNS still doesn't work?
Then it seems to be different from other DNS doesn't work when I start/stop the VPN bugs.
Do you have any other proxy settings configured?
From what I can tell DNS over HTTPS is properly off here. So it's likely some issues with the local DNS resolver are causing this.

Comment 7

[Alex Haan]

Attachment: log.succes-and-failure.moz_log

New log with success (for domain defined in /etc/hosts) and failure for a different domain. And as last a success for a normal public domain using normal DNS lookup.

Comment 8

[Alex Haan]

(In reply to Valentin Gosu [:valentin] (he/him) from comment #6)

So after a restart DNS still doesn't work?
Then it seems to be different from other DNS doesn't work when I start/stop the VPN bugs.

Yesterday afternoon I had no issues (I didn't try random domains before your message). But this morning lookups failed again.
Restarting Firefox, or starting a parallel instance with an empty profile, doesn't help. "Clear DNS Cache" doesn't change anything either.

Do you have any other proxy settings configured?

I don't have any proxy settings defined, as far as I can see. And the empty profile should have remedied any browser settings.
In my Ubuntu network settings I have no proxy configured either.

From what I can tell DNS over HTTPS is properly off here. So it's likely some issues with the local DNS resolver are causing this.

What I don't understand is, if Firefox hands DNS lookup over to the local system, why does dig not have any issues with these lookups? Clearly there's a difference. So for now I'm still pointing in the direction of Firefox.

---

I added a new log with a succes, failure and success (for normal public DNS lookup).
The failure does stand out with that Calling 'res_ninit'. line. I did see that issue you created as a test. But I didn't understand what that's doing or what impact it could have. Why is that called when it fails?

Comment 9

[Alex Haan]

Oh wait. I just tried the failing domain in Edge (beta for Linux) and it fails there too.
Only when I add it to /etc/hosts does that now recognize the url.

So I guess I'll have to close this issue. But I have no idea why the browsers fail to do these DNS lookups while tools like dig and nslookup have no issue at all.

Thank you for your time though, Valentin.