Closed Bug 947801 Opened 9 years ago Closed 8 years ago

DNS resolution sporadically fails in Android 4.4/5.0 when connected to a VPN

Categories

(Core :: Networking: DNS, defect)

Product:
Core
Component:
Networking: DNS
Version:
25 Branch
Platform:
ARM
Android
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED DUPLICATE of bug 991923
Milestone:
mozilla35
Tracking Flags:
Tracking Status
firefox35 --- fixed
fennec 35+ ---

People

(Reporter: josh, Assigned: bagder)

References

Details

Attachments

(1 file, 1 obsolete file)

v2-0001-bug-947801-use-res_init-on-Android-to-reload-DNS-.patch
2.46 KB, patch
Details | Diff | Splinter Review 
User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0 (Beta/Release)
Build ID: 20131115105349

Steps to reproduce:

Specs:
--
* Nexus 5
* Android 4.4.1
* Firefox 26.0 Beta (the issue is present in the Stable release as well)
* OpenVPN for Android 0.6.1

This issue happens reliably every time I connect to a VPN and leave the connection active for an extended period of time.


Actual results:

Firefox appears to become unable to resolve any DNS queries. Attempts to access any site will hang until the request times out, but accessing an IP address directly still works.

If a page was recently accessed, and its DNS entry is still cached, then that site will still work too. It is important to note that only Firefox is affected by this issue, so it doesn't appear to be a systemic problem with the VPN in general. Chrome and any other alternative browsers are still able to resolve DNS entries and surf the web, and no other applications lose connectivity except for Firefox.

Firefox must be shut down and the DNS connection restarted in order to get things working again.


Expected results:

Firefox should be able to resolve the IP address for the specified site and display the requested page as intended.
OS: Linux → Android
Hardware: x86_64 → ARM
I have the same problem on Nexus 4 running Android 4.4. Firefox 25.0.1. It also affects pptp and ipsec VPNs, both using the stock applications coming with KitKat. DNS stops working as soon as the VPN is activated. Works again once it's deactivated. As per the reporter no other applications are affected. I didn't have any such problems with Android 4.3.
Component: General → Networking: DNS
Product: Firefox for Android → Core
Version: Firefox 26 → 25 Branch
Lets not loose track of this.
tracking-fennec: --- → ?
tracking-fennec: ? → -
steve, can you take a peek when you're free?
BTW, a good use-case I've found, personally, is to use Onavo (https://play.google.com/store/apps/details?id=com.onavo.android.onavoics) -- it's easy to toggle it on or off, and I've definitely had DNS-resolution problems with it on my HTC One.  Let me know if I can help with logging; I'm using Fennec nightly exclusively.
probably a (mostly) dup of 939318. Plan of record is to get [new hire] working on that mentored by steve in q1.
What Pat said. On my radar now and I'll try to get time to look at it in more detail.
As an interim solution, I installed the Dante SOCKS server (http://www.inet.no/dante/index.html) on my OpenVPN box and bound it to the tun0 interface. I'm using the following Dante configuration file on Debian 7:

/etc/danted.conf
--
logoutput: syslog

internal: 10.8.0.1 port = 1080
external: eth0

user.privileged: proxy
user.notprivileged: nobody

method: none

client pass {
  from: 10.8.0.0/24 to: 0.0.0.0/0
  log: error
}

pass {  
  from: 0.0.0.0/0 to: 0.0.0.0/0
  protocol: tcp udp
}
--

Then I configured Firefox for Android with these settings in about:config:
--
network.proxy.socks = 10.8.0.1
network.proxy.socks_port = 1080
network.proxy.socks_remote_dns = true
network.proxy.type = 1
--

This has been working flawlessly for several days now. I haven't had to restart Firefox or OpenVPN for Android at all since setting this up. It's also wildly faster and, for the first time in almost two months, I have not had any perpetual blue spinners or frustrating timeouts either. I suspect that "network.proxy.socks_remote_dns = true" is responsible for making the magic happen.

Obviously this doesn't help corporate VPN users very much (unless you are the sysadmin!) but if you have the means to setup a temporary SOCKS proxy that is accessible through your VPN then this will make Firefox for Android usable on KitKat until this is resolved.
I have this problem on the Nexus 5, Android 4.4.2, Firefox 28.0
I have issues with Firefox and OpenVPN, Firefox and Orbot
I found that after disconnect the OpenVPN connection in Android 4.4, the VPN connection DNS server is still in the device, so the request will send to the non-routable VPN DNS server and thus the HTTP connection in Firefox cannot be made.

I'm still not understand why firefox seems only affected, but i got another apps "PING & DNS" also affected so I think it is the root clause... to solve the problem, enable the airplane mode for a while and disable it again.
I'm also having this issue on my kitkat Android phone. Firefox is not able to resolve domain name after some seconds or minutes. The standard browser is working.

The server is a Debian box with standard package. 

I'm not sure but it seems that the Google play application also has some issues with resolving.
I have observed this issue on both Nexus 5 and Sony Z2 tablet. Both running latest system versions (4.4.4 and 4.4.2). The bug makes firefox unusable with VPN since DNS resolution isn't reliable.

It seems the problem could be caused by using out of sync dns properties, because net.dns1 (and net.dns2) doesn't always contain correct values.

Is Firefox for Android using net.dns1 and net.dns2 and its own Dns resolver, why? I can't find any documentation for the properties on developer.android.com which lead me to think they shouldn't be used by apps.
Setting the correct dns entries using setprop and restarting firefox seems to resolve the issue (at least temporarily). Of course the device needs to be rooted foe this to work.
Status: UNCONFIRMED → NEW
Ever confirmed: true
It took me a while to make the association but in my case it happens only when OpenVPN is connected and then disconnected. It has been happening ever since I started using OpenVPN on Android probably a year and a half ago. In that case I just used another browser until reboot.
See bug 1024614, which is the Android parts for the bug 939318 approach. It possibly fixes this problem.
In other words please try the build at https://aurora.mozilla.org and see if that build does not have the issue mentioned in this bug report.
Latest Nightly works fine for me, I'm using OpenVPN for Android.
Alright, I tested Aurora and it seems to load page before, during and after VPN connections. However in my case I cannot use it since it only has Firefox Accounts which is not feature complete yet.
Fixed by 1024614
Status: NEW → RESOLVED
Closed: 9 years ago
status-firefox35: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla35
I have exactly the same problem on Lollipop. When connecting to VPN with OpenVPN, Firefox, Firefox Beta & Aurora 35.0a2 wont change DNS server to those provided by VPN, so DNS resolution fails as long as a DNS server in LAN is used (e.g. DNS from wifi router).
(In reply to milfog from comment #20)
> I have exactly the same problem on Lollipop. When connecting to VPN with
> OpenVPN, Firefox, Firefox Beta & Aurora 35.0a2 wont change DNS server to
> those provided by VPN, so DNS resolution fails as long as a DNS server in
> LAN is used (e.g. DNS from wifi router).

Are you testing Firefox 35 (Aurora) where this was fixed?
tracking-fennec: - → 35+
Flags: needinfo?(johnnygpe)
(In reply to Brad Lassey [:blassey] (use needinfo?) from comment #21)

> Are you testing Firefox 35 (Aurora) where this was fixed?

yes, i also tested with Aurora 35.0a2 and nightly build fennec-36.0a1, it does not work with Android Lollipop.

https://developer.android.com/reference/android/net/ConnectivityManager.NetworkCallback.html

introduced in API Level 21
Flags: needinfo?(johnnygpe)
Flags: needinfo?(daniel)
Bug 1024614 was the NS_NETWORK_LINK_DATA_CHANGED adaption for Android, I wonder if the VPN case is possibly different and thus doesn't trigger that same logic?
Flags: needinfo?(daniel) → needinfo?(snorp)
I am too having the same issue as milfog and am unable to use firefox over openvpn
I also am able to reproduce with Aurora 35.0a2 (2014-11-25).  As soon as I connect to a VPN I lose DNS, and as soon as I disconnect, I'm able to connect to DNS - even without a page refresh, which would indicate that firefox is attempting to connect to the previous non-VPN DNS.
(In reply to Charlie from comment #25)

> firefox is attempting to connect to the previous non-VPN DNS.

Yes that's definitely the case, you can verify this by setting a static ip and an external dns (or advertise external dns with DNSmasq to the clients), then go to a test site like ipleak.net
Does anyone know if they're going to honor and change the status on this bug or if they'd like to open a new bug report?
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Assignee: nobody → daniel
You know what, it appears still sometimes not working on CyanogenMod 11 (Android 4.4.4, LG e970), I'm using Fennec Nightly and daily builds of CM.
Hi, I have android lollipop and this problem too.
Regards
We see various reports on this not working with recent versions and Android 4.4.4 or 5.0. Is this independent of OpenVPN app/client version? Does anyone who suffers from this problem know of a version combination of the involved components when it actually _did_ work?
I am having this issue on Android 5.0 (build LXE22.46-11 on Pure Edition Moto X 2014 (Victara)).

Native Android VPN client, L2TP/IPSec PSK VPN, doesn't matter if I set a fixed DNS server in Advanced Settings or not.

Happens whether the VPN is pinned up over WiFi or LTE/carrier data.
Summary: DNS resolution sporadically fails in Android 4.4 when connected to a VPN → DNS resolution sporadically fails in Android 4.4/5.0 when connected to a VPN
OS: Andoid Version 5.0
Device: Nexus 5 32GB
VPN Software: SurfEasy VPN
Firefox Version: 34.0

After connecting to the VPN, Firefox doesn't work. A lot of other apps including chrome work. Firefox responds with the "Server not found" page.
(In reply to Muneebpeeran from comment #33)
> OS: Andoid Version 5.0
> Device: Nexus 5 32GB
> VPN Software: SurfEasy VPN
> Firefox Version: 34.0
> 
> After connecting to the VPN, Firefox doesn't work. A lot of other apps
> including chrome work. Firefox responds with the "Server not found" page.

I was able to reproduce using their free VPN service on my Nexus 5 (5.0) as well (i.e, https://play.google.com/store/apps/details?id=com.surfeasy)
Eugen, can you reproduce this at all now? If so can you try adding res_init() before the getaddrinfo() call (or whatever equivalent nspr call there is)?
Flags: needinfo?(snorp) → needinfo?(esawin)
I couldn't reproduce bug 1051637, but I can reproduce bug 991923 given the new STR in comment 7 on all Fennec versions. It is also a failed DNS lookup issue there. I can have a look at that one, as it's more reliable to reproduce without additional apps.
Flags: needinfo?(esawin)
To make it clear, I mean this new STR https://bugzilla.mozilla.org/show_bug.cgi?id=991923#c7, not the auto-linked comment from this bug.
I have the same problem with my debian openvpn server. I'm using a Nexus 4 with Android 4.4.4. The current Firefox 34.0 and the Aurora version 35.0a2 is broken. 
The last working version is 29.0.1.
(In reply to Eugen Sawin [:esawin] from comment #37)
> To make it clear, I mean this new STR
> https://bugzilla.mozilla.org/show_bug.cgi?id=991923#c7, not the auto-linked
> comment from this bug.

bug 991923 is not involving any VPN though, just switching between two networks which may be somewhat different than this bug though so we shouldn't conflate them more than necessary before we know more.
Given the recent discoveries in bug 991923, I now believe these two bugs have the exact same cause.
(In reply to Daniel Stenberg [:bagder] from comment #40)
> Given the recent discoveries in bug 991923, I now believe these two bugs
> have the exact same cause.

feel free to dup bugs to make bugzilla saner :)
This patch A) makes configure check for res_init in case res_ninit doesn't exist as this seems to be the case on Android and B) if res_init is found and not res_ninit, the nsHostResolver.cpp gets a tweak to make it use res_init instead to re-init the resolver accordingly. Try-run is pending.
Attachment #8533654 - Flags: review?(snorp)
This version also adds a call to res_ninit() to the FlushCache() method, so that the reload is already made at the time of the next name resolve instead of relying on a failure first before it is called.

The try-run of v1 of this patch looks like this: https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=286877b6ea4f
Attachment #8533654 - Attachment is obsolete: true
Attachment #8533654 - Flags: review?(snorp)
Attachment #8533700 - Flags: review?(snorp)
(In reply to Daniel Stenberg [:bagder] from comment #44)
> Created attachment 8533700 [details] [diff] [review]
> v2-0001-bug-947801-use-res_init-on-Android-to-reload-DNS-.patch
> 
> This version also adds a call to res_ninit() to the FlushCache() method, so
> that the reload is already made at the time of the next name resolve instead
> of relying on a failure first before it is called.
> 
> The try-run of v1 of this patch looks like this:
> https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=286877b6ea4f

Does this patch actually fix the bug, or are you taking a guess?
I believe it fixes the bug, and I installed a patched version on my android phone and played around with it and it worked - I'll do some more testing with the updated patch and an actual VPN asap to see how it runs with that. So, not certain yet but I figured it was worth posting here to get early comments.
How do we load the patch for testing? Do I need to compile it from git?
(In reply to lostinignorance from comment #47)
> How do we load the patch for testing? Do I need to compile it from git?

Thanks for considering this!

You can build from (mozilla-central) source with the patch applied, sure. But assuming you trust our build architecture, you can go to the try-run link, find the Android buoild link, then check the build dir and download the APK from there.

Hang on a while longer and I'll get a build of my version 2 patch done and I can paste a direct link to that APK when completed.
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/daniel@haxx.se-286877b6ea4f/try-android-api-10/fennec-37.0a1.en-US.android-arm.apk

However you will need to uninstall any versions of nightly or aurora as this build is not signed with our key for released builds.
The build with version 2 of the patch which should hopefully react to DNS server changes faster, is here: http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/daniel@haxx.se-5bd29bcd28b7/try-android-api-10/fennec-37.0a1.en-US.android-arm.apk
Was (In reply to Daniel Stenberg [:bagder] from comment #50)
> The build with version 2 of the patch which should hopefully react to DNS
> server changes faster, is here:
> http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/daniel@haxx.se-
> 5bd29bcd28b7/try-android-api-10/fennec-37.0a1.en-US.android-arm.apk

On android 5.0, was still able to reproduce the bug with this build. I was able to load a site fine until connecting to a VPN.  Upon connecting to a VPN, DNS failed to resolve.  Immediately upon disconnecting from the VPN, the DNS resolved and the browser navigated to the page.
(In reply to Charlie from comment #51)
> On android 5.0, was still able to reproduce the bug with this build. I was
> able to load a site fine until connecting to a VPN.  Upon connecting to a
> VPN, DNS failed to resolve.  Immediately upon disconnecting from the VPN,
> the DNS resolved and the browser navigated to the page.

I am experiencing the exact same thing as charlie using the build referenced by daniel.  VPN still causes DNS issues.
Thanks a lot for this quick feedback. Does it mean that all DNS resolves fail while you have the VPN activated or just some specific ones?
I get no resolution out of Firefox, but all other apps (chrome, Pandora, etc.) have no issues over the vpn. Also running a test of IP shows correctly to everything being routed over the vpn connection via chrome.
(In reply to Daniel Stenberg [:bagder] from comment #53)
> Thanks a lot for this quick feedback. Does it mean that all DNS resolves
> fail while you have the VPN activated or just some specific ones?

Ditto what lostinignorance said.  I've *never* had a successful DNS resolution since this bug started being introduced.  Loading an IP address into Firefox when it is connected to a VPN is successful, but no DNS that I've tried has been.  If you have any that you'd like me to hit, let me know.
I tried the apk with the patch as well. Does not work using Cisco AnyConnect with my 4.4.4 samsung Tab S.
Interesting enough, I found the following may give you a false positive for this bug:

1. reboot device
2. connect to a strong wifi connection
3. connect to VPN

then test, it will work for awhile, leave app and do something else, That may be why your testing worked.

I think your fix is ok, but did you try to force tcp instead of udp as was suggested in https://bugzilla.mozilla.org/show_bug.cgi?id=1051637 which is marked as a dup? I  commented that I turned on PRLOG and it shows the nsHttpConnectionMgr just looping on timeouts and not connecting to anything.
(In reply to Ken from comment #56)

> I think your fix is ok

Maybe, but as it made no particular difference now I'm not sure it is worth pursuing. I'll continue digging anyway.

> but did you try to force tcp instead of udp as was
> suggested in https://bugzilla.mozilla.org/show_bug.cgi?id=1051637

No, as we use getaddrinfo() for name resolving we don't really have that level of control.

I think the problem is more related to what magic Android does for VPN (and name servers). It seems the problem occur for Android 4.4 or later and it affects Firefox way back as even version 25 has been mentioned to be affected.
We don't have res_init() in the resolv.h in the NDK, so Daniel's patch is still not going to do the job. As we have our own resolver in other-licenses/android anyway, the configure check isn't going to be useful. I've asked Eugen to try exposing res_ninit() in our resolv.h there and call it in nsHostResolver if we have MOZ_WIDGET_ANDROID. Or maybe just ANDROID, if we use that on B2G too. Pretty sure this will end up fixing things. The reason a new thread works is because we have stale crap stored in TLS, and res_ninit() will reinit that.
Depends on: 991923
On Android 5.0, when system establishes VPN, the VPN DNS are no longer written to system properties. Instead, you can retrieve them from the new APIs added to ConnectivityManager.
Duping this to 991923
Status: REOPENED → RESOLVED
Closed: 9 years ago → 8 years ago
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.