1315738 - IPv6 failure on dual stack VPN - OSX

Status: RESOLVED WORKSFORME

(Core :: Networking: HTTP, defect, P2)

Description

[CraigT]

Attachment: log.txt.0

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.35 Safari/537.36

Steps to reproduce:

OS: OSX 10.11.6
Networking:
IPv4: rfc1918 based NAT
IPv6: OpenVPN initiated utun device
/etc/resolv.conf: Lists two IPv4 hosts, each only IPv4 connected but return AAAA when queried.

Firefox: 51.0a2 (2016-11-07) (64-bit) - Developer Edition
Profile: clean/new
network.dns.disableIPv6 = false

This has been reported to me by several developers within my organisation. I was able to quickly reproduce. IPv6 has been deployed as a VPN overlay within our organisation, allocating adresses to each user on a common /64. IPv4 and typical network services (e.g. DNS) all use the original network, with IPv6 traffic routed over the VPN.


Actual results:

Firefox only connects to IPv4 addresses on my VPN network deployment.
Confirmed using 'about:networking' and netstat.

Using 6only hosts e.g. http://ipv6.whatismyv6.com/ Firefox returns:
"Server not found
Firefox can’t find the server at ipv6.whatismyv6.com."

The attached generated logs seem to indicate the problem may sit in address resolution on OSX.
This seems to be supported by 'about:networking' DNS lookup tool which returns:

IPs
NS_ERROR_UNKNOWN_HOST

System returns:
$ host ipv6.whatismyv6.com
ipv6.whatismyv6.com has IPv6 address 2001:4810::110



Expected results:

Safari, Chrome and other system tooling ping6 etc. all resolve AAAA records and connect using IPv6. Firefox is the outlier.

I have other, non-VPN based IPv6 networks available and Firefox works 'fine' in these deployments. The issue seems to be specific to the utun/VPN network and DNS resolution.

A final observation is that 'http://test-ipv6.com/' makes a note that:
 "Your browser has real working IPv6 address - but is avoiding using it. We're concerned about this."

On Firefox, but not Chrome/Safari, even when connected to an IPv6 native physical network.

Comment 1

[Dragana Damjanovic [:dragana]]

can you check which dns query firefox sends and what is a response? (e.g. using wireshark)

Comment 2

[CraigT]

The query is only for an A record, no AAAA is ever sent.
I can send a pcap, but it's very uninteresting.

Repeated attempts at lookups, show Firefox caching initial response.

Comment 3

[CraigT]

Split view VPN (dual stack):
When Firefox initially opens in a clean profile DNS queries are sent to a few mozilla domains.
www.mozilla.org
addons.mozilla.org
self-repair.mozilla.org

These are all A records only.

----

Native dual stack v6:
When Firefox initially opens in a clean profile DNS queries are sent to a few mozilla domains.
www.mozilla.org
addons.mozilla.org
self-repair.mozilla.org

Firefox sends A, followed by AAAA DNS requests.

----

It seems like some kind of address detection must be in place that does not detect my 6 routes/addressing.

Comment 4

[Dragana Damjanovic [:dragana]]

may the problem is AI_ADDRCONFIG.
We use getaddrinfo to query for ip address and I thing it is deciding not to query for ipv6 because the host does not have one.

Comment 5

[CraigT]

I may have found it...

In 'system/mac/nsNetworkLinkService.mm'
You query the network service state to determine IPv4 and IPv6 capability.

Native dual stack updates this, however the openvpn client does not.
This leaves:

$ scutil --nwi

...

IPv6 network interface information
   No IPv6 states found

   REACH : flags 0x00000000 (Not Reachable)

Network interfaces: en0

---

I have a perfectly serviceable IPv6 network, but the OSX apis haven't been told about it.

Comment 6

[Dragana Damjanovic [:dragana]]

Looking at the code, the code from system/mac/nsNetworkLinkService.mm will not block AAAA queries, e.g. will not set disableIPv6.

Daniel, can you confirm this?

Comment 7

[--]

Rhe nsNetworkLinkService code listens to updates to IPv6 network details as well as IPv4 status. That section of the code doesn't know if IPv6 works or not and will just register "network changes".

I don't see how that code will affect Firefox use of IPv6 or not for connections. If that code fails, it would only lead to network changes not getting detected automatically, but for example forced reloads or restarting the browser would then repair that mistake.

I think comment #5 is more of an explanation: it looks like the system doesn't believe there is any IPv6 capable interfaces.

Comment 8

[CraigT]

Daniel, perhaps you can highlight the tests performed by Firefox on OSX to assert whether IPv6 is present?

I will perform tests on FF/Linux later today to confirm whether this is Mac specific.

Comment 9

[CraigT]

I can confirm that using Firefox on Debian linux with 'identical' (as close as it can be) networking, Firefox detects and uses the IPv6 connectivity. So at present this appears to be Mac/OSX specific.

I have confirmed it on exists on 10.11.x and 10.12.x (current and last), as well as FF49 and FF51.

Comment 10

[Firefox Bug Husbandry Bot]

Bulk change to priority: https://bugzilla.mozilla.org/show_bug.cgi?id=1399258

Comment 11

[Sam Birch [:smammy]]

I think this isn't a Firefox bug, but a macOS quirk. The gist of it is this:

* Programs that set up tunnels are supposed to register them as "services" with the macOS SystemConfiguration framework.
* Most don't, probably because the documentation is nonexistent.
* The macOS resolver looks at SystemConfiguration to decide whether to make A requests or AAAA requests or both.

I wrote a little guide about how to kludge this in case you have a tunnel that's not being noticed. I hope it's helpful.
https://gist.github.com/smammy/3247b5114d717d12b68c201000ab163d

Comment 12

[Selena Deckelmann :selenamarie :selena]

Thanks, Sam! Closing WORKSFORME. Let us know if there's more research to do here.