Closed Bug 1661059 Opened 5 years ago Closed 2 years ago

WebRTC/ICE prefers temporary IPv6 address even when blocked by VPN client

Categories

(Core :: WebRTC: Networking, defect, P3)

Product:
Core
Component:
WebRTC: Networking
Version:
79 Branch
Type:
defect

Tracking

()

Status:
RESOLVED INCOMPLETE

People

(Reporter: public, Unassigned)

Details

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:79.0) Gecko/20100101 Firefox/79.0

Steps to reproduce:

  1. Have a client computer with a IPv6/IPv4 dualstack setup running Firefox 79, Firefox 80 or Firefox Nightly (2020-08-24).

  2. Have a temporary IPv6 address on your client computer.

  3. Have access to a WebRTC server (e.g. Big Blue Button) which actually speaks IPv6 (and IPv4, but this is not necessary) to the clients.

  4. Start a VPN client on your computer (e.g. Cisco AnyConnect) which has the following properties:

    4.1) It should block all traffic which originates from non-tunneled interfaces.

    4.2) It should assign a VPN IPv6 address to your computer.

  5. Start the WebRTC session (audio, video or screen sharing) connection to the mentioned WebRTC server from the described client.

Actual results:

In the majority of the cases, I get a WebRTC error 1007.

More precisely: I get this error whenever the Big Blue Button website itself (i.e. the WebRTC web-socket management connection) has been loaded via IPv6 in the first place. In this case, FreeSWITCH (the component used for bidirectional WebRTC audio in Big Blue Button) insists on IPv6 being used for the actual WebRTC data.

(When the BBB website has been loaded via IPv4 and the web-socket has therefore been established via IPv4, everything works well as FreeSWITCH is only interested in IPv4 addresses in this case).

During the WebRTC/ICE step, Firefox seems to prefer a temporary IPv6 address at all costs whenever such an address is available.

In my case, this temporary IPv6 address is made up by my MacBook (macOS 10.15.6) based on the IPv6 prefix it gets from my home router. When connecting to the university's VPN network, my MacBook gets assigned another IPv6 address (from the VPN server). At the same time, the Cisco AnyConnect VPN client actively prevents my non-tunneled (both, IPv4 or IPv6) interfaces from interchanging any data. Particularly, the temporary IPv6 address my MacBook has (which is the best of all addresses according to Firefox' address algorithm) has now been "muted".

In the ICE step, Firefox offers its local IPv4 address, the VPN IPv4 address, and the local temporary IPv6 address. Due to FreeSWITCH's stickiness, the IPv4 addresses don't come into play. As the only proposed IPv6 address is muted by the VPN client, the WebRTC/ICE negotiations fail.

I work around this problem by:

  • Disabling the generation and usage of temporary IPv6 addresses on my client.
  • Using a pure IPv4 setup on the server.
  • Using a pure IPv4 setup on the client.
  • Shutting down the VPN.
  • Using another VPN client, which does not actively block side-traffic (i.e. the builtin VPN client from Apple just sends the packets aside from the VPN tunnel if Firefox forces to use a certain interface, although there exists a preferred route through the VPN tunnel - it's better we don't want to discuss these potential security implications here).
  • Using Firefox older than version 79.
  • Using another web browser (Chrome and Safari do not show this problem).
  • Using another WebRTC server. E.g., Kurento (used for video, screen sharing, and listen-only audio) just falls back to IPv4 if IPv6 fails.

Maybe it would help to tell the VPN server to let the client choose a temporary IPv6 address which is part of the VPN tunnel.

Unfortunately, the users just see the WebRTC error and are puzzled as they have nothing changed (besides an auto-update of Firefox).

Expected results:

WebRTC should work without any problems, also via IPv6. Preferring a temporary address in WebRTC might be a nice act, but it should not happen at all costs. When the address is not working, Firefox should use any of the other available IPv6 addresses.

It seems there is a relationship to the resolution of bug #1183145. When commenting out the following lines, the behavior is gone:

https://searchfox.org/mozilla-central/rev/23dd7c485a6525bb3a974abeaafaf34bfb43d76b/media/mtransport/third_party/nICEr/src/stun/addrs.c#123-127

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Component: Untriaged → WebRTC: Networking
Product: Firefox → Core

Here the patch can be found which caused these changes (and most probably introduced the bug): https://phabricator.services.mozilla.com/D76043

On first glance, I am not sure what we are permitted to do about this. The relevant IETF specifications forbid us from even disclosing (much less using) non-temp IPv6 when temp IPv6 are available. If the VPN wants to disable temp IPv6, it probably needs to remove those addresses entirely, instead of simply making them drop packets.

This is indeed a very valid objection.

I think the VPN does not actually actively want to disable temporary IPv6 addressing, it just assigns an address which is not marked as temporary, but blocks any connection which goes not through the tunnel. I can have a talk to our university's VPN administrators about whether this is a configuration setting there. Maybe it is possible to have the VPN server an IPv6 prefix assigned in which the client can create a temporary IPv6 address.

I also have a report of a colleague of mine: Her ISP does not support dual-stack, i.e. her home router only has one routable IPv4 address and her home router is doing classical IPv4-to-IPv4 NAT. When she starts VPN, her WebRTC behaves also broken.

At first, we did not understand why she is affected by this bug at all.

The reason: Her router assigns a "unique local address" prefix (fd00::/8) to her computer. Her Firefox 79 (on macOS) prefers again (according to the above-mentioned piece of code in the "addrs.c") a temporary IPv6 address (within this unique local address space) over the IPv6 address she got from the VPN client. So Firefox obviously seems to rank even a non-routable temporary address higher than the routable address assigned by the VPN client - but it is obvious when it comes to the IETF specification.

This means: In my colleague's case, even a not-so-strict VPN client (e.g. Apple's built-in VPN client) causes the same problem in Firefox 79 or 80 because announcing fd00::.... to the WebRTC server as the only IPv6 address is simply not a good idea as "fd00::/8" never works - even without actively blocking it.

(When there is no VPN client active, her computer only has an IPv4 route to the outside world. In this case, the Big Blue Button website is loaded via IPv4, the management web-socket is established via IPv4 and the whole FreeSWITCH/WebRTC stuff is also handled via IPv4 only. Everything is okay in this case).

So I think the main problem is that the VPN seems to assign a single address to the client instead of activating the "IPv6 privacy behavior".

Yeah, this is one of those cases where the theory (as laid out in IETF specs) and reality don't quite mesh. Most network applications don't involve disclosing IP addresses per se, they just try to use addys until something works. With ICE, you've got to disclose your addresses to the other end before you can really start trying to use them, which is not something we want to do with non-temp addys.

If the VPN were marking those temp IPv6 as deprecated, or disabling temp IPv6 altogether, maybe that would work too?

Maybe we could do some sort of really basic check to see whether the temp IP address was usable at all, and fall back if not. I wonder what such a check would look like in practice.

(In reply to Steffen Moser from comment #6)

I also have a report of a colleague of mine: Her ISP does not support dual-stack, i.e. her home router only has one routable IPv4 address and her home router is doing classical IPv4-to-IPv4 NAT. When she starts VPN, her WebRTC behaves also broken.

At first, we did not understand why she is affected by this bug at all.

The reason: Her router assigns a "unique local address" prefix (fd00::/8) to her computer. Her Firefox 79 (on macOS) prefers again (according to the above-mentioned piece of code in the "addrs.c") a temporary IPv6 address (within this unique local address space) over the IPv6 address she got from the VPN client. So Firefox obviously seems to rank even a non-routable temporary address higher than the routable address assigned by the VPN client - but it is obvious when it comes to the IETF specification.

This means: In my colleague's case, even a not-so-strict VPN client (e.g. Apple's built-in VPN client) causes the same problem in Firefox 79 or 80 because announcing fd00::.... to the WebRTC server as the only IPv6 address is simply not a good idea as "fd00::/8" never works - even without actively blocking it.

(When there is no VPN client active, her computer only has an IPv4 route to the outside world. In this case, the Big Blue Button website is loaded via IPv4, the management web-socket is established via IPv4 and the whole FreeSWITCH/WebRTC stuff is also handled via IPv4 only. Everything is okay in this case).

So I think the main problem is that the VPN seems to assign a single address to the client instead of activating the "IPv6 privacy behavior".

So, fd00::... can work in cases where the other end is on the same local network. Therefore we try to use it.

(In reply to Byron Campen [:bwc] from comment #8)

(In reply to Steffen Moser from comment #6)

So I think the main problem is that the VPN seems to assign a single address to the client instead of activating the "IPv6 privacy behavior".

So, fd00::... can work in cases where the other end is on the same local network. Therefore we try to use it.

O.K., this is true... In a local network, we actually can have this... Sorry...

What I am also not sure about yet is why we don't just fall back to TURN and TCP...

I'm guessing because we're trying to communicate with the TURN server via an address that's dropping all packets?

Severity: -- → S2
Priority: -- → P3

So, we have multiple things causing this:

  1. VPN software that drops traffic on pre-existing interfaces instead of disabling those interfaces or marking them as deprecated; this is bad behavior.
  2. Remote endpoint software that refuses to perform ICE with candidates whose IP was not used to load the origin; this is really bad behavior.
  3. Firefox is not ensuring that it gathers the interface used to load the origin when that interface is non-temporary, apparently? This is bad behavior, if true.

Can you attach a copy of about:webrtc for this case?

Severity: S2 → S3
Flags: needinfo?(public)

Unfortunately, I don't have access to an IPv6-enabled BigBlueButton infrastructure anymore. For this reason, I was not able to provide the required results, yet. Though, I am still working on this and going to look for a solution to provide you with the requested information.

I'm a BigBlueButton developer working on this problem from our end.

The behavior I'm seeing involves Firefox 99 in a dual stack environment where the globally routable IPv6 address is auto-configured using eui64. When connecting to the BigBlueButton server over IPv6, Firefox filters out all the IPv6 addresses (because they expose MAC addresses) and offers only IPv4 addresses over SIP, which our FreeSWITCH configuration currently rejects, as it only takes IPv6 addresses from clients that connect via IPv6.

I'm debating how to change our address selection policy in FreeSWITCH.

I'm going to read over this thread again since I don't fully understand this bug. In the meantime, let me know if you want specific logs from one of my test installations.

The question is: Wouldn't perhaps the fullaudio bridge available in BBB 2.5 (https://docs.bigbluebutton.org/2.5/new.html#two-way-microphone-connections-using-mediasoup) solve this problem?

A needinfo is requested from the reporter, however, the reporter is inactive on Bugzilla. Closing the bug as incomplete.

For more information, please visit BugBot documentation.

Status: UNCONFIRMED → RESOLVED
Closed: 2 years ago
Flags: needinfo?(public)
Resolution: --- → INCOMPLETE
You need to log in before you can comment on or make changes to this bug.