[e10s] blocks remote peer reflexive discovery through UDP packet filter behaving like address restricted NAT

NEW
Unassigned

Status

()

Core
WebRTC: Networking
P3
normal
Rank:
29
2 years ago
7 months ago

People

(Reporter: Anton Teykhrib, Unassigned)

Tracking

51 Branch
x86_64
Windows 10
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

2 years ago
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.34 Safari/537.36

Steps to reproduce:

1. Webrtc gateway and firefox deployed in the same LAN, but gateway also has public ip (will be named as ip_private_gw and ip_public_gw futher).
2. Firefox starts call (SDP contains candidates with public ip (srflx) and private ip (host) (will be named as ip_public_ff and ip_private_ff futher)
3. Gateway answers to firefox with SDP with only ip_public_gw (as host type)
4. ICE failed.


Actual results:

All checks from firefox comes from ip_private_ff to ip_public_gw. But STUN responses and checks from gw comes from ip_private_gw to ip_private_ff. Also Firefox doesn't send any STUN responses to gateway checks.


Expected results:

It works absolutely fine in Firefox 50 (on same machine with same gateway): when FF50 receives first stun response from ip_private_gw, FF50 sends all subsequent checks to ip_private_gw as well as STUN responses.

FF52 works same as FF51.
(Reporter)

Updated

2 years ago
OS: Unspecified → Windows 10
Hardware: Unspecified → x86_64

Updated

2 years ago
Component: Untriaged → WebRTC: Networking
Product: Firefox → Core

Comment 1

2 years ago
I bet this is due to bug 1297416, but can you attach some pcaps?
Flags: needinfo?(teihrib)

Updated

2 years ago
Whiteboard: [needinfo 2016/09/28 to reporter]
(Reporter)

Comment 2

2 years ago
Created attachment 8796181 [details]
two_calls_anonimized.pcapng

Here is pcap with STUN from 2 calls - first call is from FF51 (failed) and the second one is from FF50 (successfull). IP binding:
ip_private_gw  192.168.241.192
ip_public_gw   1.1.1.73
ip_private_ff  192.168.225.183
ip_public_ff   1.1.1.253
Flags: needinfo?(teihrib)

Comment 3

2 years ago
Do you think this might be due to the triggered check modification we made?
Flags: needinfo?(drno)

Comment 4

2 years ago
Can you get us console (ie; cout/cerr) output with the following in the environment for 51 (or 52)?

R_LOG_LEVEL=7 R_LOG_DESTINATION=stderr
Flags: needinfo?(teihrib)

Updated

2 years ago
Whiteboard: [needinfo 2016/09/28 to reporter] → [needinfo 2016/09/29 to reporter]
Yes sounds pretty much like bug 1298991 caused this. But I'm failing to wrap my head around what exactly is going wrong here in this scenario. The log file you requested in comment #4 would really help to understand what is happening here.
backlog: --- → webrtc/webaudio+
Rank: 19
Depends on: 1298991
Flags: needinfo?(drno)
Priority: -- → P1

Updated

2 years ago
Status: UNCONFIRMED → NEW
Ever confirmed: true
Another way how this could break is if one of the IP leakage options was tweaked in between. Just mentioning this so everyone is aware. I'm assuming Anton has not changed anything in between and/or uses the same user profile when testing with different versions.
(Reporter)

Comment 8

2 years ago
Greetings Nils and Byron.
I tried both versions, provided by Byron with no luck - both failed with STUN. Also I didn't any tweaking and use same profile (as 
Sorry, but I failed to get log. My steps to setup logging:
1. Install proper version of Nightly (from links above)
2. Run cmd
3. set R_LOG_LEVEL=7
4. set R_LOG_DESTINATION=stderr
5. "c:\Program Files (x86)\Nightly\firefox.exe" --console (or without --console flag)
6. Console opened, but no logging to console (black empty screen)
Also tried set R_LOG_DESTINATION="C:\temp\ff.log", but file wasn't created.
Also tried set NSPR_LOG_FILE="C:\temp\firefox_nspr.log", and again file wasn't created.
What I'm missing in logging setup?
(Reporter)

Comment 9

2 years ago
Tried to use zip archives (extracted to different paths) to clearly distinguish builds, and got the same results.
And nspr log file created, but empty (mistake in comment #8).

Comment 10

2 years ago
Try running all of this in a Non-e10s window, I hear that e10s logging on windows isn't working right now.
(Reporter)

Comment 11

2 years ago
Oh, it's nice thing - when I disable e10s, both builds starts work properly (also nspr log filled with proper data)! But when I switched e10s on, old erroneus behavior returns. So, I can provide logs when everithing is ok (are you need them?), but I can't provide logs with errors, sorry.
(Reporter)

Comment 12

2 years ago
In my stable Firefox about:support shows that
Multiprocess Windows 	0/1 (Disabled by add-ons)
So, when I disabled Firebug addon, my stable firefox fails with ICE as tested nightlies.
Flags: needinfo?(teihrib)

Comment 13

2 years ago
Curiouser and curiouser... perhaps this has something to do with the e10s UDP filter...

Comment 14

2 years ago
Ah, of course. The fact that the STUN responses from the gateway are originating from a different IP address than Firefox sent the STUN request to means that the e10s STUN UDP filter will drop them.

Can you attach about:webrtc? I have a feeling we are missing some candidates...
Flags: needinfo?(teihrib)
So Firefox Multiprocess has a packet filter build in which makes Firefox behave like it is behind an address restricted NAT.
Since your server does not provide its host candidates it effectively behaves as if it is behind a symmetric NAT as well.
As there is no TURN server available this is expected to fail.
Rank: 19 → 29
No longer depends on: 1298991
Flags: needinfo?(teihrib)
Priority: P1 → P2
Summary: ICE negotiation failed with peerreflexive candidate → [e10s] blocks remote peer reflexive discovery through UDP packet filter behaving like address restricted NAT
I forgot to mention in my previous comment: you have three options to fix this on your end (I'm assuming that this is actually only an issue in the development office which has private routing directly to the server):

1) trickle the host candidate
2) only send STUN responses via the advertised server reflexive candidate
3) use a TURN relay
Whiteboard: [needinfo 2016/09/29 to reporter]
(Reporter)

Comment 17

2 years ago
Thank you!
Yes, only development server is affected (production server hasn't private connectivity with clients).
My server advertises only 1.1.1.73 in SDP (as host candidate, because NIC has public ip). So, if it will advertise also 192.168.241.192 (as I understand first option), everything will be ok? 
I tested the second option and ICE completed succefully.
(In reply to Anton Teykhrib from comment #17)
> My server advertises only 1.1.1.73 in SDP (as host candidate, because NIC
> has public ip). So, if it will advertise also 192.168.241.192 (as I
> understand first option), everything will be ok? 

Yes if you include the 192.168 address from your server as a candidate in the signaling it should also work as Firefox will then build candidates with that address. Which will result in Firefox sending binding requests towards that address and then the packet filter will let requests and responses from and to that address pass.
(Reporter)

Comment 19

2 years ago
Ok, for now I'll prefer second option (with one address for SDP and STUN request/responses). Thank you again!
Mass change P2->P3 to align with new Mozilla triage process.
Priority: P2 → P3
You need to log in before you can comment on or make changes to this bug.