1305965 - [e10s] blocks remote peer reflexive discovery through UDP packet filter behaving like address restricted NAT

Status: NEW

(Core :: WebRTC: Networking, defect, P3)

Description

[Anton Teykhrib]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.34 Safari/537.36

Steps to reproduce:

1. Webrtc gateway and firefox deployed in the same LAN, but gateway also has public ip (will be named as ip_private_gw and ip_public_gw futher).
2. Firefox starts call (SDP contains candidates with public ip (srflx) and private ip (host) (will be named as ip_public_ff and ip_private_ff futher)
3. Gateway answers to firefox with SDP with only ip_public_gw (as host type)
4. ICE failed.


Actual results:

All checks from firefox comes from ip_private_ff to ip_public_gw. But STUN responses and checks from gw comes from ip_private_gw to ip_private_ff. Also Firefox doesn't send any STUN responses to gateway checks.


Expected results:

It works absolutely fine in Firefox 50 (on same machine with same gateway): when FF50 receives first stun response from ip_private_gw, FF50 sends all subsequent checks to ip_private_gw as well as STUN responses.

FF52 works same as FF51.

Comment 1

[Byron Campen [:bwc]]

I bet this is due to bug 1297416, but can you attach some pcaps?

Comment 2

[Anton Teykhrib]

Attachment: two_calls_anonimized.pcapng

Here is pcap with STUN from 2 calls - first call is from FF51 (failed) and the second one is from FF50 (successfull). IP binding:
ip_private_gw  192.168.241.192
ip_public_gw   1.1.1.73
ip_private_ff  192.168.225.183
ip_public_ff   1.1.1.253

Comment 3

[Byron Campen [:bwc]]

Do you think this might be due to the triggered check modification we made?

Comment 4

[Byron Campen [:bwc]]

Can you get us console (ie; cout/cerr) output with the following in the environment for 51 (or 52)?

R_LOG_LEVEL=7 R_LOG_DESTINATION=stderr

Comment 5

[Byron Campen [:bwc]]

Try this:

https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-win32/1472746340/

and this:

https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-central-win32/1472744960/

Comment 6

[Nils Ohlmeier [:drno]]

Yes sounds pretty much like bug 1298991 caused this. But I'm failing to wrap my head around what exactly is going wrong here in this scenario. The log file you requested in comment #4 would really help to understand what is happening here.

Comment 7

[Nils Ohlmeier [:drno]]

Another way how this could break is if one of the IP leakage options was tweaked in between. Just mentioning this so everyone is aware. I'm assuming Anton has not changed anything in between and/or uses the same user profile when testing with different versions.

Comment 8

[Anton Teykhrib]

Greetings Nils and Byron.
I tried both versions, provided by Byron with no luck - both failed with STUN. Also I didn't any tweaking and use same profile (as 
Sorry, but I failed to get log. My steps to setup logging:
1. Install proper version of Nightly (from links above)
2. Run cmd
3. set R_LOG_LEVEL=7
4. set R_LOG_DESTINATION=stderr
5. "c:\Program Files (x86)\Nightly\firefox.exe" --console (or without --console flag)
6. Console opened, but no logging to console (black empty screen)
Also tried set R_LOG_DESTINATION="C:\temp\ff.log", but file wasn't created.
Also tried set NSPR_LOG_FILE="C:\temp\firefox_nspr.log", and again file wasn't created.
What I'm missing in logging setup?

Comment 9

[Anton Teykhrib]

Tried to use zip archives (extracted to different paths) to clearly distinguish builds, and got the same results.
And nspr log file created, but empty (mistake in comment #8).

Comment 10

[Byron Campen [:bwc]]

Try running all of this in a Non-e10s window, I hear that e10s logging on windows isn't working right now.

Comment 11

[Anton Teykhrib]

Oh, it's nice thing - when I disable e10s, both builds starts work properly (also nspr log filled with proper data)! But when I switched e10s on, old erroneus behavior returns. So, I can provide logs when everithing is ok (are you need them?), but I can't provide logs with errors, sorry.

Comment 12

[Anton Teykhrib]

In my stable Firefox about:support shows that
Multiprocess Windows 	0/1 (Disabled by add-ons)
So, when I disabled Firebug addon, my stable firefox fails with ICE as tested nightlies.

Comment 13

[Byron Campen [:bwc]]

Curiouser and curiouser... perhaps this has something to do with the e10s UDP filter...

Comment 14

[Byron Campen [:bwc]]

Ah, of course. The fact that the STUN responses from the gateway are originating from a different IP address than Firefox sent the STUN request to means that the e10s STUN UDP filter will drop them.

Can you attach about:webrtc? I have a feeling we are missing some candidates...

Comment 15

[Nils Ohlmeier [:drno]]

So Firefox Multiprocess has a packet filter build in which makes Firefox behave like it is behind an address restricted NAT.
Since your server does not provide its host candidates it effectively behaves as if it is behind a symmetric NAT as well.
As there is no TURN server available this is expected to fail.

Comment 16

[Nils Ohlmeier [:drno]]

I forgot to mention in my previous comment: you have three options to fix this on your end (I'm assuming that this is actually only an issue in the development office which has private routing directly to the server):

1) trickle the host candidate
2) only send STUN responses via the advertised server reflexive candidate
3) use a TURN relay

Comment 17

[Anton Teykhrib]

Thank you!
Yes, only development server is affected (production server hasn't private connectivity with clients).
My server advertises only 1.1.1.73 in SDP (as host candidate, because NIC has public ip). So, if it will advertise also 192.168.241.192 (as I understand first option), everything will be ok? 
I tested the second option and ICE completed succefully.

Comment 18

[Nils Ohlmeier [:drno]]

(In reply to Anton Teykhrib from comment #17)
> My server advertises only 1.1.1.73 in SDP (as host candidate, because NIC
> has public ip). So, if it will advertise also 192.168.241.192 (as I
> understand first option), everything will be ok? 

Yes if you include the 192.168 address from your server as a candidate in the signaling it should also work as Firefox will then build candidates with that address. Which will result in Firefox sending binding requests towards that address and then the packet filter will let requests and responses from and to that address pass.

Comment 19

[Anton Teykhrib]

Ok, for now I'll prefer second option (with one address for SDP and STUN request/responses). Thank you again!

Comment 20

[Bulk Bug Changes for mreavy's org]

Mass change P2->P3 to align with new Mozilla triage process.