1505329 - WebRTC TURN connection not working

Status: RESOLVED INCOMPLETE

(Core :: WebRTC: Networking, defect, P5)

Description

[danilo.arcidiacono]

Attachment: bug_screenshots.zip

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36

Steps to reproduce:

Hello,
I'm having some trouble getting ICE candidates from TURN servers.
I have a coturn instance deployed locally, as a docker container.

I use this page https://webrtc.github.io/samples/src/content/peerconnection/trickle-ice/ to test the connectivity towards my local TURN server.


Actual results:

No relay candidates are gathered (it seems that Firefox can't even establish a connection with the TURN server). However, the same page does work with Chrome.

I've found this bug with the following Firefox versions:
    Firefox Quantum 62.0.3 (64 bit)
    Firefox Quantum 63.0.1 (64 bit)


Expected results:

Firefox should be able to contact the TURN server in order to gather relay ICE candidates.

Comment 1

[Byron Campen [:bwc]]

Try running your TURN server on a non-loopback address.

Comment 2

[danilo.arcidiacono]

Attachment: turn_test_cases.zip

Comment 3

[danilo.arcidiacono]

I've done a series of tests, using both my local TURN and a publicly reachable one.
I've also noticed that the URL scheme does change the behavior, both on Firefox and Chrome.

Case	Browser		Scheme				Turn			Outcome
------------------------------------------------------------------------------------------------------
1a		Firefox		https				Public IP		Doesn't work
2a		Firefox		file				Public IP		Works (2 candidates, different component)
3a		Firefox		https				Loopback IP		Doesn't work
4a		Firefox		file				Loopback IP		Doesn't work

1b		Chrome		https				Public IP		Works (3 candidates, only port changes)
2b		Chrome		file				Public IP		Works (1 relay candidate)
3b		Chrome		https				Loopback IP		Works (3 relay candidates)
4b		Chrome		file				Loopback IP		Works (1 relay candidate)

I've attached the related screenshots (turn_test_cases.zip) for each case (left side: Firefox, right side: Chrome).

Comment 4

[danilo.arcidiacono]

Also notice that I'm on Windows 10 and my work environment is behind a corporate proxy.

Comment 5

[Byron Campen [:bwc]]

Thanks for the breakdown!

Support for http relays is really minimal right now. But how would you like to test the fix over at bug 1203503? I'm interested to see how it works on some corporate http proxies! I need to fix a windows build error, and then I can point you to some binaries if you're interested.


Right now, TURN servers on loopback won't work if you don't grant getUserMedia permission, due to our implementation of draft-ietf-rtcweb-ip-handling. We will only use the default network interface in that case (ie; the interface the OS will use to connect to 8.8.8.8).  Updating our implementation to the latest version of that draft would let it work without getUserMedia I think, but only if the page was also hosted on localhost. Under the latest version of that draft, without asking getUserMedia you can only use the network interface you used to connect to the webpage.

Comment 6

[Byron Campen [:bwc]]

In order for loopback to work, I think you would also need to set some prefs.

Comment 7

[Byron Campen [:bwc]]

Here's a win64 binary with better http proxy support: https://queue.taskcluster.net/v1/task/XRcfaWuRTjyEXO7kGzYVHA/runs/0/artifacts/public/build/install/sea/target.installer.exe

And the 32-bit version: https://queue.taskcluster.net/v1/task/KJewoG5qSa6WJCynYoYhhA/runs/0/artifacts/public/build/install/sea/target.installer.exe

Comment 8

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

(In reply to Byron Campen [:bwc] from comment #5)
> Updating our implementation to the latest version of that draft would let it work without getUserMedia I think,
> but only if the page was also hosted on localhost. Under the latest version of that draft, without
> asking getUserMedia you can only use the network interface you used to connect to the webpage.

Do we have a bug open for doing that work?

Comment 9

[Byron Campen [:bwc]]

Filed bug 1506219 for that.

Comment 10

[danilo.arcidiacono]

Attachment: nightly_screenshots.zip

Comment 11

[danilo.arcidiacono]

(In reply to Byron Campen [:bwc] from comment #5)
> Thanks for the breakdown!
> 
> Support for http relays is really minimal right now. 

With "http relays" do you mean using "?transport=tcp" on ICE servers URLs? If so, using "?transport=udp" should work instead (I can't try this right now, because our corporate proxy blocks UDP traffic)?

> But how would you like to test the fix over at bug 1203503?

If I understand, the problem is that Firefox can't get past proxies protected by authentication, because it does not implement the related security protocols (at least for the WebRTC part). But then, why does it work when serving the web page from the hard drive?

> Here's a win64 binary with better http proxy support: 
> https://queue.taskcluster.net/v1/task/XRcfaWuRTjyEXO7kGzYVHA/runs/0/artifacts/public/build/install/sea/target.installer.exe

The 64-bit version of Firefox Nighty throws this error:

"Assertion failure: false (MOZ_ASSERT_UNREACHABLE: Opcode sequence includes commands after JMP), at z:/build/build/src/obj-firefox/dist/include/mozilla/interceptor/PatcherDetour.h:563

> And the 32-bit version: 
> https://queue.taskcluster.net/v1/task/KJewoG5qSa6WJCynYoYhhA/runs/0/artifacts/public/build/install/sea/target.installer.exe

The 32-bit version works instead (but my system is 64-bit).

I tried again to contact the public TURN server, but unfortunately the results did not change (nightly_screenshots.zip).

Comment 12

[Byron Campen [:bwc]]

(In reply to danilo.arcidiacono from comment #11)
> (In reply to Byron Campen [:bwc] from comment #5)
> > Thanks for the breakdown!
> > 
> > Support for http relays is really minimal right now. 
> 
> With "http relays" do you mean using "?transport=tcp" on ICE servers URLs?
> If so, using "?transport=udp" should work instead (I can't try this right
> now, because our corporate proxy blocks UDP traffic)?

   Sorry, that should have read "http proxies", like the corporate one you said you're using.

> 
> > But how would you like to test the fix over at bug 1203503?
> 
> If I understand, the problem is that Firefox can't get past proxies
> protected by authentication, because it does not implement the related
> security protocols (at least for the WebRTC part). But then, why does it
> work when serving the web page from the hard drive?

   That is an excellent question. It would seem that something about that scenario caused the proxy to be bypassed.

> 
> > Here's a win64 binary with better http proxy support: 
> > https://queue.taskcluster.net/v1/task/XRcfaWuRTjyEXO7kGzYVHA/runs/0/artifacts/public/build/install/sea/target.installer.exe
> 
> The 64-bit version of Firefox Nighty throws this error:
> 
> "Assertion failure: false (MOZ_ASSERT_UNREACHABLE: Opcode sequence includes
> commands after JMP), at
> z:/build/build/src/obj-firefox/dist/include/mozilla/interceptor/
> PatcherDetour.h:563

   Hmm, looks like bug 1342690. Some sort of antivirus conflict thing.

> 
> > And the 32-bit version: 
> > https://queue.taskcluster.net/v1/task/KJewoG5qSa6WJCynYoYhhA/runs/0/artifacts/public/build/install/sea/target.installer.exe
> 
> The 32-bit version works instead (but my system is 64-bit).
> 
> I tried again to contact the public TURN server, but unfortunately the
> results did not change (nightly_screenshots.zip).

   Thanks. I'll try to figure out what is going wrong here.

Comment 13

[Nils Ohlmeier [:drno]]

(In reply to danilo.arcidiacono from comment #10)
> Created attachment 9024360 [details]
> nightly_screenshots.zip

In the screen shot which compares Firefox vs Chrome I'm wondering why Firefox only shows host candidates for one local IP address, but Chrome shows hosts candidates for three different IP addresses?

One possible explanation could be that you have given Chrome access to your camera for the webrtc samples page. That would then result in Firefox and Chrome not gathering the same way, because Firefox only gathers default route, but Chrome because of the camera permission appears to gather in mode 1.

Comment 14

[Paul Vitale [:iidebyo]]

In the attached screenshots, I see Firefox is using a TcpsocketChild when an http proxy should be configured.  It is likely that Firefox does not "see" the http proxy.

Comment 15

[danilo.arcidiacono]

Attachment: proxy_settings.zip

Comment 16

[danilo.arcidiacono]

(In reply to Byron Campen [:bwc] from comment #12)
> 
>    Hmm, looks like bug 1342690. Some sort of antivirus conflict thing.
> 

It's probable, we also have a corporate antivirus installed on all our machines.

(In reply to Nils Ohlmeier [:drno] from comment #13)
> (In reply to danilo.arcidiacono from comment #10)
> > Created attachment 9024360 [details]
> > nightly_screenshots.zip
> 
> In the screen shot which compares Firefox vs Chrome I'm wondering why
> Firefox only shows host candidates for one local IP address, but Chrome
> shows hosts candidates for three different IP addresses?
> 
> One possible explanation could be that you have given Chrome access to your
> camera for the webrtc samples page. That would then result in Firefox and
> Chrome not gathering the same way, because Firefox only gathers default
> route, but Chrome because of the camera permission appears to gather in mode
> 1.

Yes, by denying the access to camera and microphone, Chrome gathers only one candidate (chrome_settings.png).

(In reply to Paul Vitale [:iidebyo] from comment #14)
> In the attached screenshots, I see Firefox is using a TcpsocketChild when an
> http proxy should be configured.  It is likely that Firefox does not "see"
> the http proxy.

Yes! It seems that Firefox (both Quantum and Nightly) can't auto-detect my system proxy settings (ff_with_system_proxy.png).
If I set the proxy manually, it works! (ff_with_manual_proxy.png)

Comment 17

[danilo.arcidiacono]

(In reply to Byron Campen [:bwc] from comment #12)
> > If I understand, the problem is that Firefox can't get past proxies
> > protected by authentication, because it does not implement the related
> > security protocols (at least for the WebRTC part). But then, why does it
> > work when serving the web page from the hard drive?
> 
>    That is an excellent question. It would seem that something about that
> scenario caused the proxy to be bypassed.


At this point it seems to me that Firefox can detect the proxy when the URL scheme is file, while it can't with https scheme.

Comment 18

[Honza Bambas (:mayhemer)]

Note that by default we bypass proxy for `localhost,127.0.0.1`.  this is configurable in proxy preferences (the "No Proxy for" field when you have set proxy manually.)  this might interfere with test builds bwc has provided in comment 7.

Comment 19

[Honza Bambas (:mayhemer)]

Maybe http logs may shade some light as well why those test builds don't work:
https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging

Comment 20

[danilo.arcidiacono]

Both Firefox Nightly (32-bit) and Firefox Quantum (63.0.1 64-bit) work if I configure the proxy explicitly in the settings;
moreover, I used a public TURN server in my tests (except for cases 1b, 2b, 3b and 4b in comment 3), so the proxy bypass should not play a role here.

Comment 21

[danilo.arcidiacono]

(except for cases 3a, 3b, 4a, 4b)

Comment 22

[Paul Adenot (:padenot)]

Byron, I'm assigning you on this one but it's a bit unclear who's taking it, feel free to adjust.

Comment 23

[Paul Vitale [:iidebyo]]

I did some testing on a VirtualBox Win10 VM [1] with the 64-bit Nightly build in comment 7.

WebRtc's PeerConnectionMedia (PCM) queries the ProtocolProxyService (PPS) in a content process. In a sandboxed Firefox content process, PPS cannot read registry keys storing proxy information and resolves all URIs to DIRECT when using system settings. PCM's PPS was able to successfully access the registry keys when sandboxing was disabled or enabled with level 0,1 and was blocked at levels 2,3,4,5.

In the same VM, I tested Firefox release 63.0.3 (64-bit) at sandbox levels 1 and 5. As expected, the registry keys at level 1 were allowed and blocked at level 5 (default).

I was not able to reproduce this on my Win8.1 PC.

The registry keys moved in Win10 from:

HKCU\Software\Microsoft\Windows\CurrentVersion\Internet Settings (Win8.1)

to:

HKC\Software\Policies\Microsoft\Windows\CurrentVersion\Internet Settings

Steps to reproduce:

1. Enable and assign a script address in Win10 proxy settings (open start menu, type 'Change proxy settings') OR assign a manual proxy but a script address was used in testing (manual was only verified with release FF level=5).

2. Disable automatically detect settings (shouldn't matter but it was disabled)

3. Open cmd.exe and navigate to relevant firefox install dir

e.g. cd "c:\Program Files\Firefox Nightly"

4. Set environment variables and start firefox e.g.:

set MOZ_LOG=webrtcProxy:5,proxy:5
firefox.exe 2> c:\moz\stderr.txt

5. Set preferences:

security.sandbox.logging.enabled = true
security.sandbox.content.level = { 0,1,2,3,4,5 }

6. navigate to https://webrtc.github.io/samples/src/content/peerconnection/trickle-ice/ and 'Gather candidates'

7. open standard error and search for lines like 

'[Child 2524: ProxyResolution]: D/proxy Use proxy from system settings: DIRECT'
'[Child 4768, StreamTrans #1] WARNING: 'Process Sandbox BLOCKED: NtOpenKeyEx for : Software\Policies\Microsoft\Windows\CurrentVersion\Internet Settings', '

https://developer.microsoft.com/en-us/windows/downloads/virtual-machines [1]

Comment 24

[Byron Campen [:bwc]]

Are we going to need to create an IPC interface to query the ProtocolProxyService on the parent process? Or maybe we could move this query into the socket process?

Comment 25

[Nils Ohlmeier [:drno]]

Two short notes to clarify things:

- ICE TCP does not support going through HTTP(S) proxies yet. So even if a HTTP proxy is configured it's going to bypass that. That is this bug https://bugzilla.mozilla.org/show_bug.cgi?id=1254891
- WebRTC doesn't support Proxy Auto Configuration. I don't think that we have a bug for that yet. And it would probably require to make our proxy detection depend on the actual TURN server URLs (instead of blindly asking for example.com).

Comment 26

[Honza Bambas (:mayhemer)]

(In reply to Byron Campen [:bwc] from comment #24)
> Are we going to need to create an IPC interface to query the
> ProtocolProxyService on the parent process? Or maybe we could move this
> query into the socket process?

I think this is one of the questions you will talk about with Dragana.

Comment 27

[Mike Kaply [:mkaply]]

What's the status on this? Customer has asked.

Comment 28

[Byron Campen [:bwc]]

Bug 1569183 might be enough to fix this.

Comment 29

[Mike Kaply [:mkaply]]

(In reply to Byron Campen [:bwc] from comment #28)

Bug 1569183 might be enough to fix this.

Should we dupe? Is there any way to know for sure?

Comment 30

[Byron Campen [:bwc]]

It is difficult to tell. If the only problem is that we're doing proxy config lookup stuff on the content process, that should already be fixed. However, there's a bunch more stuff that could be going wrong here.

Comment 31

[Byron Campen [:bwc]]

Is this working for you now on nightly?

Comment 32

[Mike Kaply [:mkaply]]

The customer I was in contact with said it is working for them. Need to hear from reporter.

Comment 33

[Mike Kaply [:mkaply]]

Danilo can you please let us know if this is working for you?

Comment 34

[GODIER]

Hi,

I'm working for a major network operator 150 000 employee company -> Orange
This bug of proxy automatic not working with TURN prevents all of us of using Firefox. I can't ask everyone to force their proxy in config because first the configuration is erases each reboot (company policies) and second everybody doesn't have skill for.
Firefox is the recommended browser ATM.
So do you plan to solve this bug "soon" or Do I have to ask everyones to download Google Chrome ?

best regards

JG

Comment 35

[Mike Kaply [:mkaply]]

What version of Firefox are you testing? We believe this was fixed in 71 via bug 1569183 We couldn't back port that fix to the current ESR, so it will be in ESR 78 in two months.

Comment 36

[Byron Campen [:bwc]]

See :mkaply's question in comment 35.

Comment 37

[GODIER]

(In reply to Mike Kaply [:mkaply] from comment #35)

What version of Firefox are you testing? We believe this was fixed in 71 via bug 1569183 We couldn't back port that fix to the current ESR, so it will be in ESR 78 in two months.

Currently all the group is using ESR 68.

Comment 38

[Byron Campen [:bwc]]

Ok. The next ESR (release date in late June) is going to be based on Firefox 78, which is the current nightly. It would be really good to know whether 78 works in your environment.

Comment 39

[GODIER]

(In reply to Byron Campen [:bwc] from comment #38)

Ok. The next ESR (release date in late June) is going to be based on Firefox 78, which is the current nightly. It would be really good to know whether 78 works in your environment.
I could test with last official available version (75) and it works.
2 months + the time the new version is propagated across all employees, is very long :(

Comment 40

[Byron Campen [:bwc]]

Yeah, that's the trade-off with ESR, sometimes it takes a while to get new features.