807494 - Set up public stun servers for WebRTC

Status: RESOLVED FIXED

(Core :: WebRTC, defect)

Description

[Ralph Giles (:rillian)]

WebRTC NAT traversal requires a STUN (or turn) server so clients can resolve their public IP address. This bug is about setting up a distributed network of such servers for our users.

Comment 1

[Ralph Giles (:rillian)]

I plan to set up some servers for testing on Amazon EC2, with traffic distributed by geo-variant dns responses.

The servers would be public for anyone to use, but the primary intent is to have something we can test against as we develop and deploy WebRTC where we can control the availablity and privacy protections.

Comment 2

[Ralph Giles (:rillian)]

Basic plan is to set up an instance running the default 'stun-server' package, see how that goes.

Comment 3

[Ralph Giles (:rillian)]

dev-media post for wider discussion:

http://thread.gmane.org/gmane.comp.mozilla.devel.media/141 or
https://groups.google.com/forum/?fromgroups=#!topic/mozilla.dev.media/1p4eLDOOM9g

Comment 4

[Ralph Giles (:rillian)]

I tried using boxgrinder.org to generate Scientific Linux VM images. Code for this is at

  https://github.com/mozilla/stun-vm

Unfortunately, SL 6.3 doesn't include a stun server package, which removes most of the impetus for using a long-support-lifetime distribution.

There are also so drawbacks with boxgrinder:
- it has to run as root
- docs suggest there can be information leakage between the host environment and the image
- it makes images with a default root password and doesn't set 'PermitRootLogin no' in sshd_config

So it looks like this isn't a great way to go.

Comment 5

[Ralph Giles (:rillian)]

Next I tried installing the 'stun' package on an official ubuntu 12.04 ec2 image.

I immediately hit the snag that stund wants two independent IP addresses so it can determine if the client is behind symmetric NAT, and ec2 only provides a single address by default.

This can be worked around with the new "Virtual Private Cloud" features, which allow assigning multiple private IP addresses, and then mapping public IP addresses to them. More complicated to set up; I don't have automation written for this yet.

With this procedure I got the instance to respond to stun queries:

$ stun-client 50.112.159.100
STUN client version 0.96
Primary: Indepndent Mapping, Port Dependent Filter, random port, no hairpin	
Return value is 0x000016

Next issue is with ec2 instances always being behind NAT. We'll need to hack the server to know what public addresses the internal ones map and use those instead when constructing response packets, to remove the EC2 NAT from the measurement.

Comment 6

[Timothy B. Terriberry (:derf)]

One other issue that juberti just brought up at the mmusic meeting is that AWS only has one datacenter in Asia, and it's in Singapore, which means that anyone trying to set up a call from India is going to have 500 ms RTTs.

Comment 7

[Ralph Giles (:rillian)]

That is an excellent point. (There's also one in Tokyo, but that's likely even further away from India.)

Do you have an alternative suggestion? I am wary of looking for an established service partner for our default, because of the privacy implication for our users. Longer term we can of course find dedicated server space in more locations, but I think AWS is still helpful to get something up quickly.

Comment 8

[Ralph Giles (:rillian)]

AWS also doesn't support ipv6.

Comment 9

[Ralph Giles (:rillian)]

To clarify:

STUN needs two public IP addresses. This is possible with AWS by launching an instance on a private subnet and connecting two elastic IPs. This is what I've done for the one available at 50.112.159.100. Abr has confirmed this setup works with our current code.

The stun-server software itself needs patching, to handle this cleanly, I'll open a new bug for that.

Comment 10

[Adam Roach [:abr]]

Attachment: Temporary patch to use new server

This patch isn't intended to be checked in. I'm attaching it as a convenience for anyone who wants to move over to the new server before Bug 825703 lands.

Comment 11

[Mark Mayo [:mmayo]]

Note you won't be able to use that IP above in #c9 - we'll have to bring up this instance in a Service VPC, where we can't take that IP.

Can we use a FQDN in there instead of an IP?

Comment 12

[Eric Rescorla (:ekr)]

Not right now, no, because we don't support FQDNs for STUN servers.

Comment 13

[Adam Roach [:abr]]

Mark: I think we all agree that a FQDN would be preferable to an IP address. Right now, we're working to get DNS resolution into the STUN handling, but we really don't want to block getting a new STUN server into the code on finishing that task.

Comment 14

[Eric Rescorla (:ekr)]

Comment on attachment 701941 [details] [diff] [review]
Temporary patch to use new server

Review of attachment 701941 [details] [diff] [review]:
-----------------------------------------------------------------

adam, we should land this patch.

Comment 15

[Eric Rescorla (:ekr)]

(In reply to Ralph Giles (:rillian) from comment #9)
> To clarify:
> 
> STUN needs two public IP addresses. 

This is only for NAT characterization via STUN. ICE does not rely on this.

> This is possible with AWS by launching
> an instance on a private subnet and connecting two elastic IPs. This is what
> I've done for the one available at 50.112.159.100. Abr has confirmed this
> setup works with our current code.
> 
> The stun-server software itself needs patching, to handle this cleanly, I'll
> open a new bug for that.

I don't understand what the patch is here. We seem to be getting reasonable responses
with the existing code.

Comment 16

[Mark Mayo [:mmayo]]

Also, per bug 830476, we (:whd, really) have server patches we're trying in svcops to handle the AWS NAT issue. We'll work these into our automated deployment of the server.

We'll update this ticket when we have a services-managed stun server up, and provide the new IP. ETA: today.

Comment 17

[Ralph Giles (:rillian)]

(In reply to Eric Rescorla (:ekr) from comment #15)

> > STUN needs two public IP addresses. 
> 
> This is only for NAT characterization via STUN. ICE does not rely on this.

That's great news, and simplifies the vm setup. I was just following the stun server docs.

> I don't understand what the patch is here. We seem to be getting reasonable
> responses with the existing code.

I'm referring to the 10.0.0.27 (and 10.0.0.103, since this server has two public addresses) in the stun response:

Received stun message: 88 bytes
MappedAddress = 64.213.70.194:46281
SourceAddress = 10.0.0.27:3478
ChangedAddress = 10.0.0.103:3479
XorMappedAddress = 64.213.70.194:46281
ServerName = Vovida.org 0.96
Received message of type 257  id=1

Comment 18

[Wesley Dawson [:whd]]

I have set up a patched stun server running at:

ec2-23-22-244-173.compute-1.amazonaws.com
ip: 23.22.244.173

This is without a VPC, as per comment 15. Relevant output:

$ stun-client -v ec2-23-22-244-173.compute-1.amazonaws.com 1
STUN client version 0.96
running test number 1
Opened port 19578 with fd 3
Encoding stun message: 
Encoding ChangeRequest: 0

About to send msg of len 28 to 23.22.244.173:3478
Got a response
Received stun message: 88 bytes
MappedAddress = 63.245.220.240:62379
SourceAddress = 23.22.244.173:3478
ChangedAddress = 0.0.0.0:3479
XorMappedAddress = 63.245.220.240:62379
ServerName = Vovida.org 0.96
         ok=1
         id=1:32:65:100:144:194:220:120:134:103:63:11:74:141:19:27
         mappedAddr=63.245.220.240:62379
         changedAddr=0.0.0.0:3479


Return value is 0x000000

Comment 19

[Adam Roach [:abr]]

I'm not going to be able to get online for another 3 hours or so. If someone can rev the patch to use the new IP and land it, we can get STUN working on Nightly at the next build. I'll check back later and move the ball forward if no one else has by then.

Comment 20

[Mark Mayo [:mmayo]]

Wrong IP above.  :whd fixing right now.  ;)

Comment 21

[Wesley Dawson [:whd]]

ec2-23-21-150-121.compute-1.amazonaws.com
ip: 23.21.150.121

Comment 22

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

Attachment: Temporary patch to use new server. r=ekr

With 23.21.150.121.

Comment 23

[Adam Roach [:abr]]

Jan-Ivar: thanks for grabbing this and running with it! I've verified the patch using the signaling unit tests, and everything appears to be working smoothly.

Comment 24

[Randell Jesup [:jesup] (needinfo me)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/c6f474736934

Comment 25

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/c6f474736934

Comment 26

[Ralph Giles (:rillian)]

I've taken down the experimental server at 50.112.159.100.

Comment 27

[Randell Jesup [:jesup] (needinfo me)]

landed on m-c but wasn't marked resolved