Set up public stun servers for WebRTC

RESOLVED FIXED

Status

()

RESOLVED FIXED
6 years ago
6 years ago

People

(Reporter: rillian, Assigned: mmayo)

Tracking

Trunk
x86_64
Linux
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [WebRTC] [blocking-webrtc-][leave-open][qa-])

Attachments

(1 attachment, 1 obsolete attachment)

WebRTC NAT traversal requires a STUN (or turn) server so clients can resolve their public IP address. This bug is about setting up a distributed network of such servers for our users.
(Reporter)

Comment 1

6 years ago
I plan to set up some servers for testing on Amazon EC2, with traffic distributed by geo-variant dns responses.

The servers would be public for anyone to use, but the primary intent is to have something we can test against as we develop and deploy WebRTC where we can control the availablity and privacy protections.
(Reporter)

Comment 2

6 years ago
Basic plan is to set up an instance running the default 'stun-server' package, see how that goes.
(Reporter)

Comment 4

6 years ago
I tried using boxgrinder.org to generate Scientific Linux VM images. Code for this is at

  https://github.com/mozilla/stun-vm

Unfortunately, SL 6.3 doesn't include a stun server package, which removes most of the impetus for using a long-support-lifetime distribution.

There are also so drawbacks with boxgrinder:
- it has to run as root
- docs suggest there can be information leakage between the host environment and the image
- it makes images with a default root password and doesn't set 'PermitRootLogin no' in sshd_config

So it looks like this isn't a great way to go.
(Reporter)

Comment 5

6 years ago
Next I tried installing the 'stun' package on an official ubuntu 12.04 ec2 image.

I immediately hit the snag that stund wants two independent IP addresses so it can determine if the client is behind symmetric NAT, and ec2 only provides a single address by default.

This can be worked around with the new "Virtual Private Cloud" features, which allow assigning multiple private IP addresses, and then mapping public IP addresses to them. More complicated to set up; I don't have automation written for this yet.

With this procedure I got the instance to respond to stun queries:

$ stun-client 50.112.159.100
STUN client version 0.96
Primary: Indepndent Mapping, Port Dependent Filter, random port, no hairpin	
Return value is 0x000016

Next issue is with ec2 instances always being behind NAT. We'll need to hack the server to know what public addresses the internal ones map and use those instead when constructing response packets, to remove the EC2 NAT from the measurement.
One other issue that juberti just brought up at the mmusic meeting is that AWS only has one datacenter in Asia, and it's in Singapore, which means that anyone trying to set up a call from India is going to have 500 ms RTTs.
(Reporter)

Comment 7

6 years ago
That is an excellent point. (There's also one in Tokyo, but that's likely even further away from India.)

Do you have an alternative suggestion? I am wary of looking for an established service partner for our default, because of the privacy implication for our users. Longer term we can of course find dedicated server space in more locations, but I think AWS is still helpful to get something up quickly.
(Reporter)

Comment 8

6 years ago
AWS also doesn't support ipv6.
(Assignee)

Updated

6 years ago
Assignee: giles → mmayo
(Reporter)

Updated

6 years ago
Blocks: 825703
(Reporter)

Comment 9

6 years ago
To clarify:

STUN needs two public IP addresses. This is possible with AWS by launching an instance on a private subnet and connecting two elastic IPs. This is what I've done for the one available at 50.112.159.100. Abr has confirmed this setup works with our current code.

The stun-server software itself needs patching, to handle this cleanly, I'll open a new bug for that.
(Reporter)

Updated

6 years ago
Depends on: 830476

Comment 10

6 years ago
Created attachment 701941 [details] [diff] [review]
Temporary patch to use new server

This patch isn't intended to be checked in. I'm attaching it as a convenience for anyone who wants to move over to the new server before Bug 825703 lands.
(Assignee)

Comment 11

6 years ago
Note you won't be able to use that IP above in #c9 - we'll have to bring up this instance in a Service VPC, where we can't take that IP.

Can we use a FQDN in there instead of an IP?
Not right now, no, because we don't support FQDNs for STUN servers.

Comment 13

6 years ago
Mark: I think we all agree that a FQDN would be preferable to an IP address. Right now, we're working to get DNS resolution into the STUN handling, but we really don't want to block getting a new STUN server into the code on finishing that task.
Comment on attachment 701941 [details] [diff] [review]
Temporary patch to use new server

Review of attachment 701941 [details] [diff] [review]:
-----------------------------------------------------------------

adam, we should land this patch.
Attachment #701941 - Flags: review+
(In reply to Ralph Giles (:rillian) from comment #9)
> To clarify:
> 
> STUN needs two public IP addresses. 

This is only for NAT characterization via STUN. ICE does not rely on this.

> This is possible with AWS by launching
> an instance on a private subnet and connecting two elastic IPs. This is what
> I've done for the one available at 50.112.159.100. Abr has confirmed this
> setup works with our current code.
> 
> The stun-server software itself needs patching, to handle this cleanly, I'll
> open a new bug for that.

I don't understand what the patch is here. We seem to be getting reasonable responses
with the existing code.
(Assignee)

Comment 16

6 years ago
Also, per bug 830476, we (:whd, really) have server patches we're trying in svcops to handle the AWS NAT issue. We'll work these into our automated deployment of the server.

We'll update this ticket when we have a services-managed stun server up, and provide the new IP. ETA: today.
(In reply to Eric Rescorla (:ekr) from comment #15)

> > STUN needs two public IP addresses. 
> 
> This is only for NAT characterization via STUN. ICE does not rely on this.

That's great news, and simplifies the vm setup. I was just following the stun server docs.

> I don't understand what the patch is here. We seem to be getting reasonable
> responses with the existing code.

I'm referring to the 10.0.0.27 (and 10.0.0.103, since this server has two public addresses) in the stun response:

Received stun message: 88 bytes
MappedAddress = 64.213.70.194:46281
SourceAddress = 10.0.0.27:3478
ChangedAddress = 10.0.0.103:3479
XorMappedAddress = 64.213.70.194:46281
ServerName = Vovida.org 0.96
Received message of type 257  id=1
I have set up a patched stun server running at:

ec2-23-22-244-173.compute-1.amazonaws.com
ip: 23.22.244.173

This is without a VPC, as per comment 15. Relevant output:

$ stun-client -v ec2-23-22-244-173.compute-1.amazonaws.com 1
STUN client version 0.96
running test number 1
Opened port 19578 with fd 3
Encoding stun message: 
Encoding ChangeRequest: 0

About to send msg of len 28 to 23.22.244.173:3478
Got a response
Received stun message: 88 bytes
MappedAddress = 63.245.220.240:62379
SourceAddress = 23.22.244.173:3478
ChangedAddress = 0.0.0.0:3479
XorMappedAddress = 63.245.220.240:62379
ServerName = Vovida.org 0.96
         ok=1
         id=1:32:65:100:144:194:220:120:134:103:63:11:74:141:19:27
         mappedAddr=63.245.220.240:62379
         changedAddr=0.0.0.0:3479


Return value is 0x000000

Comment 19

6 years ago
I'm not going to be able to get online for another 3 hours or so. If someone can rev the patch to use the new IP and land it, we can get STUN working on Nightly at the next build. I'll check back later and move the ball forward if no one else has by then.
(Assignee)

Comment 20

6 years ago
Wrong IP above.  :whd fixing right now.  ;)
ec2-23-21-150-121.compute-1.amazonaws.com
ip: 23.21.150.121
Created attachment 702101 [details] [diff] [review]
Temporary patch to use new server. r=ekr

With 23.21.150.121.
Attachment #701941 - Attachment is obsolete: true
Attachment #702101 - Flags: review+
Attachment #702101 - Flags: checkin?(rjesup)

Comment 23

6 years ago
Jan-Ivar: thanks for grabbing this and running with it! I've verified the patch using the signaling unit tests, and everything appears to be working smoothly.
https://hg.mozilla.org/integration/mozilla-inbound/rev/c6f474736934
Whiteboard: [WebRTC] [blocking-webrtc-] → [WebRTC] [blocking-webrtc-][leave-open]
I've taken down the experimental server at 50.112.159.100.
landed on m-c but wasn't marked resolved
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED

Updated

6 years ago
Attachment #702101 - Flags: checkin?(rjesup) → checkin+

Updated

6 years ago
Whiteboard: [WebRTC] [blocking-webrtc-][leave-open] → [WebRTC] [blocking-webrtc-][leave-open][qa-]
You need to log in before you can comment on or make changes to this bug.